Big data is a problem of two halves: data at rest and data in motion.
More important, says Matt Quinn, Chief Technology Officer of TIBCO, big data needs to focus on operational outcomes. Why? To allow customers to quickly identify the business problem — and the desired outcome – and then devise the right strategy to get them there.
Sounds good – but how do you pull it off? At the Transform 2013 conference in London, Dennis sat down with Matt to find out more.
One of the challenges, Matt explains, is that big data forces companies to think about data as not just something that sits in the database. Machine-generated data and log files can be just as important as the classic database box. Dennis points out that machine data might matter more because, a) there’s more of it; and, b) you can do more with it. Matt agrees, adding that it’s also easy to access. The trick? Processing it millions of times per second.
The guys talk fraud detection – a prime big data example. Writing a report and running it three weeks later to see how much fraud happened is far less valuable than capturing fraud as it’s actually occurring. Correlating data from various event streams can identify those patterns. Action can then be taken, either right then and there, or by predicting when fraud is most likely to happen.
To that end, Matt says that TIBCO relies on a team of data scientists to basically reverse engineer the data gathering process. Once you know what data you have and what you’re trying to do, you can then look for the information needed to complete the picture. The next step in big data? Operationalizing it.
:56 big data breaks down into data at rest and data in motion
1:46 The challenge lies in pulling machine log files millions of times per second
2:06 Data in motion involves taking action on patterns as they are happening
3:18 Fraud detection can be a function of using both kinds of big data
4:27 Context to the purchase needs to be considered, not just relying on algorithms
5:57 TIBCO’s data scientists can reverse engineer to discover necessary data
6:54 Next step in big data is operationalizing it