Formerly the bailiwick of highly tech savvy Web analytics companies and Silicon Valley dotcoms, “big data” technology is gradually beginning to show up in more traditional organisations such as banks, telcos and insurance firms. And this is just the beginning, according to Stephen Brobst, the chief technology officer at data warehousing firm Teradata.
Brobst, one of many IT industry insiders to frame the big data discussion in 2012, spoke with SearchDataManagement.co.UK twice this year about why he thinks the hype around big data has been focused in the wrong place. He also gave his thoughts on the prospect of data warehousing in the cloud, the “hype” surrounding in-memory technologies and the future of mobile consumer intelligence. What follows is an edited version of those interviews.
For more on big data technology adoption
Find out how big data laggards can expect competitive disadvantage
Jill Dyché explains why big data is both good news and bad news for IT
Find out how big data MDM is key to unlocking business value
What’s your view of the big data phenomenon?
Stephen Brobst: The hype has been around the volume of big data, which is the least interesting thing about it. Let’s apply Geoffrey Moore’s “crossing the chasm” technology adoption curve. We have been on the left hand side of the chasm, with the inventors and early adopters. All are high-tech, dotcom companies. What is different this year has been the adoption of big data analytics by more mainstream companies, like banks and telcos, who are not technology companies, precisely not dotcoms.
But aren’t big data technologies like Hadoop, MapReduce and so on still the preserve of hard-core software engineers who typically work for Silicon Valley Internet companies?
Brobst: Hadoop has a lot of work to do in crossing the chasm. But big data is not synonymous with Hadoop, which is still for very techie, savvy companies. No, Moore’s concept attends to the transition from the tech lovers to the pragmatists, who are not interested in technology for its own sake. For us, [Teradata’s] Aster Data builds the bridge to cross the chasm. In 2011, the big users of Aster Data were all dotcoms -- LinkedIn, BarnesandNoble.com, and so on. Today the biggest growth is in mainstream businesses -- telcos and banks, mainly in the US, true, but this will come to the UK, via the East Coast.
Much of the discourse around big data technology has focused on opposition between open source and traditional proprietary relational database technologies. How do you view the relationship between the two?
Brobst: For us, our partnership with Hortonworks is [emblematic]. There you have a lot of the original developers of Hadoop. Their technology is all open source. What they are doing that is different and interesting is that they are developing a metadata infrastructure that allows you to be more productive in getting access to the data -- rather than doing raw programming.
What’s your take on in-memory analytics?
Brobst: It’s hype. It is fine to say that the price of memory is going down every 18 months by 30%, but it is not cheap enough to store everything. The other side of the math is that the volume of data is growing by at least 40% every 18 months. So economics are not pointing all to in-memory. SAP is marketing HANA as solving all of the world’s problems. Larry Ellison has suckered them into it. It’s a lion in a cage thing, with SAP acting irrationally against the prods from Oracle’s marketing machine. SAP wants to kick out Oracle databases. And so it acquired Sybase, which was not even certified by SAP, making it the only choice! But it does not scale. It is not rational for enterprise customers. They’d have been better teaming up with IBM’s DB2.
But isn’t enterprise data warehousing antiquated now? Some of your competition would say that.
Brobst: Well, if you can’t do it yourself, discredit it. Now nobody is able to put all their data in one place. Does it make sense to put all emails or all your .wav files in a data warehouse? Probably not. But enterprise content management is a different thing than data warehousing. For relational data, integrated data warehouses are more cost-effective and of higher business value than a collection of small data marts, which is the Sybase model. You end up duplicating data, aggregating in different ways, and the cost mounts.
How do you see cloud computing impacting your approach to data warehousing?
Brobst: Public cloud infrastructure is making no impact on data warehousing because no CIO in their right mind would put financial data or customer data in the public cloud. That would be insane. But private cloud does work. First you can [use a private cloud to] consolidate data marts to reduce under-utilization. You get a factor of four to five times infrastructure cost savings that way. Second, you can integrate to deliver business value in an agile way. In centralised solutions, change takes too long from a business point of view. But you can use a data lab concept in a private cloud when, [for example], integrating another data source, internal or external. The critical idea is that the data lab is in the same environment as the integrated data warehouse. So you are not duplicating the centralized data that way, you don’t have the security issues, and you are empowering the users in a way that was not true of grid computing, which was a similar concept to cloud.
Cloud and mobile are often mentioned in the same breath as game changers in corporate IT. How do you see mobile as a vector for data warehousing and business intelligence?
Brobst: The key concept, for us, is “consumer intelligence.” We have, as a community, been mainly focused on decision making for employees in companies. The shift here is to providing intelligence for consumers who are your customers. Mobile devices are critical to that. A UK example is Lloyds Banking Group. A US example is Wells Fargo. The idea is that you are the CEO of your own household. Another use case is in utilities. Southern California Edison is implementing smart metering. On Sce.com consumers can analyse their patterns of energy consumption. [Mobile] consumer intelligence is a big trend coming.