Hadoop silos need integration, manage all data as asset, say experts

Big data hype causes disorders in organisations, including Hadoop silos, exaggerated importance of data scientists and too much experimentation

Big data hype has caused infantile disorders in corporate organisations over the past year. Hadoop silos, an excess of experimentation, and an exaggeration of the importance of data scientists are among the teething problems of big data, according to experts, who suggest organisations should manage all data as an asset.

Steve Shelton, head of data services at consultancy Detica, part of BAE Systems, said Hadoop silos have become part of the enterprise IT landscape, both in the private and public sectors. “People focused on this new thing called big data and tried to isolate it [in 2011 and 2012],” he said. 

The focus has been too concentrated on non-traditional data types, and that has been driven by the suppliers. The business value of data is more effectively understood when you look at it all together, big or otherwise, he said.

Have big data technologies been a distraction? “I think it has been an evolutionary learning step, but businesses are stepping back now. When it comes to information governance, you have to look at data across the patch,” said Shelton.

He said Detica had seen complaints about Hadoop silos, and these were created by people going through a proof-of-concept phase, setting up a Hadoop cluster quickly and building a team. But a Hadoop platform involves extra costs on top, in terms of managing it and integrating it into your existing business processes.

The business value of data is more effectively understood when you look at it all together, big or otherwise

Steve Shelton, Detica

“It’s not been a waste of time and money, it is just a stage. And it is not an insurmountable challenge. The next step is to integrate those silos, but the thinking is immature relative to the technology itself,” said Shelton.

Big data still teething

Debra Logan, information management distinguished analyst at Gartner, said there is a consistent element of exaggeration in the rhetoric of big data. 

“It’s the number two most searched term on Gartner.com, after Magic Quadrant, but my sense, from one-to-one conversations with clients, is 97% or so organisations are in an exploratory phase only," she said.

“We do have media clients, managing video, and so on. But even the banks are not really doing big data. Their data is well organised. Fraud detection analytics, for example, uses methods that have not really changed much,” said Logan. Though Hadoop could be interesting from an underlying infrastructure viewpoint, she added.

Logan said the current market excitement around information presents a “great career development opportunity” and confirms a mathematical expertise skills gap. Gartner clients have gone looking for data expertise in the supplier community and have not found it, she added.

There has been an explosion of job titles around information, reflecting the fact that largely no one has been controlling the management of data as a business asset in corporate organisations, said Logan. 

Gartner has talked to around 50 chief data officers, mostly in North America and the UK, who are distinct from CIOs. Most are in financial services, often with a compliance agenda, and are trained on getting value from data.

While CEOs clamouring for the recruitment of data scientists are reacting to hype, she said, they are responding correctly to the recognition that there is an asset in the company – data – that is not being managed as an asset.

Image: Tay Jnr/DigitalVision/Thinkstock

Read more on Master data management (MDM) and integration