This article is part of our Essential Guide: Data centre analysis and business intelligence guide

'Big data' wades into mainstream

“Big data” is gathering interest beyond high-tech companies. But there is much confusion, a lack of governance and a skills shortage that won’t go away soon.

The term big data is on fire. And while it is still early days, with much confusion in the market, corporate organisations are broaching big data projects, confirm analysts, suppliers and users.

UK blue-chip user group The Corporate IT Forum is one group that has expressed interest in big data. “While our members have not really done much in this area, they are interested in how you manage data to stop sprawl and in the issues of managing and analysing unstructured data,” a representative said. Big data encompasses unstructured and semi-structured data types and is a term of convenience for petabytes and exabytes of data volume.

On the vendor side, Stephen Brobst, chief technology officer at Teradata sees 2012 as the year that big data “crosses the chasm.”

“We have been on the left side of the chasm with dotcoms. But this year we’ll see banks and telcos doing real big data analytics projects,” he said. “It’ll still be early adopter, but no longer just [software] engineering companies.”

For more on big data’s journey to the mainstream

Read Bill Goodwin’s account of how organisations are getting to grips with big data at the Computer Weekly 500 Club

Watch Mike Ferguson’s interview at the Computer Weekly 500 Club

Learn how new data scientists are navigating the seas of big data

Joshua LeCure, a US-based business risk intelligence manager for pharmaceutical company GlaxoSmithKline’s audit and assurance group confirmed the company is starting up a data analytics and risk intelligence capability. The company’s internal audit group oversees a range of activities globally, from manufacturing standards and research practices to corporate finances and IT processes.

“Each of these areas generates massive amounts of data, and so far it has been an immense challenge to ascertain not only how best to leverage this, but to identify which data elements are of value,” LeCure said. “We want the individual auditors to perform self-discovery analyses on the ground, as well as to act as a central hub to perform advanced analytics on the available enterprise data.”

“However,” he went on, “even the early adopters are struggling to get their arms around [big data]. It is still early days.”

Gartner research director Roxane Edjlali and Mike Ferguson, an independent analyst, agreed that there is much confusion about big data.

Edjlali said: “There is no way out. Smart meters, social media, it is all there. And there is value in managing it. Organisations that will leverage the new data types will outperform their peers by 20% by 2015. In Europe, the energy companies are getting equipped with smart meters, and at Gartner, we are monitoring an increase in interest in sentiment analytics [among clients]. But companies are still in investigation mode.”

Ferguson confirmed he is seeing “a lot of mainstream interest gathering around big data to gain competitive advantage” in sectors such as logistics, manufacturing and “the sleepy world of insurance,” where “you’ve got smart boxes going into cars,” generating data that bears on premiums. “And marketers beyond e-commerce companies are interested in graphing maps of influence in social networks.”

But Ferguson also reported a “lot of confusion,” with a lack of familiarity among senior IT leaders with big data technologies, such as Hadoop, graph databases and massively parallel databases. “The confusion is around when to use what for the analysis you want to do and how to integrate it. There is still a lot of tyre kicking going on. But at back end of 2012 and the start of 2013 we will start to see big projects emerging. We are at the thin end of a very big wedge.”

Ferguson spoke at a recent Computer Weekly 500 Club meeting on the subject of big data. Hannelie Gilmour, chief architect at Standard Life, was another of the speakers. She related that data at the insurance company is increasing at 30% a year and confirmed that it has projects looking at how to enhance analytics, including sentiment analytics, from that data, “exploring what is the commercial value.” If you ignore big data, your competition will not, she warned the chief information officers and IT directors at the club’s meeting.

While the meeting attendees heard about organisations exploring big data for reasons of mere fashion, Ferguson argued that this will pass. “There is no doubt that organisations have spun off big data projects in an uncontrolled manner, but those will be stopped if there is no benefit. Governance is not there yet in the big data world, but it will be.”

A lack of skilled staff will, though, continue to be a serious issue for a long time to come, he said. “People are not finding those [big data analytics] skills in their BI teams and sourcing from other places. It’s a developers market.”

Edjlali concurred. “This [trend] requires multifaceted skills,” she said. “For example, the programming skills to do pattern identification to identify customer churn, the statistical background, the business focus. This is not common.”

Read more on Business intelligence and analytics