Dark data could halt big data’s path to success

Dark data – data that lacks any control or classification, but is all too prevalent in many organisations – could halt big data's path to success

One of the attributes of winning companies is how deeply they understand their market, customers and competitors. Getting better at that may well be the key to survival in our hyper-competitive global market.

Analysing data is one of the ways to gain insight and uncover new opportunities. Advanced content management, search, business intelligence and other big data technologies have offered the prospect of being able to affordably do so at hitherto unimagined scale.

Given their potential to deliver massive value and competitive advantage, these technologies have seen tremendous growth and adoption over the past 18 months, according to research conducted among AIIM members.

According to the survey report, titled Big Data and Content Analytics: measuring the ROI, while big data analysis is increasingly seen as an essential core competence, 60% of organisations admit to "inadequate" BI (business intelligence) reporting capability, with an even larger number, 65%, confirming "somewhat disorganised" content management approaches.

The evidence points to the unresolved problem of what is termed "dark data" – data that lacks any control or classification, but which is prevalent in all too many environments.

It seems that connecting and linking multiple systems is the biggest challenge for potential big data projects – particularly joining structured and unstructured data sets and analysing textual data.

The study also highlights another big data obstacle: a growing skills gap. Having sufficient skilled users is rated as the next biggest challenge; no wonder 34% of early adopters either outsourced projects or brought in outside expertise (13%).

A further third say they could only make progress by identifying and training internal specialists, with the remaining third relying on existing in-house expertise.

Security is the third major big data adoption challenge – a potential show-stopper for nearly one in five respondents.

Protecting personal data is the primary concern, though commercial and financial information is also sensitive. Some of this can be fixed using automated cleaning and sanitising of content repositories – big data application in its own right.

Changes in the past 18 months

How does this picture compare to the last time we looked? When AIIM last probed big data, in April 2012, we found reaction to the hype, and a good deal of confusion.

CIOs need to bring their big data project back to BI basics – and concentrate on content management, enterprise search and conventional BI capability

Doug Miles, AIIM

Now early users are reporting reasonable levels of success – in terms of reliable results, with 60% of early users feeling they have achieved payback (30% are less certain on that question due to high technology and expertise costs). 

Big data projects are starting to contribute to core business, with over half of early users (56%) sufficiently confident in the outcomes to use them in decision-making, including 6% using them as a basis for strategy choices.

Beyond the early adopters, the understanding of big data technologies and the genuine business potential has climbed through the hype, with most respondents seeing many potential applications and benefits, albeit more to help smooth-running and profitability than to produce dramatic competitive advantage.

What is the big picture?

The overarching situation is that many organisations are too immature in their content management, search, and basic reporting to contemplate big data projects just yet – although they are making technology decisions today with a view to a big data future.

In response, CIOs need to bring their big data project back to BI basics – and concentrate on content management, enterprise search and conventional BI capability. The simple fact is that these issues will need to be resolved before big data projects can be realistically considered.

Doug Miles, director of market intelligence for AIIM, the global community of information management professionals, examines what challenges organisations are facing when it comes to big data.

Read more on Regulatory compliance and standard requirements