Sergey Nivens - Fotolia

Why it pays to get messy with big data

Big data does not have to be perfect to give useful insights. The ability to make messy data usable quickly could become business critical

Big data is no longer a novelty in corporate circles. It is a regular agenda item at boardroom meetings across the business landscape, and c-level executives are increasingly seeing the value of being at the helm of an insight-driven organisation, using data analytics and information to understand the market in which they operate.

In short, big data is the new normal for organisations that want to understand their customers.

However, there is a misconception among businesses that data analytics can be achieved only through precise, accurate data collection. On the contrary, being exact is not necessarily essential to achieving the desired results.

Sometimes the most comprehensive insights can be created by analysing heterogeneous data from many different sources. In today’s big data world, an organisation that wants to determine a general trend or (dis)prove a broad hypothesis can do so with a ‘quantity over quality’ approach.

As datasets continue to grow and open up, businesses are increasingly having to cope with information that is presented in different ways from a variety of sources. The data can be raw, plain and unstructured, but such chaos could be hiding a gem of insight for the business. The skill is to turn ‘messy’ data into usable, smart data.

Messy data can cause headaches for statisticians or analysts who value precision and process, not least because of the way it is presented. Too many anomalies and errors can lead to useless results.

Common sense is a crucial skill to successfully evaluate the uncertainty of the conclusion and the validity of the quantitative method. By acknowledging that the input data is imperfect, organisations will have made the first step to turning the data into understandable, useful insight.

Accept errors and imperfections

Fundamentally, the way you handle your data determines how much value is yielded from it. With this in mind, imperfections and errors must be an accepted part of the analysis – otherwise the numbers will not be crunched in an efficient and timely manner.

The few erroneous figures will simply be dwarfed by the sheer volume of information, and will not be enough to skew the results when analysed.

Equally, treating big data as something imperfect and imprecise can enable organisations to make superior forecasts and accurate predictions about their particular industry. In fact, the acceptance of messy data makes it easier to eradicate bias and preconceptions about the data being input.

How to make order out of chaos

And it is now easier than ever to make sense of the mess. Cognitive analytics offers a way to bridge the gap between big data and the reality of practical decision-making.

Through cognitive analytics, and deep learning in particular, it is possible for algorithms to abstract information from messy sources, deriving high-level concepts that can be manipulated more easily.

Businesses can capitalise on these recent advances to process and analyse complex data types, such as images, language files and video.

By having a positive approach to tackling data, businesses will be able to make practical decisions and gain a competitive edge in the market.

Harvey Lewis is analytics research director at Deloitte

Read more on Business intelligence and analytics