michelangelus - Fotolia

The big data project challenge

On discovering that many big data projects fail Billy MacInnes wonders how the industry can ensure more of them are successful

My eye was caught by press release from Gartner, which made the startling claim that 60% of big data projects up until 2017 “will fail to go beyond piloting and experimentation, and will be abandoned”.

Gartner argued that for companies getting started with advanced analytics, changing mindsets and culture was as important as acquiring tools and skills.

Research director Lisa Kart said many business intelligence and analytics leaders “were unsure how to get started with advanced analytics, and many organisations feel they must make a significant investment in new tools and skills. But a successful advanced analytics strategy is about more than simply acquiring the right tools. It's also important to change mindsets and culture, and to be creative in search of success.”

Gartner set out four best practices to get advanced analytics projects off the ground:

* Choose a business problem that offers an initial win;
* Use outsourcing and buy packaged apps when you lack advanced analytics expertise;
* Identify stakeholders in the organisation that need to be convinced of the value of advanced analytics;
* Decide if you want to build the skills and tools internally.

While interesting in its own right, I found myself wondering if big data projects might continue to fail or disappoint for the very simple reason that the more we try to analyse the ever growing volumes of data being generated, the more likely it is to yield fascinating but useless results.

We live in an age where data is being generated by more and more devices in our daily lives and the advent of the internet of things is going to make this trend even more pronounced. But how valuable is most of that data? And how do we go about extracting the valuable data and deleting the dross?

When I look around my kitchen in the morning, I can see the leftovers from breakfast and the dog hairs scattered over the floor mat. I know I have to throw the leftovers away, wash the dishes and hoover the dog hairs up. I am never in any doubt that the dog hairs have any value to me. I don’t have to store them for a specified period in case they may be of value to me further down the line. I can remove them in complete confidence.

Similarly, I can wash the dishes and erase the evidence of the morning’s cereal without a qualm. No one tries to convince me that I would be best served stashing all my leftovers and dirty dishes in a wardrobe and keeping them for several years on the off chance they might elicit some interesting information at a later date. Eventually, I’d have to build another kitchen to house all the hairs and leftovers.

I’d have to be mad to do that.

And yet, for the most part, that seems to be what the IT industry is saying to customers: keep everything, just in case. And the rate of data generation is so rapid that it’s actually quicker and cheaper to add storage capacity than to try and delete non-essential data before you store it. How mad is that.

No wonder our data keeps getting ‘bigger’ when the approach to data storage is akin to keeping all the dog hairs and leftovers rather than throwing them away and building more kitchens to store them. No wonder organisations are so desperate to find a mechanism to extract value from all their data.

But if you think about it, it’s a bit like those hoarders you see on TV programmes that fill their houses with stuff and end up with less and less space for their daily living. They think that stuff is valuable but, in reality, nearly all of it is junk. Even if there is anything of value, it’s buried under so many layers of rubbish that no one can ever find it. In the end, someone has to come in and throw everything out.

Most of us sort the rubbish we accumulate in our lives from items of value at a much earlier stage and throw the junk out long before it has any chance to overwhelm us. It’s bizarre that so many people and organisations don’t expect the IT industry to help them do the same when it comes to their data.

Read more on Data Storage Hardware