If you want big data, start small

If you believe everything the IT industry tells you, then by now “big data” should have resolved all your decision-making problems and presented executives and employees with the insights they need to do their jobs better.

It hasn’t? What’s wrong with you? The reality is, of course, that only a handful of early adopters can claim to be using big data at sufficient scale to transform their business.

There’s no doubting the potential for making much better use of the vast amounts of data being generated by our digital world, nor the opportunity that modern analytics tools present. But perhaps IT leaders need to rein in the enthusiasm from business chiefs that has led to such high expectations. There are a lot of basics that need to be addressed first.

Look at the government, for example. The Conservatives have bitten the data bug – and good on them for doing so. The new Cabinet Office minister, Matt Hancock, called this week for a “data culture” in government, instead of a target culture. He said data can help make better, more objective decisions on where to invest in public services and how to deliver savings – using data analysis to “guide service in real time”.

He’s not wrong – he’s just a long way from being able to do so. Only last year, Computer Weekly revealed that the government’s open data programme was being held back by problems with data quality and a lack of standard, usable formats. Let’s see ministers making real-time decisions based on the sort of “dirty data” sloshing around in Whitehall databases.

But let’s not be critical of the intent – it just needs the aims to be realistic.

Royal Bank of Scotland (RBS) – a company not without its share of IT problems – says it wants to use big data to deliver levels of personalised customer service not seen since the days branch managers knew their account holders by name.

But when RBS started to look at big data, it realised it had the same sort of problems the government faces. “When we started, everything was a mess, things were terrible. There was a conclusion: your data quality is terrible,” said the RBS head of data analytics.

As a result, the company first targeted small initiatives that could make a big difference, such as warning customers who were paying twice for identical products, or texting people who forgot to take their cash out of an ATM.

As more organisations tackle the big data challenge, further best practice along these lines will emerge. But the message at this stage for most IT leaders is: if you want big data, start small.