Researchers claim that 90% of all information was created in the past two years. Fast-forward to 2020, and it’s a sure bet that companies will be counting a lot more zeros in their data volumes – getting to grips with it now and turning it into meaningful business intelligence (BI) is essential for future survival.
Bill Limond, former CIO at City of London and an IT adviser to government, says data is coming from everywhere. “There are sensors gathering climate information, social media sites, digital pictures and videos, purchase transaction records and cellphone GPS signals, to name a few. This is big data,” he says.
Limond points out that “big data” has been around for years, right back to Sumerian records of wealth and power in 3400BC, but he says it is now a growing torrent.
“There are 30 billion items shared on Facebook every month, for example, and the projected global data growth per annum is 40%. In the digital economy, data is the new currency,” he says.
Harvey Lewis, analytics research director at Deloitte, says by 2020, even the term big data will be obsolete. “It will be massive data, but I expect we will run out of adjectives to describe it,” he says.
In 2012, humans created 2.5 zettabytes of data, according to research firm IDC. By 2020, we will be looking at 40 zettabytes per year – one zettabyte is a billion terabytes.
“If you converted all of that data into text and printed books, the pile would stretch from Earth to Pluto and back 16 times,” says Lewis.
It might be hard to get your head around such mind-boggling sizes, but one thing experts are agreed on, however, is its importance.
Big data and bigger data
A third of the data in the digital universe, amounting to more than 13,000 exabytes, will have business value by 2020, but only if is tagged and analysed, according to researchers at IDC.
There is much work to do, as only 3% of the potentially useful data in 2012 is tagged, with even less analysed. IDC calls this the untapped big data gap.
Large amounts of information will remain untapped and unexploited, unless organisations begin to exploit opportunities now to gain an edge over competitors, and get closer to their customers.
Even today, 23% of the information in the digital universe (643 exabytes) would be useful for business analytics if it were tagged and analysed, and as big data is on a steep upward trajectory, it presents an opportunity to unlock value, but this will take hard work and investment.
Big data technologies must economically extract value from large volumes of data, of different varieties, by enabling high-velocity capture for analysis.
The vast majority of new data is unstructured, which means little is known about the data unless it is characterised or tagged using metadata.
IDC foresees exciting developments, with some sectors well placed to reap the benefits by 2020.
Data that is particularly useful for analytics includes surveillance footage, with date, time and location automatically attached to a video file. As internet-connected cameras develop, there is more scope to embed intelligence into the camera so footage can be captured, analysed and tagged in real time. This has potential for crime investigation, retail analytics, and military intelligence.
Embedded and medical devices with sensors will capture biometrics, and be able to do things such as track medicine effectiveness or monitor potential outbreaks of viruses in real time.
Entertainment and social media data will be pumped out for free by more and more users, and used for predicting trends and helping to spot the next big thing.
Consumer images that divulge much information about them will be increasingly exploited by 2020, but it will rely on the introduction of sophisticated tagging algorithms that can analyse images, either in real time when pictures are uploaded, or in bulk after they are harvested from websites.
Essential to business
Darren Vengroff, chief scientist at retail personalisation specialists RichRelevance, says BI and analytics will not just be more important, it will be essential to running a successful organisation.
“As more data becomes available and the cost of deriving insight from the data continues to drop, organisations that don’t use it will simply not be able to compete,” he says.
This is a stark wake-up call, but there are a lot of things that need to happen in the meantime and opening up big data to a flat hierarchy within organisations is a key development, so that all workers are able to make use of the information that can be mined. Of course, not everyone will like it, as it involves a major cultural shift.
“We call it democratising the data. Every employee at every level is going to have to make more data-driven decisions and far fewer gut decisions,” says Vengroff.
“This is going to be a major cultural change for many businesses. In many cases, the hardest part of the organisation to change will not be the rank and file, but the senior management, with decades of experience and at least some amount of success making decisions without access to much data.”
There may be resistance but such a move is essential for future survival, because real-time decision-making will be critical to capitalise on growing current trends, such as globalisation, growth in the use of smart devices, and different ways of interacting with organisations – all of which are fuelling data growth.
“The most critical reason to make decisions in real time is because customers change context in real time, without even giving it a second thought. A shopper may be buying a dress for herself one moment and the next moment she is looking for a birthday present for her son. She has mentally made the context switch, so any data-driven system or process that is interacting with her, had better make the switch too. What was relevant 10 seconds ago may now be completely irrelevant,” says Vengroff.
Technology has to develop to allow this cultural shift, and Deloitte’s Lewis says the data and technology will become completely transparent to users and consumers.
“People won’t think about data driving the analysis or the technology – all that matters will be the content and the insight delivered,” he says.
He compares it to television – where once people talked about the technology, such as the move from black and white to colour, now it is more about accessing content, however that is achieved.
“The same principle will apply to data and analytics,” says Lewis.
Surge in innovation
Today, there is a major surge in innovation from the open-source community and the big service providers to create tools that can work effectively with big data. Vengroff believes over the next seven years we will see leaps in development to meet required demand.
“A lot of today’s tools look like the same basic reports and spreadsheets that people have been using for years. If anything, they may be harder to work with than in the past because there are so many metrics and graphs,” he says.
“The best tools of tomorrow will not just report what happened, or what is happening in real time, but will offer alternative scenarios, for example a price drop, more advertising spend on a product, etc, along with good predictions of the likely outcome of those scenarios. The user of the tool can spend their time thinking about what their business objectives should be, and then ask the system to implement the changes for the scenarios with the best predicted outcomes relative to those objectives.”
Lewis says there is a lot of effort being focused on “discovery technologies”, given the vast volumes of data to deal with. He points to how search engines have evolved and become more intelligent, changing from technologies that initially just helped you search, to become technologies that find what you actually need.
“Advanced analytics will shift from looking backwards to looking forwards – technologies that predict and anticipate,” he says.
Discovery, visualisation and data management will integrate and become seamless to deliver insights and trends to users.
“The NoSQL movement and the SQL movement, the unstructured database camp versus the relational database camp, will all have to be integrated to create a hybrid approach that focuses on insight,” says Lewis.
The best tools of tomorrow will not just report what happened, or what is happening in real time, but will offer alternative scenarios
Darren Vengroff, RichRelevance
A better automated understanding of exploring data will push semantic technologies that make connections between structured and unstructured data.
“The volumes of data will challenge what humans can do in terms of pattern recognition. As the complexity and volumes increase it will be less possible to add the human element,” he says.
Government IT adviser Limond says organisations need to think about how they will benefit from big data in the future, and be in a good position to exploit it. He suggests looking at multiple data sources and creatively sourcing internal and external data, and focusing on easy data merging – which may involve upgrading IT.
He also says it is important to strike a balance between complexity and ease of use, and to transform company capabilities.
“Develop usable business-relevant analytics, embed analytics into simple tools for the front line, and upgrade processes and develop capabilities to exploit big data,” he says.
Security is contentious
One bone of contention will be chewed over for many years – security.
Vengroff says the debate needs to be conducted without hysteria, otherwise it could affect the quality and therefore the relevance of big data.
“When we talk about privacy, it is critical that we frame the question as openly and honestly as possible, so everyone on every side of the debate can decide how they want to participate. Otherwise we’re either going to end up with a near total lockdown on use of personal data, or a wild west free-for-all. Neither of these is a desirable outcome,” he warns.
Read more on big data analytics
Vengroff says the key to framing the question is to recognise that we will never get privacy right without trust and value creation.
“If I give you personal or private information about myself, I have to trust that you will be a good steward of that information and that you will use it to create value for me. This isn’t new; it has been the case for centuries. The merchants with the best customer service have always known the most about their customers, not just about their tastes, but about their lives, their relationships, their hopes and their dreams. They have then used that information to make sure they offered the most personal, bespoke service to their customers to meet their every need,” he says.
A hundred years ago, very few people could afford this level of service, although it existed. But now that level of service may be attained by many more customers.
“Today, because much of the process can be automated at modest cost, more consumers than ever can benefit from the value of personalisation. However, we have to circle back to trust. If I as a consumer don’t understand what you will and won’t do with my data and who you will or won’t share it with, that is going to offset any value you create for me. It’s no different than it was a century ago. If a bespoke tailor started gossiping about their customers, you can bet they would lose those customers very quickly,” says Vengroff.
Transparency and trust
Lewis agrees that transparency and trust will become increasingly important.
“Customers and users will get to know the companies they interact with better and increasingly take control of their data,” he says.
Moves such as the UK government’s MiData initiative are a taste of the future. MiData aims to create an information-based consumer society where citizens have access to the data that companies have on them so they can find better deals or new services.
“Information exchange will be a two-way process, much more dynamic and therefore volatile and businesses will need to respond very quickly to changes and focus on retaining their customers through data transparency and engagement. There will be an emphasis on the responsible use of data,” says Lewis.
Already, we are seeing the growth of disruptive businesses based on data transparency, such as peer-to-peer lending in the financial sector, and this trend will grow, says Lewis.
“People will be able to make more informed decisions based on increased amounts of data, and smaller companies will be able to compete with big companies more effectively. It is a fantastic opportunity for business,” he says.
Vengroff says trust is the key to success.
“Businesses that collect and use big data need to maintain the trust of their customers and use the data to create value for their customers. This is no different than when data was much smaller," he says.
“If they overstep the bounds and abuse the trust of their customers, value is destroyed rather than created, and they invite the kind of draconian regulation that is not ultimately productive. The ball is in the big data collectors’ court.”
This was first published in July 2013