Wanted: people with the ability to see the relationships in big data, says Stephen Brobst, CTO of Teradata
We are running out of people with the imagination, skills and tools to take advantage of the zettabytes of data our information systems are starting to produce, and this shortage promises to become a crisis for businesses that seek to compete based on their capacity to innovate.
The present corporate mantra is to compete using better and faster business intelligence. The ability to transform data into competitive advantage is, therefore, a critical success factor today.
Increasingly, companies are directing their business intelligence activities at operational decision-making as well as strategic decision-making. After all, a strategy is only as good as its execution. In practical terms, this means using new insights to deliver superior supply chain efficiency, risk management, sales execution and customer experiences, so creating sustainable advantage.
We have created more data in the last three years than in the previous 40,000 of human history. We have moved from the exabyte age of storing and analysing transactions to the zettabyte age of storing and analysing interactions.
There is a subtle difference. A transaction might be a purchase from a website, whereas the interactions include the individual clicks on the website that led up to the purchase. Interactions might also include exchanges on a social network between individuals.
Much of this data is unstructured, which adds hugely to the processing and storage problem. In fact, the data created from interactions and other generated content has more than quadrupled in the last three years.
Most business and government decision-makers have no clue how to deal with the onslaught of big data. Knowledge workers have neither the skills nor the tools to cope effectively with data overload.
A recent study from the Data Warehousing Institute found 80% of all knowledge workers cannot create their own reports or analysis. In other words, most are mere consumers of reports created by others. This means that they are receiving answers to pre-determined questions. While these answers are useful to monitor conditions, they are not fuel for innovation.
True analysis is not about finding the answers to questions. Rather, it is about finding the right questions to ask. Once the right questions are known, getting the answers is relatively easy.
High value relationships
We know from experience that it is relatively cheap to produce answers to questions we already know how to ask. But the answers have low value compared with the discovery and use of new patterns and relationships in the data.
These skills of discovery are rare in most organisations. And they cannot be created on a two-week training course. The required skills include creativity, to drive out-of-the-box thinking, as well as the mathematical skills to use the data to test new hypotheses and see if the numbers add up.
Company culture makes a huge difference in creating a healthy analytic environment. Analytics thrives when experiment and failure is fast, cheap, and safe, initially sandboxed, if you will, from the company's mainstream activities.
It takes many failed ideas before we stumble on brilliant new insights. Organisations that welcome lessons from failed ideas enjoy more growth from innovation than those that punish time "wasted" on ideas that don't pan out. Companies such as eBay, Capital One, GE, and Harrah's Entertainment have embraced this approach, and have profited from their attitude.
Rising data volumes, time pressure, and decision complexity mean that the tools used by decision makers have become increasingly ineffective. Firms need data aggregation, filtering, and compression to deal with big data volumes without breaking the bank.
Wedding colour screens to personal computers lets us package data more meaningfully in pie charts, bar charts, and line graphs. But these fundamental constructs are little different today than they were 20-plus years ago.
This is dire for decision-makers. Given that a picture is worth a 1,000 words, we need better visualisation tools to make sense of big data. Making such tools is easier said than done because they must portray many different perspectives.
In the discovery phase of analysis we want the visualisation tool to behave like a telescope, going from the big picture in the data, seeing it from different angles and zooming into areas that look interesting. Once we find them, we need the tool to behave like a microscope to reveal the individual "cells" of the business and to show the relationships between the cells.
We also need our visualisation tools to behave like binoculars, bringing clarity to information hidden in a field of fuzzy data. We need to be able to "see" the money, "see" the customer, "see" the interactions, "see" the annual report on one page, and finally to "see" the relationships between them all. So far, we don't have the right tool.
To succeed in future organisations need to re-evaluate their investments in people, culture, and tools. Push button reports and dashboards are not enough to guarantee success.
Innovation often means doing different things rather than things differently. Getting there means asking new questions of the detailed data. People who can do this are seldom easy to find, train or retain. We need to cultivate and celebrate these folk who challenge conventional thinking. With the right tools they can find patterns in data that are opaque to most observers. Some of them will make all the difference to the organisation's future.
This was first published in November 2010