The explosion of data being managed by organisations is driving demand for solutions to help them extract value and better understand their business and customers. But as competition to extract intelligence from data intensifies, companies will need to be cleverer about the questions they ask if they are to get useful answe...
For the past few years, organisations have been struggling to manage a truly staggering increase in commercial and corporate data volumes. Now, as it becomes clearer that there may be money in these bulging datacentres, something of an arms race is underway to uncover it.
"There has been an order of magnitude increase in the amount of data companies are attempting to manage. Now there is a fear that 'if my competitors are doing more with information than me then I'm at a disadvantage'," says Gartner research director Gareth Herschel.
The volume of data being generated online alone is encouraging organisations to get faster and smarter at processing it, while new and emerging tools increase their chances of finding hitherto undreamt of connections between things.
The drive to improve customer service has led to call centres becoming data collection points in their own right as companies try to match, for instance, conversations with customers with online behaviour. Advances in speech recognition technologies are even allowing companies to throw other information, such as emotional content and triggers, into the mix.
Deloitte strategy consultant Tim Willey says that the field of data analytics has sprouted from something of a niche industry, catering to only the most market-savvy companies, to become a core business function. "We are seeing data analytics employed for real competitive differentiation rather than merely a back office function."
The number of companies offering various data analytics services has mushroomed.
Gartner estimates that over the past five years, the volume of "structured" data, such as spreadsheets and databases, being handled by organisations has risen by 220% year on year. Unstructured, or emergent, data types, such as information on online behaviour, are growing even faster.
Herschel says companies such as Amazon have seen their total data volumes increase by around 1,000% year on year, as more and more detailed information on customers is collected.
Poster child for business intelligence, hotel and casino chain Harrah's realised early on that the better it understood its customers the more money it would potentially make. More than 10 years ago it deployed a solution from WINNet Systems which allows it to compare data on all of its customers across all of its businesses.
By analysing information such as gender, age, place of residence and types of casino games played, Harrah's achieved a $100m (£60.7m) increase in revenue from customers who visited more than one property. "It found out that happy customers spend more, and that different customers will have a different point of unhappiness compared to others," explains Willey.
Such data modelling has proved to be especially useful for industries such as insurance, as well as airlines, for which understanding human behaviour is so critical.
The potential value of customer data is informing the way in which companies in industries such as retail develop their club cards and rewards programmes. A few years ago, there was debate within the retail sector about the value of managing rewards programmes and club cards.
So successful has UK retail giant Tesco's Club Card been, however, that it inspired a best-selling book, Scoring Points. The retailer has announced plans for a major investment to relaunch the card in a bid to stave off competition from rivals, including ASDA and Sainsbury's.
Gartner's Herschel feels that data analysis will soon play a central role in informing the strategies of most significant organisations. "An inflection point is now being reached. Smart companies are starting to ask 'what are the wide open spaces we haven't looked at before?'," he says.
Recent developments in the hardware space are also creating new possibilities, especially with the emergence of in-memory analytics. The idea behind this is that all the data resides in the memory of the server, with queries therefore not requiring any disk input/output, a development which promises immense increases in processing speeds. While generating activity among manufacturers, established business intelligence companies, including SAP and Microstrategy, are also making serious investments to add in-memory analytics as part of their offering.
Tracking behaviour online is a key driver for data analytics, with the perceived potential value of tapping into Web 2.0 services pushing up demand for technologies that might tap into and reveal connections within and between "viral networks" such as Facebook, Myspace and Twitter.
Behavioural advertising company Phorm is yet to set a date for the UK launch of its online content gatherer Webwise. The idea behind the application is that regular internet users indicate their various tastes and preferences, which Webwise uses to present the viewer with information about things that might interest them, such as shopping and entertainment. For example, a user who has been looking at a website advertising digital cameras is afterwards presented with adverts from companies selling cameras or other photographic products and services.
The company has come under fire from privacy advocates, but stresses that users can opt in or out. Further, it states that users are assigned a random number and that no IP addresses are stored, while page records are destroyed once links have been made. Phorm is believed to be in discussions to license its technology, with Virgin and BT two names so far mooted.
Of course data analytics is not all about customers. The ability to collect and interpret data within organisations is increasingly viewed as an important tool for understanding worker behaviour and improving productivity.
Following a minor exodus from its senior ranks - including advertising sales boss Tim Armstrong, engineering director Steve Horowitz, and search quality chief Santosh Jayaram - Google recently set about developing an algorithm that would help it predict when someone within the organisation might be planning to leave. Data analytics is becoming a more important tool generally for HR professionals, especially now that many companies are looking to identify their most talented staff for fear of losing them once the economy recovers.
Google is seen as the inventor of what is now referred to as the semantic web, or semweb, which is now the dominant search methodology on the internet. The company's model for linguistic searching revolutionised the internet experience, both for users as well as the companies trying to market to them.
A recent addition to the semweb, US movie distributor Netflix is currently running a competition to improve development of a system designed to make people's movie choices for them based on their past viewing history and other criteria.
While the ideas now being presented by the semantic web are yet to have any serious impact on the corporate world, they may have implications for business in years to come.
Launched amid much fanfare in May, the science and academic search service Wolfram Alpha has been criticised for being overly particular about how questions are worded. Whether or not this criticism is justified, it nevertheless raises a fundamental issue with regard to how organisations think about information.
According to Tim O'Neill, co-founder and director of business solutions provider Avolution - a Gartner "cool vendor" - it is no longer adequate for firms to be merely passive consumers of data. To truly harness its value, they need to be more thoughtful and aggressive in finding out what it is that they need to know.
"Often this comes down to IT executives asking the questions that are normally too painful to ask," he says. "For example, Where do I cut 30% off my budget without affecting my service levels?"
Formed out of the University of Technology in Sydney, Avolution markets a unique 3D-enterprise modelling system called Abacus.
3D visualisation is an important new trend in information management, and Abacus is designed to provide organisations with a deeper understanding of the relationships between the myriad components that comprise their business and the emergent behaviour of those components.
O'Neill says that for companies to achieve this, they need to get better at asking the right questions and better at interpreting their answers. "It is not good enough to just draw a few box and line diagrams and hack together a spreadsheet. You have to look deeply into your organisation and manage it with specific metrics and rich visualisation techniques."