As organisations accumulate increasing volumes and diverse types of information, known in the IT industry as big data, they are being presented with a major new commercial opportunity to gain insight into their customers and identify the patterns that will help them predict future consumer behaviour.
To make the most of these opportunities business leaders need to focus on three key areas: managing the volume of data; accessing the commercial insights; and securing the right skills.
Managing the explosion of data
Data volumes are increasing dramatically. Just over a decade ago, only a small number of organisations had data warehouses that reached or exceeded a terabyte in size. Today, however, companies like the internet businesses Netflix and Facebook have data warehouses that have already exceeded a petabyte of data and are continuing to grow fast.
The types of data being stored have also increased in complexity. We have moved from the simple transactional data of the 1990s, held in a structured way within organisations’ own systems, to far more diverse data sets such as video, text, voice and email that rests within, outside, and between the organisation and public domain. As this data increases in scale, organisations will need to turn more to cloud-based technologies to handle not only the space required to store the data, but to gain access to far more powerful analytical capabilities. Providers are responding, for example Google is building a service to allow the analysis of large amounts of data in the cloud. The service, called BigQuery, would help organisations analyse their data without the need to build new IT infrastructure.
Accessing the commercial insights from the data
Earlier this century, most organisations’ data analysis would have been limited to the simple historical reporting of what was sold, by whom, how often and where. There is now a growing need for organisations to perform far greater and faster data analytics on their data sets. The goal is to not only gain insight into how people use their products but, more importantly, to start to predict what may happen next. However, to realise this commercial advantage, organisations need to make sure they can access the data in an efficient way from their limited IT assets.
More from Computer Weekly on big data
The approach will vary by organisation. For example there are data warehousing appliances, which are purpose-built, hardware-software solutions; massively parallel processing (MPP) databases running on commodity servers; columnar databases; and distributed file systems running MapReduce and other non-SQL types of data processing languages. Many companies are now rethinking traditional approaches to performing analytics. Instead of downloading data to local desktops or servers, they are running complex analytics in the database management system itself. This so-called in-database analytics minimises or eliminates data movement, improves query performance, and optimises model accuracy by enabling analytics to run against all data at a detailed level, instead of against samples or summaries.
The ever-diminishing pool of skilled resource
While technology and data volumes are expanding at a substantial rate, research has highlighted the shortage of workers with the deep analytical skills to handle that data. Finding the right people who can undertake analysis within the commercial context of an organisation, especially at the right price, could prove the biggest barrier to organisations benefiting from their big data.
This is a challenge due to both the jump in demand for these skills but also because the skill set needed to exploit big data has several components that need to be applied in a balanced way. First, individuals require the technical skills to find patterns and draw conclusions from the data. Second, they need to use the conclusions to draw a set of hypotheses that are linked to the commercial drivers, and last they need to be able to present these to senior executives in a compelling and realistic way.
Executives should be thinking about their critical appointments now so they have the people in place to help jump-start or improve the success of their planned big-data initiatives. They also need to recognise that the individuals will require investment, nurture, and some patience, to get results.
There is a real commercial advantage to be gained from big data and organisations now need to make sure they have the right elements in place to seize that advantage.
Stephen Gallagher is an expert in big data at PA Consulting Group. For more information visit www.paconsulting.com/smart
This was first published in April 2012