James Thew - Fotolia

Big data projects need business input and careful management

Big data projects need to be managed. Ocado and Graydon have been figuring out how

For around £80 per month, businesses can now gain access to some of the most powerful software ever developed. It is about the same price as a high-end home broadband package.

This is what Ocado pays for its artificial intelligence natural language processing application programming interface (API) from Google to help manage and prioritise the 2,000 emails it receives from customers every day.

The significance is not only the low cost – it is also transparent, and helps business leaders understand the cost and value of their big data projects, says Dan Nelson, head of data at Ocado Technology, the IT division of the online grocery store.

“You could not get access to this kind of technology just a few years ago. But also, because we know the cost, it helps the business decide how much time it wants to invest in new data projects,” he says.

Engaging the business in these priorities is important, says Nelson.

Data scientists now have a mass of tools to analyse huge volumes of data arriving from myriad sources at phenomenal speed. Email, social media, the internet of things (IoT) and smartphones are just of few of the sources businesses can draw on to reach useful conclusions and improve their performance. But executing a successful big data strategy is not just a technical problem.

Approach big data from a business angle

Ocado Technology has developed a model for data projects, which sees product owners in the business lead projects with the help of development teams and data scientists. Among them is a natural language project to analyse customer emails to ensure customer service agents are not overrun at peak times (see the Ocado case study below).

Trading since 2002, Ocado strives to steal a march on competitors by building much of its technology in-house. However, it is not only businesses born in the dotcom era that are demonstrating the importance of a thoughtful management structure to data projects.

Graydon, a credit reference and company information provider, has a history dating back to 1862. For the past two years it has been working on creating a central approach to business intelligence (BI) which will lay the foundation for its adoption of a big data approach to providing solutions to its clients.

Bart Redder, director of customer and business intelligence at Graydon, says this team moved out of the marketing department to become a shared cross-group activity so it could work more closely with all stakeholders, including IT, finance, operations and commercial departments.

Gain value from data insight

To ensure equal service to all these parties from a coherent set of data, Graydon developed a value chain of data insight model.

“We notice that you need different departments, different sets of data and different business rules if you want to generate insight,” he says.

To bring these together, Redder and his team identified a six-stage value chain of data. It starts by looking at the data sources. Second, it asks if the input of data has been done correctly. Then it looks at data processing. Fourth, it considers data transformation. The fifth step is to build up dashboards. And finally, it looks at how those insights trigger action in the business. The team calls data governance meetings where the business owner for each stage discusses progress and problems.

Redders says the approach helps the BI team and the business understand where the problems are, and where resources are needed, right from raw data to an insight and business action.

Graydon has implemented Birst, a cloud-based BI platform, to bring together information sources across the company and provide self-service analytics dashboards to teams in marketing, sales, human resources and finance.

“To succeed in big data, you will have to have financial data and all the other processes correct and in line before you start”
Bart Redder, Graydon

The platform is currently used to analyse Graydon’s internal performance and processes, but will be extended to the data and analytics it provides its customers. It is also experimenting with providing big data analytics to its customers, for example by assessing whether restaurant reviews can provide early indications of these businesses struggling financially.

However, Redders warns that strong management of structured data projects is necessary before taking advantage of unstructured sources.

“If you want to succeed in big data, you will have to have financial data and all the other processes correct and in line before you start,” he says.

Bridge the skills gap

While big data is helping companies in established markets, it is also opening up new business models. Open Energi is taking advantage of smart meters measuring the electricity demands of buildings, factories, machines and fridges to help its customers understand their energy consumption and sell back unused resource to energy providers. The model, known as demand-side response, relies on effective analysis of big data from thousands of meters sampling energy consumption every few minutes. 

Once the company reached a certain scale, it realised it needed a big data platform, which it found in the Hortonworks Hadoop distribution. However, it still needed to build a team that could work together, says Open Energi technical director Michael Bironneau.

“When we started out, our skills were on the data science side. We underwent training and figured out how to use the new technology. We made a good start, but when we ended up hiring software engineers we found we needed to bridge the gap between the two groups,” he says.

“Any company that has never looked at big data and is considering a project in that area should hire a data engineer”
Michael Bironneau, Open Energi

“That was a big challenge. It was like they spoke different languages. A data scientist would talk about a machine learning model that runs in real time, when a software engineer would ask for the component design. Data scientists care about accuracy and the elegance of the model, whereas software engineers want to build something that does not break,” he says.

Bironneau initially worked as a translator between the two roles, but the two disciplines are now more aligned and able to understand each other’s priorities.

He also recommends the recruitment of a data engineer to bridge the gap between the two fields. “Any company that has never looked at big data and is considering a project in that area should hire a data engineer. That was the catalyst for us, and it has turned out to be a very important role.”

Get management and governance right

Ray Eitel-Porter, leader for Accenture’s analytics practice in the UK and Ireland, says that as businesses move big data projects from a pilot phase into production, they need to get management and governance right.

In terms of organisational structure, he recommends a hub-and-spoke model where a central team takes responsibility for tools, data quality and definitions, while satellite data scientists are placed within the business units to understand how they can benefit from big data and analytics.

This also ensures the activity is jointly funded, he says. “It is critical to have business sponsorship. If you have all the funding centralised, you might not hit the business need. Some business ownership and funding is essential. But you also want to do cross-enterprise investment in a Hadoop platform and build a data lake.  You want central funding earmarked for certain things that are needed across the enterprise.”

Businesses can be easily dazzled by big data, both in terms of the impressive technologies cloud computing makes accessible and affordable, and in terms of innovative use cases. But before organisations can reap the rewards new sources of data can offer, they need to ensure they have the right management structure, funding and data governance in place. Otherwise, the big data concept is unlikely to live up to its hype.

Case study: Ocado brings data projects into the business

Data scientists are not always the best leaders of data projects, Ocado Technology is learning. The IT division of the online grocery retailer and supermarket is striving to help the business take advantage of data with a 60-strong central team, but they do not lead projects.

“We have resources at our fingertips but try not to do data science projects. If you do, you end up with something very clever that is difficult to implement or nobody feels the ownership to support,” says Ocado Technology head of data Dan Nelson.

Instead, it is placing leadership for data projects within the business teams, in the form of a product owner, whose task it is to identify business problems that could be solved with data analysis.

A recent example saw Ocado take advantage of Google natural language processing artificial intelligence APIs to help understand and prioritise customer service emails, particularly when volumes are exceptionally high.

Using the head of service delivery as business sponsor, the project was managed by the product owner of the contact centre working closely with the central data science team and development team that would be implementing the technology.

This structure helped data scientists understand governance problems in terms of customer privacy, without removing all the value from the data. Development teams, meanwhile, were able to guide the project from its beginning to ensure it could be rolled out quickly in a production environment once the technology was ready.

Support for data science goes to the top of the organisation, though. Nelson says his department’s priorities are set by a business steering committee, chaired by the CEO.

Read more on Big data analytics