Big data technology has its work cut out to harness web analytics

English: eBay Logo

English: eBay Logo (Photo credit: Wikipedia)

What can we learn from companies such as eBay and Amazon? These internet businesses are at the cutting edge of technology.

The recent Gartner CRM summit gave delegates an understanding of what CRM means to a web-only retailer. The processing eBay conducts to understand customers better, for example, is eye-watering. The web gives retailers incredible insights into customer service. It is not only possible to track a customer’s identity but, thanks to smart web analytics, eBay can follow the buyer’s journey.

David Stephenson, head of global business analytics at eBay, says it’s a bit like strapping a video camera to a customer’s head. Recording every interaction a customer makes means the auction site collects millions of hours of web analytics. Making sense of it all is a big data problem. In fact, eBay produces 50TB of machine-generated data daily. It also needs to process 100PB of data every day to understand what its customers are doing. Sampling this data may have worked in the past, but this only gives a statistical snapshot.

In the era of customer focus, eBay strives to collect and analyse all the data it collects. With this information, Stephenson believes eBay can offer its customers intuitive, almost intelligent, recommendations. The technology supporting the web analytics eBay undertakes does not come cheap. Nor is it available off the shelf. There is no such thing as a “big data solution” for the level of data processing eBay shoulders

The company needs to work with suppliers to build bespoke hardware and software for its requirements, because using a traditional data warehouse would be too slow and prohibitively expensive to scale. But even a custom data processing engine cannot comprise the whole answer.

The firm uses three systems: a traditional data warehouse appliance, a NoSQL database and the custom appliance to analyse its customers’ journeys. So while it makes perfect sense for businesses of all sizes to use web analytics to understand customer interaction, an immense amount of technical investment and expertise is required to do so effectively.

Enhanced by Zemanta