A report from Splunk Live 2015: the real world use of machine data

Given the chance to address customers, partners, staff and the media en masse, any company likes to lay out its vision. This was certainly true when Splunk’s CEO Godfrey Sullivan spoke to an audience of almost 600 at Splunk Live in the London on May 13th 2015. Vision is all well and good, but only if it chimes with the problems faced by customers and prospects. In a well-orchestrated event, there was plenty of evidence that Splunk’s customers endorsed and benefited from initiatives being undertaken by the vendor.


In a nutshell, Splunk turns all the data churned out by computing infrastructure, applications and security systems into operational intelligence, aiding both IT and security management. The volume of this machine data has increased a lot as infrastructure been extended to include cloud services; the numbers of users has increased as online applications are opened up to outsiders; the layers of security have increased and the internet-of-things has taken shape. Splunk says it has moved from the static review of machine data to dynamic big data analytics; i.e. more insight from more data with the capability to respond in real time.


As the landscape Splunk is collecting data from has changed the tools it provides need to evolve too. Two initiatives help with this. First, along with many others in the industry, Splunk has moved to DevOps, enabling agile development and making new features available as soon as possible. This applies to its core Splunk Enterprise product and is native to the way the new Splunk Cloud service is delivered. Splunk has also extended its reach, with a Splunk Light for smaller businesses and Hunk, which enables its tools to be used directly against Hadoop big data clusters. Second, it encourages customers to develop and share their own applications, testing and certifying the most popular ones for download from its app store.


So, what do Splunk’s customers think? The vendor is not shy to talk about them; big European names came up in presentations again and again including John Lewis, Tesco, the NHS, Sky and VW, the last of these using Splunk to help manage its connected cars program, a true internet-of-things challenge. Three customers presented during the morning sessions, with more time being given to them than Splunk’s own spokespeople. Their testaments underlined the reality of the vision outlined by the CEO.


First up was Paddy Power; a familiar name in the UK and to many who gamble online, a service provider. It has all the IT challenges of a 21st Century online company; thousands of virtual machines, a mainly mobile customer base and huge spikes in demand, for example up to 12,000 bets to process per minute during the recent Grand National steeplechase. Splunk helps address all sorts of worries about performance and security. Perhaps, most interesting was Paddy Power’s approach to development, agile DevOps, mirroring Splunk’s own need to get new innovations to customers as soon as possible. The company’s use of Splunk was initially in the area of security, but now it is providing business insight to senior execs via smartphones though analysis of machine data.


Next was Ticketmaster, another quintessential online operator, with thousands of virtual machines and over a quarter of a billion registered users. It experiences huge peaks in demand when tickets for popular events first become available; sales can top $1M a minute! Application failure is expensive and unacceptable. In Ticketmaster’s own words “life was not pleasant before Splunk!” Initially Splunk was used for incident investigation, forensics, security/compliance reporting and monitoring known threats. In line with Splunk’s own vision Ticketmaster has moved on to real time advanced threat and fraud detection and monitoring the insider threat.


Finally was CERT-EU, not an end user organisation but one providing security intelligence to a community of sixty plus opt-in European Union institutions. Here, in partnership with a range of IT security vendors, including Splunk, CERT-EU monitors threats in real time across all its members and is therefore able to provide much broader protection than any individual organisation could do for itself. Whilst this includes crime detection, nation state and terrorist activity are now an ever present threat for government bodies and a target of CERT-EU’s monitoring.


In 2014 Quocirca worked with Splunk to get a better idea of the extent to which EU-based businesses recognised the value of machine data and were able to collect and analyse it. The results were published in a free report Master’s of Machines. A new report, to be published in June 2015, will look at how similar businesses are using operational intelligence derived from machine data to manage IT complexity, improve the cross-channel customer experience (or omni-channel as some call it) and tighten security. Some are as advanced as Paddy Power, Ticketmaster and CERT-EU, but the research shows that for the majority machine data is a free resource that they are yet to fully exploit.