Splunk .conf2013: a data management turning point?

With the release of the latest version of BI specialist Splunk’s platform, has the data management market hit a turning point? Nick Booth visits Splunk’s .conf2013 to find out

It’s often said that IT workers are better at talking to machines than they are to humans. Is that a cliché because it’s true, or because the people who say it are jealous? Who knows?

Perhaps we should turn that criticism on its head, now that machines are taking over the world. With M2M (machine to machine) communications taking more and more decisions for us, maybe it’s time we all learned to speak to the hardware. You can trust a machine. Machines know exactly what’s going on in your business and – unlike humans - they don’t lie. I’ve never known a machine to say that it had a trial for Man United, or to mis-sell me a mortgage or make any kind of exaggerated claim. I’m starting to see the attractions of shooting the breeze with a piece of hardware. If only I could understand their syntax.

The thing is, metadata is an unimaginably complex language to learn. If you haven’t learned to speak Chinese yet, don’t even think about trying to talk to a machine. They won’t tell you a thing without a struggle. For example, I learned at the recent Splunk conference, .conf2013, that there are 550 types of tags alone. Then there are all the log files to delve into - from firewalls, HTTP web proxies, DNS servers - the list is endless. It’s like getting information out of a stone. Unless you enlist the services of a company called Tealium, you are doomed to spend half your day, hunched over a book about Java commands, which is no way to spend your days if you are a freelance writer, a marketing hack or a data scientist.

Splunk is the company that wants to make it easier for us to talk to machines - sort of a Digital Dr Dolittle.

It’s no coincidence that Splunk, the de facto leader in searching machines, is named after the practice of dropping yourself deep into chasms in the earth’s infrastructure: spelunking being the American word for potholing.

In a way, Splunk is the Google of big machine data search. In another way, it’s the Microsoft, because it has just launched version six of its platform for searching machines, in a move that historians will one day liken to the launch of Windows for the PC. In terms of cultural significance, this event is like a cross between the Sex Pistols’ first gig, the Iranian Embassy siege and the London Olympics. All kinds of blowhards and bluffers will pretend they were there. But they weren’t.

I was though. Here are some people who were there way ahead of me who actually know what they’re doing. They helpfully explained how it works, how they’re making a fortune and how Splunk 6 could help you emulate them.

Meet the users

Fortscale writes algorithms that discover the stuff that goes under the radar of a security desk. Often what looks like legitimate user behavior is some devious act by a cyber-scammer. Fortscale’s ability to establish what behaviour is normal, and what isn’t, is created by its complete mastery of all the afore-mentioned logs from firewalls, web proxies and DNS servers, among other things. It’s a new company and looking for UK partners, according to Idan Tendler, the CEO and founder.

Similarly, the Splunk app used by FireEye takes all the machine data from the usual networking devices and identifies patterns of suspicious data. It uses all the alert data to automate the forensic examination that would take 10 times longer if it was carried out by humans. Jamie Andrews, the UK channel manager, probably wants to talk to you. It would make a change from a machine.

There are all kinds of starts ups expanding into this area and looking for UK partners. Prelert, Cloudmeter, Pentaho, ExtraHop, Securonix, VelloInetco, NetFlow Logic – the list is endless. Unless you’re talking about UK start-ups, then it’s a very short list.

Palo Alto Networks uses Splunk to talk some sense into virtualised datacentres, where there’s been a breakdown in communications between virtual machines and firewalls. By pulling machine data and inserting agents, the service provider can restore order between the web, database and app servers, who threaten to run riot otherwise.

Talking of losing control, companies like GrandSLA and Wipro use Splunk to talk to machines and get the real story about the service level agreements offered by Amazon Web Services and other cloud providers. One of the great unreported scandals of the cloud industry is that most customers are misled over their true levels of service. Not deliberately, it should be pointed out, but because Amazon tells you what the infrastructure is doing (it offers infrastructure as a service) but your SLAs should be a reflection of how the applications perform. But that comes from a completely different data set.

Now that Splunk is easy to use, it’s not your techie-ness that makes you stand out, it’s your specialist knowledge. Splunk’s director of business development, Andrew Morris, seems more interested in vertical market knowledge. “We want people with domain knowledge,” says Morris. Companies like Intermap, who know the challenges faced by universities, are more interesting than machine nerds.

Some of the early adopters of Splunk have made a fortune. If you have Splunk skills, you are the person everyone goes to at work to find out how to run data analysis. Equally channel partners are doing pretty well out of it. “It’s like being the first person in the company to know Google,” says Kevin Tunsley, operations director for Eqalis, “we’ve grown on the back of Splunk. Once people see what it can do, someone else in different department will want you to carry out a project for them.”

Now that Splunk’s easier to use, there might be a lot more competition though. A firewall told me that – it heard it on the network.

Read more on Business Intelligence Tools and Services