Tidal wave of data is heading our way

Brace yourselves, this is going to get rough. The coming tidal wave of data is likely to sweep away old perceptions and practices, leaving many of us floundering...

Brace yourselves, this is going to get rough. The coming tidal wave of data is likely to sweep away old perceptions and practices, leaving many of us floundering, wondering what happened, and profoundly disconnected in a connected world.

This is the somewhat apocalyptic vision of Stephen Brobst, CTO of data warehousing firm Teradata. You may think he would say that, wouldn't he? There is nothing like a little fear, uncertainty and doubt to boost sales of Teradata's systems for sifting and sorting petabytes of data.

But for all his enthusiasm for what changes are in the offing, one gets the sense that even Brobst is watching the oncoming wave with some trepidation.

Why should that be? Simply, our ability to make sense of the data is falling behind our ability to generate and collect it. As we connect more things to networks, the quantity of data out there is going to grow by orders of magnitude, Brobst believes.

If it moves, or if it changes state over time, it will generate data traffic, he says. "If you looking a little into the future, every moving device - vehicle, plane, human - there will be sensor devices on all of them," he says.

Darryl McDonald, Teradata's chief operating officer, uses an example from health care. He says don't be surprised if one day soon you are walking down the street and an ambulance stops next to you and the paramedics say, "Hop in, the monitoring system thinks you are about to have a heart attack."

Shift to detail data

Brobst says we are about to shift from using summary data, which allows us to answers question that we suspect we know the answers to, to detail data, which will allow us to answer questions to which we really don't know the answers. This is the well-spring of innovation, he says.

Until now we have been using data about transactions to analyse behaviour and make predictions about it. "The next level is to go to interactions. With interactions we say 'I want to understand not just the transaction, but everything that led up to the transaction, and the customer experience in receiving that transaction or product," he says.

Brobst cites the mobile network's call data record (CDR) as an example. The CDR is a summary record that allows the network to bill you for the call and establishes your value to the network. But to understand the customer's experience of the call means you have to understand the data at the level below the CDR, he says. This is all the information associated with the call, such as where and when the call was made, whether you moved from one cell tower to another, whether you changed providers in the middle of the call, what the congestion was on the network, what type of traffic was on the call, and so on.

"This is orders of magnitude bigger data - absolutely huge," he says.

As one journalist told Brobst, "I feel like I'm already living in Gattaca," the movie where society was ordered by one's pre-planned genetic make-up, and every person's behaviour monitored and "anomalies" removed.

"You would be absolutely correct," Brobst says.

That sharpens the horns of the social dilemma. Is it good that we will be able to save more lives, or is it bad that even the most intimate moments of our lives will be monitored, and the data collected, stored, analysed, aggregated and shared with people we don't know?

Socialisation of data

This "socialisation of data" is the big theme of Teradata's Partners conference this week in San Diego. And it is an anomaly, a step out of character for an IT company. But it is not the first, nor is it the only one that is starting to worry about the implications of what IT enables.

The relationship between society and technology has always been fraught. The ability to control and apply technology has always lent power to he who has it. So what is the socially acceptable role for this truly awesome ability to collect, analyse and act on data?

We know the downside. World War Two records showed the use of systems to collect and process information that helped to manage the mass murder of Jews and others deemed unfit by the Nazis.

Because of that, it should be no surprise that the European data protection supervisor has deep reservations about the collection and subsequent processing of airline passenger information by governments, led by the US. The safeguards in the agreements are too weak, he believes.

This obsessive collection of data appears to be pervasive. Google, whose motto is "Don't be evil", is being shown to have done "evil" by secretly collecting user names and passwords of private Wi-Fi networks as it collected images for Streetview. And if it didn't lie about it, Google was certainly economical with the truth about its activity. The mystery is why Google collected the data in the first place.

The UK government appears to have forgotten pre-election promises to roll back the database state. It may have scrapped the ID card project, but it appears set on reviving the Interception Modernisation Programme (IMP) to monitor and collect all electronic messages that start, end or transit the UK. The £12bn-plus IMP will cost might be better spent creating jobs for disaffected Muslim youths.

Are we destined for an Orwellian or Minority Report-like society? No doubt some of their characteristics will be part of it. The improvements in efficiency and convenience will make it so.

But throughout history all oppressive regimes have generated the seeds of their own destruction. The music industry's campaign against file sharers attracted denial of service attacks against their websites and those of their agents. Wikileaks publishes hundreds of thousands of electronic records that document a callousness little expected by some in the US military in Iraq. Alberto Gonzales turns his hacking talents from serving the state to serving himself, and goes to prison for it.

Brobst says, "Transparency drives better behaviour." But behind that assertion is a concern that perhaps the balance is tilting, where there is too much data, where irresponsible or unaccountable access to it is too free.

Should we continue to rely on the mavericks, the anarchists and the whistleblowers to keep the authorities honest? Perhaps not.

Read more on Data centre hardware