Data Integrity - The Final Frontier

Regular readers of this blog will know that I’ve been forecasting for some time that data integrity will be the next big thing. That’s nothing new. But what’s really interesting is that many of my fellow security professionals are now starting to say the same thing. Data integrity was certainly one of the hottest issues raised at last week’s Infosecurity Europe Advisory Panel. I’ve previously commented that it might take five years for people to respond to this challenge. Hopefully, awareness of the problem space might start to take off during 2010.

Data integrity is the third and arguably the most significant phase of information security. It’s the final frontier to be tackled in contemporary information security, which is based on the three pillars of confidentiality, integrity and availability: a long-standing fusion of three distinct objectives that collectively map out a solution space that still contains many gaps. It’s understandable that people tend to notice the availability and confidentiality aspects of security well before they spot the integrity issue. But the integrity challenge is quietly building up into a dangerous exposure. Bad data undermines business confidence, and in extreme cases it can permanently reduce the value of business services.

So why is data integrity such an issue? Firstly, much of our data is already bad but we don’t advertise that fact. We keep it quiet. In many databases, it’s not unusual to find that up to half the records contain errors of one sort or another. That’s due to a combination of factors, ranging from transcription errors in call centres to the inevitable temptation to re-use old data outside of its original context. On top of that we have a range of network effects that distort incoming data through Chinese whispers, rumour, spin or good old fear, uncertainty and doubt. There’s a tendency to believe anything that you hear from several different sources. In large networks, that can be deadly. But the most disturbing concern is the threat of an unauthorised intruder deliberately changing data to cause harm, whether for financial gain, spite or sabotage.

The starting point in addressing this relatively new problem space is to recognise that we need standards to assure customers, citizens and other stakeholders of the quality of the information in our databases. It’s quite outrageous that none exist for services that can have a major impact on people’s lives. A single percentage of error in a national database can represent a population the size of a major city. That demands scrutiny. Once we can see the size of the current exposure, there’s no doubt that society and the media will demand action. But until that happens we’re sitting on a ticking time-bomb that’s just waiting to explode.