Behind Closed Doors:Valuable data or just plain sludge?

We IT professionals need to start asking the right questions about data. Just what is worth capturing and how do we keep our...

We IT professionals need to start asking the right questions about data. Just what is worth capturing and how do we keep our databases in line with the law? Colin Beveridge reckons it's time to start a spring clean.

I don't know why the local council bothers with street names and signs anymore - everybody seems to know my full address simply by combining my postcode with my house number. Forget the quaint old forms of address, such as The Farthings, Penny Hassett by Ambridge, Royal Berkshire - I am now plain old Mister HG12 9PX number 14.

I am seriously thinking about getting this vital combination tattooed in my left armpit, probably as a barcode, so shop assistants can just scan me without having to ask me anything.

Yes, all thanks to the modern wonders of information technology, our high-street shops and businesses can instantly determine a full postal address from a few significant characters, thus saving themselves time in recording details while also improving the accuracy of their sales data.

It's dead easy, when you have an online database of postal addresses that even a five-year old can operate.

But have you ever stopped to think why this personal information is being gathered quite so frequently and quite so pervasively?

After all, is it so important for a major retail chain to know who I am, where I live and how often I buy a four pack of AAA batteries? Where is the inherent value in the data?

Too much information
For sure, I can understand that the retailers, distributors and manufacturers all need to know plenty about volume of sales and geographic distribution. But do we need to record absolutely everything?

"Is it so important for a major retail chain to know who I am, where I live and how often I buy a four pack of AAA batteries?"
Colin Beveridge
it's getting harder and harder to buy even the most trivial item these days without being expected to hand over our full personal credentials, to be stored by the point of sales system.

I dread to think what eventually happens to all of the data that is captured so eagerly.

I suspect that most of it simply ends up as digital "sludge" clogging up the storage system in a data centre somewhere and being backed up each night, just in case it is ever needed - which it won't be.

And yet somebody, somewhere must have thought that all this "sludge" was worth capturing in the first place and of sufficient value to spend money and effort building a suitable computer system to process and store the information.

Which is where we come in, the IT professionals. Like it or not, we are supposed to be the guardians of technology, charged with maintaining the quality and integrity of our various information systems.

We are the ones who should be keeping an eye on the nature of absolutely everything that goes into our systems, not just making sure that the systems themselves are up and running to the appropriate service level agreement.

The simple questions are not always asked when implementing new systems. No-brainers such as why are we capturing this information, what are we going to do with it, are we allowed to keep it anyway, and for how long?

Sadly, in my experience, this essential gate-keeping obligation is not always adequately addressed by companies and organisations - all too often the IT function simply translates the business requirements into technology, without first challenging or checking the specification properly against the prevailing compliance regime.

I have already pointed out that our current obsession with data capture has gravely affected our storage liabilities, creating sometimes huge pools of expensive, but worthless, sludge in our data storage systems.

Storing up trouble
Perhaps the more serious issue we should consider, however, is whether or not the contents of our databases are completely legal because I suspect that there are hundreds and hundreds of illegal databases out there.

Let me correct that last statement. I don't just suspect, I know, which is probably why recent reports have indicated that many businesses remain blissfully ignorant of their obligations under the Data Protection Act, let alone the plethora of other laws and regulations that govern modern data processing operations.

I can't think of a single computerised business that can afford to ignore the complex hierarchy of rules but I constantly find examples of non-compliance in everyday life.

Here's an example. My car needed some minor bodywork repairs recently, putting it off the road for a few days. My motor insurance policy provides me with a substitute car in such circumstances so the insurer arranged with a local branch of a national car hire firm to rent me a vehicle for the duration of the repair.

I duly went along to the car rental company with my policy details and driving licence, expecting the transaction to be conducted fairly simply as the insurer was paying for it directly. Oh, how naïve I was.

Right to reply?
Nothing is that simple anymore - at least not where the computers have taken over. Not only was I expected to provide my basic credentials and a valid credit card to secure the cost of petrol, I was also obliged, apparently, to furnish the name, address and contact details of both my current employer and another "friend or relative, not resident at my address, who would vouch for me".

The rental agent was adamant that all of these details were required from me as essential input, without which "the system" would not allow him to rent me a car.

Here was a classic example of an illegal, non-compliant system in operation - a system designed to capture and store details that could not be legally given by me, let alone processed or stored by the rental business.

Was I just being bolshy by refusing to give this computer system the name, address and telephone number of a friend, or was I justifiably protecting their data privacy rights?

Either way, it took an argument to proceed with the minimum of essential data being provided by me. And yet, miraculously, the system did eventually allow the rental without the contentious items.

Of course, the offending system did not write itself - it was the product of a software developer, possibly aided and abetted by a systems designer. So how did such a poor system ever get out into the world when it was so plainly breaking the data laws?

Obviously, the so-called information professionals who built the software were either unaware of, or chose to disregard, their data compliance obligations.

Keep it clean
My message is quite straightforward: as information technology professionals, we all owe a fundamental duty of care to our colleagues and customers. Firstly that we don't accumulate great steaming piles of useless, "sludge" data that clogs up our valuable corporate infrastructure. Secondly, to ensure that all systems are fully compliant with all of the appropriate legislative, regulatory and commercial obligations.

As a result we will all have lean and legal databases and I should be able to buy an ice-lolly without disclosing both my inside leg measurement and my mother's maiden name.

What's your experience?
Are your systems trapped in the sludge? What's the best way to keep databases legal and sludge-free? Let us know with an e-mail >> reserves the right to edit and publish answers on the Web site. Please state if your answer is not for publication.

Colin Beveridge is an interim executive who has held top-level roles in IT strategy, development services and support. His travels along the blue-chip highway have taken him to a clutch of leading corporations, including Shell, BP, ICI, DHL and Powergen.

Read more on Privacy and data protection

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.