Government plans to monitor suspect communications are doomed to failure because of the difficulty and cost of analysing the data from different databases, a database technology expert said today.
The Home Office yesterday published proposals to have communications services providers (CSPs) collect information about electronic communications between people so that public authorities such as police and national security agencies could monitor their traffic.
Stephen Brobst, Teradata's chief technical officer, said the government's plan appears to involve "federating" databases across networks.
"Federation is death," he said. This was because of the difficulty in managing different types of data from dissimilar databases, and then trying to analyse all the data across a network. "The cost is just too high," he said.
"You can do federated data analytics with multiple online transaction processing databases if you have a very aggressively limited subset of data," he said. But data mining to fish out likely suspects from a wealth of data from different sources was nearly impossible, he said.
This view was corroborated by a spokesman for Detica, the BAE Systems firm that has 30 years' experience in data mining, and which has close links with law enforcement and government security agencies.
The head of Detica's technical consulting practice, Anthony Golledge, said, "There is now so much data out there that brute force attempts to identify individuals are unlikely to succeed."
The chief technology officer for a company that specialises on intercept technology for law enforcement and national security agencies, agreed. He added that internet traffic volumes were doubling each year while hardware technology was doubling every 18 months.
Hardware limits meant that law enforcement officials would be able to look at only a small fraction of the data in a short space of time, he said. It was infeasible to collect, store and search all the data passing through the UK in realtime, he said.
He said a study of a large German internet service provider showed that the average traffic through the site was 5Gbps. He said it would take 70 "probes" running at 10Gbps to monitor all the traffic in the site. "And that's just for one site," he said. "What about all the other hubs?"
The CTO, who requested anonymity because of his clients' sensitivities, said Germany was now debating the value of allowing law enforcement officials to get court permission to install secret Trojan software, the BundesTrojan, on suspects' PCs.
"This may be a better solution for the police and citizens because it is targeted, so it doesn't invade everyone's privacy, and it doesn't cost so much," he said.
The government launched a consultation on its proposed £2bn Interception Modernisation Programme yesterday. The plan aims to give the police and security agencies the ability to intercept suspects' communications over networks that use the internet protocol (IP).
The Home Office has called for legislation to allow all data that public authorities might need, including third party data that passes through the UK to be collected and retained by CSPs.
It wants CSPs to process annd store the data so that specific requests for the data from public authorities are answered quickly and comprehensively.
It said it had given up its original plan, rumoured to have a £12bn budget, to have a central "Big Brother" database of all communications data.
This included the who, what, where, and when calls were made, but not the content. CSPs collect this data normally to bill customers, and last month were forced to keep it for 12 months under the UK implementation of the European Data Retention Directive.
However, Computer Weekly discovered that apparently not all internet service providers had to comply with it. This was partly to prevent double counting, and partly to save the government from paying lots in compensation for collecting the data. The Home Office refused to provide Computer Weekly with a list who was eligible for compensation, saying it was commercially confidential.
The internet protocol was designed to allow alternate routing to get messages through even if the most direct path to the receiver was broken. This means it is very difficult to tap phone calls and messages between suspects on IP networks.
But as nearly all network operators are converting their systems to IP, law enforcement officers were losing the ability to monitor suspects' communications traffic.