NRC study questions Big Brother profiling database

Data mining has limited value in the search for terrorists, an influential US government research organisation has found.

Data mining has limited value in the search for terrorists, an influential US government research organisation has found.

After an exhaustive study, the National Research Council concluded: "Identifying terrorists from the huge flow of internet traffic was extremely difficult."

The National Research Council (NRC) said the terrorist threat to the US was real and serious. However the NRC questioned whether it warranted the corrosion of civil liberties posed by counter-terrorism methods. "The threat does not justify government activities that violate the law, or fundamental changes in the level of privacy protection to which Americans are entitled," it said.

"Each time a person makes a telephone call, uses a credit card, pays taxes or takes a trip, he or she leaves digital tracks, records that often end up in massive corporate or government databases," the council said.

"Through formal or informal agreements, government has access to much of the data owned by private sector companies. Agencies use sophisticated techniques to mine some of these databases, searching for information on particular suspects and looking for unusual patterns of activity that may indicate a terrorist network. But successfully identifying signs of terrorist activity in these masses of data is extremely difficult," the council said.

This assessment came a week after it was revealed the UK government had given GCHQ, its electronic surveillance unit, £1bn for a pilot project to develop a centralised database of all electronic communications, including internet phone calls and searches. The project, the Interception Modernisation Programme (IMP), could cost as much as £12bn, reports say.

The project would depend on the development of technology that would copy all communications traffic from telecommunications suppliers and internet service providers.

The technique, known as profiling, is widely used by retail and marketing companies as well as law enforcement agencies.

Profiling is also used to detect fraudsters. The Insurance Fraud Bureau has successfully used network detection and profiling software from UK software house Detica to reveal gangs that rip off insurers.

The NRC called for every electronic surveillance project to be evaluated before, during and after the project to determine its cost effectiveness. It also called for innocent people misidentified through a data mining project "to be given a meaningful form of redress".

The NRC's study, "Protecting individual privacy in the struggle against terrorists", was sponsored by the US Department of Homeland Security and the National Science Foundation, among others.

Read more on IT governance