The fight against terrorism justified a massive invasion of privacy by government, says the architect of the government's national security strategy.
But this was "greatly preferable to tinkering with the rule of law, or derogating from fundamental human rights," said David Omand, the Cabinet Office's former security and intelligence co-ordinator, in a research paper to the Institute of Public Policy Research, quoted by the Independent.
Omand said the government needed access to personal details in order to build profiles of terror suspects by data mining. However, reports from the US suggest such profiling is only partially successful at best.
The data included information on airline bookings and other travel data, passport and biometric data, immigration, identity and border records, criminal records and other government and private sector data, including financial and telephone and other communications records.
Omand said some of this information was held offshore in other jurisdictions, but that the government would require access to it.
Governments are already collecting and sharing much of this information through bilateral and multilateral agreements covering passenger name records, visa applications and border surveillance systems, among others.
"Access to such information, and in some cases to the ability to apply data mining and pattern recognition software to databases, might well be the key to effective pre-emption in future terrorist cases," Omand said.
Omand's paper outlined plans to track terror suspects through a state database that would also contain the details of innocent people.
He said modern intelligence access often involved intrusive methods of surveillance and investigation that might have to be at the expense of some aspects of privacy rights. "This is a hard choice, and goes against current calls to curb the so-called surveillance society," he said.
Governments had to be able to convince people that this was in their interests, he said. The intelligence community might have to show it complied with "proper legal authorisation and appropriate oversight of the use of such intrusive intelligence activity".