The Home Office is considering radical plans to develop a centralised surveillance system to track in real-time every kind of electronic activity undertaken by citizens.
The project, driven by intelligence services, would require the development of a surveillance system unprecedented in its scope and technical sophistication.
The work is still at the discussion stage and has not been agreed by ministers. But if the project goes ahead as expected, it would require the development of untried technology to tap into phone lines and the internet, retrieve details on every individual's browsing and communications traffic, and store it in a central database.
The envisaged database would not record the content of telephone calls, e-mails or other internet messages. However, it could hold records of telephone and interent traffic data, which would enable investigators to build up a proile of an individual and identify their network of contacts.
The information gathered, for example, could include the time an individual sent an e-mail or instant message, and who received it. It could also record details of websites visited by members of the public, and even who had used which online computer game or video clip, when and for how long.
The project represents a major esclation in the government's powers and the speed at which electronic surveillance can be undertaken. Under existing legislation, telcos are required to hand over limited data to public authorities holding a relevant notice under the Regulation of Investigatory Powers Act (RIPA).
This data, which is held for specific lengths of time, includes subscriber details, logs of phone calls, text messages, e-mails and when users have logged on and off the internet . ISPs currently only have to subscribe to a voluntary code, but retain subscriber data as a matter of course for billing purposes.
Under the new proposals being discussed by a Home Office project team, however, public authorities would retain a much wider range of internet traffic and communications data. They would also be able to access it themselves, rather than wait for network providers to hand it over.
Because Computer Weekly understands that the government has not yet decided who would operate the database or under what rules, it is unclear whether government officials would need to present a RIPA notice to access an individual's communications data in future.
ISPs and telcos are currently not allowed to hand over the volume of data currently envisaged in the proposals. The government would need to introduce legislative change to make such a move permissible. If the project is given the go-ahead, new legal rights are expected to appear in a proposed new Communications Data Bill. The Bill, which includes plans to enact the remainder of the European Union's Data Retention Initiative into UK law, is expected to be introduced in the Queen's Speech in November.
"Ministers have made no decision on whether a central database will be included in the draft Bill," said a Home Office spokeswoman.
Fight against terrorism
The Home Office sees the proposals as an essential step in its fight against terrorism, but the potential cost and the technical sophistication of the undertaking has raised eyebrows among technical specialists.
The success of the project will depend on the development of black boxes, known as network probes, that could extract traffic and communications data from raw network traffic.
Computer Weekly understands the project would require thousands of boxes to be positioned at different points on ISP and telco networks. They would be programmed to tap into messages, decode them and pass them on for storage.
Early cost estimates for the prototype work alone are huge. The costs would rise to "eye-popping" levels if applied to a full production national database. This is not least because the network probe technology being considered does not yet exist. Some experts think it may not even be possible to get the idea to work.
Peter Sommer, professor at the London School of Economics, says that no off-the-shelf systems exist to create a specialist database of this type. A new database would have to be custom-made and would require huge amounts of highly performant - and expensive - hardware to run fast enough.
The aim of creating a massive electronic communications database is to save public authorities time in having to approach individual network providers for logs. This time factor is considered important if anti-terrorism officials want to trace possible gang members by examining transaction logs quickly in order to prevent possible loss of life, the Home Office claims.
But Sommer says, "The exercise only has value if the data is available more or less online. In other words, data can be instantly searched and a result obtained in real-time as opposed to it being stored on tape. But we're talking about considerable resources for that and there's all the usual problems of government projects that don't altogether go according to the initial plan."
Value for money
Whether the project offers value for money by focussing on those rare situations where there is an imminent threat to life rather than on improving intelligence-gathering in a general sense to prevent activities progressing that far, was another question, said Sommer.
How such a database could be made secure is unclear. If the database was hacked by members of organised crime rings, foreign governments involved in espionage or terrorists, the potential misuse of the information gathered on individuals' online habits could have serious repercussions.
The cost of securing such a database would possibly be higher than building it. Paul Vlissidis, technical director at the NCC Group, said necessary protection mechanisms would include encrypting the data, rigorous access controls to ensure only authorised personnel could access it and all the usual anti-hacking measures.
But these are not the only concerns around the project. Another is technical feasibility, particularly in the area of data formats. Network providers use a wide range of incompatible data formats. Although there have already been attempts to standardise them to make it quicker and easier to deliver information to public authorities for use in court, little success has been evident so far.