When the British government claimed emergency snooping powers in July, it did so in contempt of a court ruling that said they ought to be illegal.
Ought to be. The government got its snooping powers
all the same. But it stirred up a stink in the process. And rightly so: the affair has exposed how it has taken possession of a part of cyberspace and begun administering it with the civil sensibilities of a colonial power.
The European Court of Justice had outlawed
these snooping powers - called data retention - in April. But it outlawed them only on a technicality of European law. It didn't actually revoke the UK powers.
The result was like a sharp telling off. But its repercussions could be momentous. It could decide one of the most important and most difficult questions of our time: where to draw the line on surveillance.
Theresa May, home secretary, responded to the court ruling by staging
an emergency that effectively forced parliament to reassert her snooping powers
. If she hadn't done this, she may have been forced to give some powers up.
No matter that British and other EU police had been using these data retention powers routinely since 2009. Their credibility had been severely damaged by the hullabaloo
struck up by CIA whistle-blower Edward Snowden last June. Snowden's first revelation had been a "Top Secret" US data retention programme just like
the one in Europe. What had been routine was suddenly alarming.
Snowden had coincided with a European Court hearing about data retention. Six months later, after continuous revelations about the US data retention programme, the court released a preliminary opinion condemning it in Europe. It outlawed data retention three months later. It might have been a technical matter that had no direct influence in UK law, but the court used its authority to set a powerful example.
That example was a list of principles to distinguish good data retention law from the kind of regime that would characterize
a police state.Carry on snooping
May's emergency legislation
accommodated most of the court's principles. So it was not quite the all-or-nothing stand-off it was widely assumed to be, with the court revoking data retention law and May defiantly reinstating it.
The court's principles were also not as strict as they might have been. It had even applauded data retention in principle. In practice as well, it applauded the basic mechanism, by which the Home Office ordered telecommunications companies to retain records of people's emails, telephone calls and web browsing so that police could look at them.
The home secretary did defy the court however on the most fundamental of all the principles it set out to make sure police didn't abuse these powers.
It was hard to understand why she did this. The court had merely said, these data retention laws are fine in principle but they should not be so totalitarian. It merely wanted police to get a warrant before they started rifling through someone's communications records. Just as they must get
a warrant before they search someone's house.
May's response to the court was basically, lets carry on snooping and talk about it later. She commissioned a review
to soak up any complaints. And she claimed her emergency powers as temporary legislation, called the Data Retention and Investigatory Powers Act, that would expire in 2016.
Yet she could easily have implemented the court's most important recommendation if she had a will to do it.
Part of the stink was that May's emergency legislation looked like it hadn't been produced in an emergency at all. It strengthened the Home Office's snooping powers with intricate legal filigree. But it overlooked warrants, which would have been simple to draw up in law.Anarchy
It may not have been so simple in practice. Police have grown accustomed to doing data searches under a lax regime. It has been so lax for so long that it might have become difficult to impose more stringent rules.
The lax rules do in fact require police to ask permission before they search people's comms records. But they have only to ask a designated police officer. They have consequently been making many more snoops than if they had to get a search warrant. They have been making about half-a-million comms data snoops a-year.
Contrast that with US police, who must get warrants. They have been making roughly 40,000 to 60,000 comms data searches a-year, according
to David Davis, one of few members of parliament who opposed May's emergency legislation.
US police have been searching comms records at a rate, per head of population
, that was just 2.5 per cent of the UK rate. French police have to get data search warrants as well. They make about 36,000 searches a-year - 7 per cent of the UK number in a country of about the same population.
His point was that police were making too many comms searches in the first place. So while their high volume seemed to obstruct the imposition of data search warrants, it might be feasible if the warrants discouraged what were presumed to be high numbers of unnecessary police searches.
Sir Anthony May, the Interception of Communications Commissioner, gave some support to this idea in his 2013 annual report
. UK police might, he said, have been using unwarranted comms data searches so routinely that they had grown to presume any snooping they did was justified if they thought it was.
"It seems to me to be a very large number," said his report.
"It has the feel of being too many. I have accordingly asked our inspectors to take a critical look at the constituents of this bulk to see if there might be a significant institutional overuse of the powers," it said.Crime data
Might have been. There was however a rough correlation between police searches on comms data and numbers of serious crimes. The founding principle of the data retention regime was that police could use its powers when investigating serious and organised crime.
In 2013, when police did about 500,000 comms data searches, their tally of the most serious crimes - homicide, rape and violence with injury - totalled 343,873, according
to the Office of National Statistics.
And then the rest: it does depend how you define serious crime, but in the 12 months to March 2014, another 20,620 people were caught with a weapon. Vandals and arsonists struck 506,190 times, and thieves stole 75,330 cars. Police also recorded about 400,000 drugs and public order offences. And 211,344 fraud cases, mostly cheque, card and online banking.
Police still might have done 98 per cent more snoops than they ought to, which the frequency of US searches suggested to Davis. But the government has shown no desire to draw a line. The UK doesn't even have a legal definition for 'serious and organised crime', according to the ONS. Even the National Crime Agency, which is charged by the Home Office with tackling serious and organised crime, couldn't say what it was when CW asked. The Home Office had a stab
in its Serious Crime Strategy last October. But it was debatable.
A warranting authority would therefore have the same wide discretion as the police seemed to give themselves. It would face the prospect of going from a standing start to 500,000 warrants-a-year. That might be handled by the same magistrates who issued conventional search warrants. But both the Home Office and Department of Justice refuse to release numbers
of search warrants issued. Their capacity is unknown
. The way is unprepared. The Home Office has
interest, though some pretence
The European court ruling nevertheless condemned the whole data retention regime as disproportionate. That there was too much snooping was, in other words, official.
This conclusion, however, derived primarily from the court's other most substantial criticism of the regime: it retained everyone's communications data indiscriminately, regardless of whether they were suspected of a serious crime.
This had been the whole point of data retention in the first place. As Jack Straw, who helped establish these powers as a former home secretary, put it
to parliament in July: you can't know whose communications you need to snoop before you know you need to snoop them - so watch everyone. It would be too late to go round retaining suspect's comms records once a murder was already done.
Accordingly, the Home Office justified its legislation by citing serious crimes where retained comms data had helped police catch the culprit.
Mobile phone records had helped
undermine the alibi Ian Huntley had used to try and evade conviction for the murders of Soham school girls Holly Wells and Jessica Chapman in 2002. They had helped
convict a gang of youths for the cross-fire killing of 11-year old Rhys Jones in Croxteth, Liverpool, in 2007. Likewise, Oxford and Rochdale child grooming gangs were rounded up with the help of comms data in 2013 and 2012.
It sounded obvious when they put it that way. But the court thought the power to track everyone's comms all the time just in case they committed a crime was not a trivial matter. This used to be what totalitarian regimes were supposed to do.Because
Used to be. Now computers have made it more feasible for liberal states to intrude in people's private lives, they have established a regime that captures everyone's comms data. The ultimate justification of this was that it was possible. It is the age-old vindication of power: because.
Totalitarian policing is not normally so easy to justify. Conventionally, if police had the resources to put a roadblock on every street, taking statements from every person they stopped about the who where when of their journey, they would no doubt catch the odd child murderer. If they conducted house-to-house searches on every street, kicking in every door and rifling through every draw, they would no doubt catch some more.
If the Home Office ever trialled such a scheme, they would have caught a paedophile and said, see how this totalitarian power is justified? They would thwart a terrorist attack and they would say, this would not have been possible without an omniscient state.
So society has assumptions against things like blanket surveillance and heavy-handed policing. Human rights law made principles of them in the real world. But in cyberspace, the data retention regime has dispensed with them swiftly.
This suggests they were never as principled as they were merely pragmatic. The "right" merely marked the limit of all that the authorities found it was practically possible to do. As human rights lawyer Geoffrey Robertson QC has put it
when describing the frailty of those rights we do have, states usually find it more convenient to be pragmatic than principled.
The data retention regime for this reason extended police powers into areas of cyberspace where they would normally seem oppressive.
Computers made blanket surveillance possible and harder to refute.
You can rake 6m people with a dragnet stitched from algorithms that look for certain traits, and patterns of behaviour and biography, where before such activities were only possible by the use of extreme and massively overwhelming force.
Totalitarian police powers may have been tolerated only by fascist, military and imperial states, because only they had no qualms about using force to impose more stringent rules than would otherwise be pragmatic.Limbo
Cyberspace is meanwhile becoming a fully-fledged correlate of the real world. In terms of assembly, you can pretty much do in cyberspace what you can do in person. You can meet. You can talk. You can plot a socialist revolution if you happen to have realised it is the only way to get equality. In the physical world, if you plot revolution in a public square, the authorities can watch you: because they can. In a house they need a warrant.
In replicating these powers in cyberspace, the Home Office claimed special circumstances: it worked on the premise that the usual rules didn't apply.
It was able to do this because the data retention regime established an in-between space - a limbo between cyberspace and the real world, where matters such as who owns what and who can do what are still being worked out.
This limbo was constituted of meta-data, a cloud of attributes that describe things in abstract, impersonal terms that allow them to be categorised and processed by computers: colour, shape, size, time, duration, name, preference. Meta-data gives computers a handle on the real world.
The data retention regime captures meta-data about people's communications. It records who spoke to who, when, from where, and for how long. It doesn't record what was said. It just records the major attributes of the communication.
The regime begged a question: who should have rights over this comms data. Before data retention law demanded its retention, it was so ephemeral it almost didn't exist at all. Your old emails might be stored in your inbox archive for a while. Your old texts might be in your phone memory. The audit trail of these comms was not something that was purposefully retained. It was like Eric Dolphy, the outlandish jazz musician, said
of music: "After it's over, it's gone, in the air. You can never capture it again."
Civil liberties campaigners have been trying
to keep it that way. But the data retention regime effectively put comms data in limbo by ordering its capture.Meta-data
The regime staked its claim by contrasting comms meta-data with the actual private comms it described. Private conversation had migrated to cyberspace with its assumptions of privacy intact. Police can't listen to them in either domain without a warrant. But comms meta-data was different. That someone said something is less private than what they said, so it should be afforded less protection: that's the idea.
Hence the data retention regime treated comms meta-data as though it were the sort of information police would get in the real world by watching over a public square with a pair of binoculars, or by sitting in a car outside somebody's door: who goes in, who comes out, how long they stay.
Anybody can sit outside a house to see who goes in and out. This is a civil liberty because it is possible for most people to do it, and difficult for anyone else to stop them. This sort of snooping occurs entirely in the public domain, the social domain, the world outside: in the street - that place where the mob rules when civil society doesn't.
So Police tend do this sort of surveillance in the physical world, lest vigilantes and lesser thugs do it. They do it without a warrant, sitting in their cars with their coffee and doughnuts, and their walkie talkies or whatever.
They can do this sort of surveillance without a warrant in cyberspace as well, thanks to the data retention legislation. The regime has assumed for police the same sort of power.False
This view, however, is false. Searching somebody's comms data is more like seeing inside their house than watching their door.
It is not like knowing merely that two people are in the same house. It is like knowing that two people in the same house were in the same bed talking together, say, from 23:00 hours until 24:30 - and knowing who they are, and who else they spoke to that day, where they spoke and for how long, and for all people, in all houses, at all times.
Cyberspace has its public squares, its forums and mail lists, where police and anybody else can watch at will. But data retention law has given government agents - who are almost entirely police - snooping rights over private meta-data as though it resided in a public space. It assumed snooping powers as a fundamental right of police. In the real world though, police don't have a right to snoop: it is a privilege extended to them under warrant, in respect of their work as agents of civil society.
By defining the comms meta-data limbo as less than private, the Home Office allowed police to circumvent those rules that normally govern their access to private data, and it made this sphere a place where only police could go. It was like they had put a digital panel on everyone's front door that delivered up the who where when of intimate conversations but only to people employed by a security agency.
The home secretary and her supporters made a song and a dance about the old-world rules when they put their emergency legislation though parliament. Police would still have to get a warrant to actually listen in on people's conversations. But their boisterous assurances distracted attention from the fact they had let their meta-data regime sidestep the rules.Dysfunctional
And even the system of interception warrants looked dysfunctional. The home secretary administers those herself. Every single time a police officer wants to listen in on somebody's conversation, they have to ask permission not from a magistrate but from the home secretary.
May claimed her oversight of this was rigorous, when she laid out her justification for the regime on 24 June.
"The warrant application gives me the intelligence background, the means by which the surveillance will take place, and the degree of intrusion upon the citizen," she said in a speech laying out her plans at Mansion House, the City of London's municipal centre.
"I do not take my responsibilities lightly. I approve warrants only on the basis of detailed intelligence and a reasoned explanation of their likely benefit. Sometimes I demand more information before taking a decision or I make my approval conditional. On some occasions I refuse the application.
"On the basis of a detailed warrant application and advice from officials in my department I must be satisfied that the benefits justify the means and that the proposed action is necessary and proportionate," she said.
She did this on average eight times every day in 2013, according to statistics in the Interception of Communications Commissioner's 2013 annual report. She approved 2,760 interception warrants that year, it said.
The oversight of the regime looked a mess from this perspective, from a conventional view of computer-led policing: where automated methods create activity in such high volumes that it overloads people-powered circuits designed in an age when magistrates made polite enquiries before signing pieces of paper that said yay or nay to a gentlemanly request for permission to snoop. That is of course assuming the police aren't simply doing too many snoops: that the benevolent oversight of an actual person with judicial authority is not just a memory of bygone times, bygone
The court ruling
seemed though to imply a solution to this problem. That was a legal framework for a system of oversight so complex that it could only be computer-powered. The home secretary's emergency legislation was in such close accord on this that there can have been no real disagreement between her and the court at all.
Indeed, with the exception of data search warrants, and when it came down to the letter of the law, the court judgment and May's legislation were effectively identical.
The court's over-riding concern was that the data retention regime should not be indiscriminate. But it did not quite say the regime should not retain everyone's comms data: just that there was no need to retain all data about all people for equally as long.
It wanted the regime to treat people's calls, text messages and emails differently. Likewise data that identified a suspect and that identified their buddies. Or suspect and non-suspect. It wanted the confidential work of lawyers and journalists to be afforded some sort of protection from police snooping. The trouble until now was it didn't matter if you were Ian Huntley or Joan of Arc: it was all game, and all stored for just as long, and all made accessible to police under the same lax terms.
Civil liberties campaigners and humanitarian-minded
members of parliament like the Green Party's Caroline Lucas liked
this idea that the regime should discriminate. Blanket surveillance has become emblematic of totalitarian oppression, of pre-crime science fiction where smart, wealthy people decree that you should watch everyone else just in case they upset the established order. Some MPs were concerned this legislation might not respect their own professional right to privacy.
Blanket surveillance sounded for a long time like the worst of all possible worlds. It threatened deranged vigilance over every nook and cranny. But it transpires that the alternative is no maypole either. The alternative is medieval.
The court's alternative would impose gradations of vigilance over people's communications. The degree of vigilance would reflect the risk that they were up to no good: whether somebody was connected to a public security threat, or perhaps associated with a certain place at a time of threat, or just a place of special importance, or circle of people, or particular known individuals - people perhaps already identified as suspects in a serious crime, or who were otherwise identified for whatever reason as people at greater risk of being involved in a serious crime than, say, the last person but maybe not as much as the person before that. These were all distinctions specified in the European court ruling.Privilege
So if you had the good fortune of an upbringing, sailed through school, got a decent job, happy marriage, prosperous children, nice house, safe town, glowing reports, community work, public commendations, right place, right time, antibacterial acquaintances, the Home Office would retain your comms data for the briefest time of all, and this would effectively make it less likely that police would search it. To you, the data retention regime would seem to be defined not by gradations of discrimination but gradations of liberty.
The court said these gradations should be given the force of law. To the common thinking humanitarian, this implied that every gradation of vigilance would be written down in law: a tight and therefore just definition: who got watched more closely and who got let off, with each category of liberty hammered out by public consultation, committee, draft definitions and hours of parliamentary debate.
But the court wanted the regime to discriminate by fluid factors such as time and social group. Police, immigration, intelligence and military agencies have meanwhile
developing computer systems
that discriminate between people according to a statistical measure of risk they might cause trouble.
The Home Office's 2010 to 2013 strategy for Science and Innovation in the Police Service, for example, said
: "In the last two decades, the police service has developed new strategies of crime prevention and control such as problem orientated policing and hot spots policing.
"It has pioneered new analytical and statistical techniques to support these. Crime analysis is now an established part of core police business and new methods are being explored to increase its predictive power."
The data surveillance regime might likewise satisfy the court's impossible demand for granular discrimination by employing computer algorithms to classify who should be watched vigilantly.
Such a system might match biographical and security databases against social patterns and behavioural indicators. It would retain more communications data for longer from those people who appear most likely to murder children or hang themselves.Dissent
It would also collect more comms data - and keep it for longer than average - from people deemed most likely to shoplift from Tescos, splash paint up an investment banker's Bentley, sneak a spliff in the park, keep vigil over a policeman connected to a death in custody, abseil down a coal-fired power station, sabotage an arms fair, be subject of a slander etched in the police rumours database, have sex in a public toilet, go naked or just go incognito.
And it would employ general definitions as well, like all those travelling on high-speed train, walking within the square mile of the City of London, attending protest rallies, visiting Pakistan or Yemen. And maybe those such as members of the Communist Party, school janitors, buyers of teenage porn, who ever associated with anyone who has attended a certain radical mosque at a certain period of time; or maybe all mosques at any time, only with different gradations of risk according to how radical they are. There is most probably a radicalism indicator used to classify mosques. When the UK's "Terrorism Threat Level
" edges up, so will the degree of vigilance taken over data related to anyone travelling by air, and so on. Everyone probably fits into some serious risk indicator somewhere. So watch everyone.
This was the sort of regime envisaged by the court judgment. And for all the hullabaloo in parliament when the home secretary shoved
her emergency measures through, this is just the sort of granular discrimination her legislation imagined: as much on paper at least as the court demanded.Paradox
The real trouble with data retention seems to have been beyond either the wit or remit of the court. For the regime did pose a serious humanitarian threat like those instinctive critics of the surveillance state expected. But the threat was contained in the very solution the court proposed, and that the Home Office was so ready to deliver.
The trouble was that the court left the Home Office to define the gradations of vigilance that determined who it watched more closely than who. Its only condition was that the home secretary herself should make the final decision. She would do this by issuing retention notices to communications providers, just like the one Snowden
leaked last year. These would say what data should be kept and for how long.
So the Home Office stitched gradations of vigilance into its definition of a retention notice. Very specifically, it gave the home secretary power to create a retention notice around any desired category of data, and to describe it with any criteria she desired - whether that be time, place, type or whatever else. And it said she must set for each such category a shelf life: how long it should be retained for police use.
Thus the regime would accommodate changing situations, as it must if it was going to be granular. The home secretary could issue blanket notices if she wanted, telling comms companies to treat all phone records the same way regardless of where, when and who. More likely, she would use blanket notices to form a basis of retention that applied to all people, and further notices to impose greater vigilance over people who fell under the domain of specific security alerts or police operations.
Her notices could even be computer-generated: produced from deep analysis of crime trends, risk barometers and sociological intelligence, such as has preoccupied police science and technologists for "decades
". Perhaps the notices would grow over time a distinctive shape, as police operations and statistical experiment determined that one set of people - perhaps defined by their biographical or psychological misfortunes - was always more suspect than another: a hierarchy of liberty.
The operational result would anyway always be the same, whether the gradations were determined by state-of-the-art, computer-powered risk analysis or by a Home Office team of actuaries using slide rules and tables of social logarithms. They would produce a retention notice for the home secretary to sign and send to a communications provider, and presumably a supporting report of evidence to reassure her pen.Hullabaloo
This all made the parliamentary hullabaloo look farcical. It was fed by human rights groups who raised an alarm over the emergency legislation because, they said, it did not discriminate. But it did. So May's opponents were protesting against legislation they said they wanted, demanding instead a law just like the one they opposed.
They were led by Isabella Sankey, policy director of human rights group Liberty, who said
May's legislation was in "direct contradiction" of the court, which had said "blanket indiscriminate retention of communications data breached human rights."
The court ruling did beg to be interpreted this way. The court was alarmed that the regime watched "practically the entire European population". But it did not say it shouldn't happen.
On the contrary, it merely said people's data should be retained only when it could be clearly justified in the pursuit of serious criminals. That might mean people's data was retained because they were suspected of being connected to a serious crime. But, the court said, it might also apply to anyone at all if that might contribute to the "prevention, detection or prosecution" of crimes for some other reason.
The court ruling would in other words allow a home secretary to order the retention of everyone's data on the basis that you wouldn't know in advance who's data you needed to retain until after a crime had been committed.
Thus the home secretary, her shadow home secretary and her predecessors argued
in parliament for a base retention period - for blanket retention. But it would still discriminate: non-suspect people's data would be retained but not for as long as suspects. That was no less discrimination than the court demanded.
On the face of it, the humanitarian position was that there is a class of people who shouldn't have their records retained at all. That we should not all be treated as suspects implies either that nobody should have their data retained or only some of us should.
The data retention debate was therefore a squabble over how short should be the time that the average person's data was retained, and whether there was a class of people who are so squeaky clean that their data should not be retained at all. But since government, court and humanitarians were all in agreement that the regime should discriminate, they didn't really disagree about much. Their squabble was over the extent of liberties enjoyed by the privileged.Class discrimination
The trouble with the data retention regime was not then that it didn't discriminate enough. Its trouble was it did not disclose enough about the discrimination it did.
It would not disclose the reasoning by which it set its lines of discrimination. Police and intelligence agencies would not publish the algorithms that do their social sorting as open source computer code. The Home Office even refused to publish the actual retention notices it issued. So the public would have no idea what data was being retained and for how long. They would not know what class of people - in both the mathematical and the sociological sense - the Home Office was watching more closely than who.
Their justification for such secrecy was that if people knew how their algorithms determined what was suspect, criminals would know how to act to avoid being netted by them.
But this was a demented logic. It would keep a shadow over the limbo-world their legislation made of meta-data. Because meta-data might have formed the base substance of this limbo. But its structures - its landscapes and thoroughfares - would be defined by the algorithms that categorized the things the data described. A class of comms meta-data whose attributes described patterns of statistically normal, law-abiding behaviour would in effect form the grand parades of this region of cyberspace, where people who tick all the boxes stroll unmolested by civil authorities. And a class of meta-data that described behaviours however more likely associated with crime would form the back alleys more than usually frequented by those who in the language of security service computing are called anomalies.
The regime's secrecy casts a shadow over the grand promenades as much as the seedy alleyways. In the real world, of course, the streets are lit. Criminals, when they're not doing their shopping and taking the kids to school, operate in the shadows. Everyone knows where the High Street is. Everyone knows those places where it's less usual to hang about. Everyone knows what constitutes normal behaviour in any given place. The patterns are familiar. Everyone knows the rules and the risks.
Civil authorities don't cast their streets in shadow to bring criminals out. They turn lights on so everyone can see.
It is police state logic that keeps the algorithms of mass surveillance secret. It is fearful, suspicious and mean. It is report your neighbour and bug your friend. It is a phone call to the police from behind a lamp post, a complaint to the council from a gap in the curtain, and satisfaction from the drone sound that follows. It is a convenience for a society atomized by techno-powered individualism: watch everyone so they don't have to look after one another. It is the ultimate convenience of the consumer society.
The shadows, however, are also where police and state abuses go unchecked.Injustice
The likelihood of police abusing their powers was elaborated last week in a radio interview with Gareth Pierce, a solicitor famous for her defence of people wrongfully
She had exposed how police fabricated
evidence against striking miners
in the 1980s. She put their actions down to institutional prejudice.
"The miner's strike was one example, which is you have made a whole community suspect," Pierce said on the Radio 4 programme
, A Law Unto Themselves.
"One can see it happen again and again in this country. The West Indian community in Notting Hill was made such a suspect community. It's frequently said the whole of the Irish community, over 25 or 30 years, was similarly criminalised. And it's said now that the Muslim community in this country is on block made suspect."
This came up when parliament debated emergency data retention. Katy Clark, Labour MP for North Ayrshire and Arran, asked
the government what it had done to stop the state abusing these snooping powers for political ends. She feared
a repeat of the past, where the security services had taken sides against unionised workers in industrial disputes, as they had in the miners' strike
"The miners were considered to be the enemy within, and much of the rhetoric we hear from Government Members considers trade union activity and people who use democratic means to assert their rights to be a threat to the state," she said.
James Brokenshire, the Home Office minister handling May's legislation, seemed to treat the whole idea as a joke.
It is even likely that social vivisectionists at the Home Office had already determined that union activists were a greater risk to society than, say, managers who paid themselves too much.
But Brokenshire couldn't answer Clark's question. He had no assurances to give against his regime being used for political repression. He would only - in unfortunately patronising parliamentary tones - say
that the Interception Commissioner would keep an eye on it.Aristocracy
The aristocratic manner of the Interception Commissioner's oversight was one of the more substantial complaints
campaigners made against May's emergency legislation. His office inspects just a fraction of police searches on comms data. It does this by reviewing police records of data search applications after the searches have been done. This is a far cry from the court's demand for individual searches to be vetted beforehand by a third party. They are vetted beforehand by a dedicated police officer.
Jo Cavan, head of the Interception Commissioner's office, told Computer Weekly it reviewed 15,000 such police applications last year. That was 2.9 per cent of total requests made. Each request moreover might contain a much larger number of individual requests to search comms records.
The civil liberties groups were concerned that such lax oversight had missed innumerable erroneous snoops. The Commissioner himself said
(vaguely) that he received 869 reports last year of errors police made when searching people's communications records. He found another 101 erroneous snoops in his routine inspections.
Of those errors the Commissioner did handle, half involved snooping on the wrong people, and two led to raids being made on the wrong people's houses.
"I don't think we would say it's a high rate of error," said Cavan. "Because when we inspect small public authorities, we inspect 100 per cent of everything and in larger authorities we inspect 10 per cent."
The Commissioner inspected only 75 of 214 authorities permitted access to communications records last year.
He nevertheless found that of those cases he did review, a majority of errors were were made by police.
This detail was obscured in the Commissioner's report. While it did say 99.2 per cent of search requests were made by police and intelligence agencies, the majority of errors - 87.5 per cent, it said - were made by public bodies.
Other public bodies such as local authorities and central government departments have data retention powers as well. The Home Office blamed them for the errors. But Cavan told Computer Weekly that the majority of errors were made by police agencies.Errors
Introducing her "safeguards" to address problems raised by the court and her Commissioner, May said
in June: "We have stopped local authorities using electronic communications data and other surveillance techniques to deal with a raft of relatively trivial problems."
She later told
parliament her safeguards would protect public privacy would "reduce the number of public authorities able to access communications data."
A single police agency was responsible for 60 per cent of the 101 errors unearthed during the Commissioner's routine inspections, said Cavan. The source of the other 869 reported errors is not known because the Commissioner has not broken it down.
"We need to be vague," said Cavan. "There's huge problems with the record keeping requirements. The overall numbers are flawed."
Still, she said: "The majority [of errors] were from police forces or local authorities. Local authorities were statistically quite high last year. But they weren't this year or we would have made a specific comment.
"So it would suggest there are more police force cases in there," she said.
Local authorities meanwhile did less than one per cent of all comms data searches recorded last year. Though they comprised 62 per cent of those public bodies permitted to search retained data, they were responsible for just 1,766 of 514,608 searches done.
Other public bodies like central government departments, which account for 12 per cent of authorised bodies, did less than 1 per cent of searches as well.
The 54 police agencies, which account for just 25 per cent of authorised bodies, did 88 per cent of all searches. With the three intelligence agencies (MI5, MI6 and GCHQ), police and intelligence agencies accounted for 99.2 per cent of searches.Catch
There was a greater problem with his department's oversight, however, than its being slight. It was short-sighted as well.
It was short-sighted not only because it did not scrutinize the reasoning - the algorithms - the Home Office used to generated its retention notices and determine who should be watched more closely than who.
It was short-sighted because it did not oversee people's comms data after police had obtained it, bar some cursory interest noted in the Commissioner's annual report. The oversight primarily scrutinized the way police acquired comms data. Granted, it was concerned with proportionality, a principle of human rights law: that police didn't get more data than they really needed. But it verified in effect the accuracy and efficiency of police data searches and strayed no more.
The court had raised an issue about this, momentarily. An old maxim of data protection law held that someone could only use any data they acquired for the purpose they originally acquired it. And when they had finished with it for that purpose, they were supposed to delete it. They were supposed to tell people what they were doing with it as well. The court said this meant police should tell people what had been done with their data when there was no longer any need for stealth.
This might mean, for example, police couldn't stuff suspect comms records into a database for future reference. They couldn't share them with another intelligence or police agency. They couldn't load them into a system that analysed social networks and patterns of behaviour.
When Pedro Cruz Villalón, ECJ Advocate General, delivered the court's preliminary opinion on data retention last December, he upheld these rules against the regime. But the court watered them down for its final ruling in April.
The ruling said only that police should have good reason for putting data to subsequent use. The only reason they could keep hold of retained data, it said, was for detection and crime prevention. And only for "precisely defined serious offences". The UK Criminal Justice Act 2003 defined
these as offences punishable with 10 years or more imprisonment - crimes such as the Soham murders that now justify the collection of all comms data for all people in case it might be used to detect and even prevent such crimes in the future. But the UK legislation used the justification of organised crime as well.People data
With British police doing about 500,000 comms data searches-a-year, they would have done approximately three million comms data searches by the time the emergency legislation ran out in 2016. Actual numbers of accesses are not published because, as the Commissioner said, police don't keep proper records. But with many police applications for data accesses containing many more individual data requests, the numbers of people searched could be much higher than records initially suggest.
Those people made subject to police data searches were thus disregarded by the system of oversight that was meant to protect them. The system recorded no tally of people data-searched. It made no attempt to report the categories of people whose data is retained and searched. The oversight was conducted from the perspective and for the sake of police operations.
Even the Commissioner's proposed reforms of police search statistics did not address this problem.
"We have consulted with the Home Office and set out the revisions and enhancements of the statistical requirements that we believe are necessary both to assist us with our oversight role, and, to inform the public better about the use which public authorities make of communications data," he said in his report.
The Home Office and Commissioner concluded from this non-public consultation that their oversight of data searches would be improved if police were required to keep numbers of the total applications they submitted. Not people, but the applications that each contain many data requests that may concern numerous people.
The situation might also be improved, they concluded, if police were required to record the number of items of data they requested, and whether it was done for crime detection or prevention, or for the sake of national security. But not people.
They did want police to record what type of crime had justified their data search - whether it was for the sake of murder and whatnot. But they didn't care to record even numbers of people.Back of an envelope
A Conservative estimate would equate three million comms searches to 6m people, on the basis that most comms occur in two-way-conversations. But an audit trail of somebody's comms for police purposes would cover more than one conversation. It might include all emails sent in a day, all mobile phone calls in a given week. It is not absurd to imagine police and intelligence agencies populating their statistical crime analysis and prediction systems with enough comms data to map the social networks of a significant portion of the UK's 64 million people - perhaps even all suspect people. From another perspective, perhaps, all the underprivileged
, stooping lower now under the weight of one less liberty.
The fundamental problem with the data retention regime, said the court, was that it allowed police to draw an exhaustive map of people's private lives, an intimate portrait of their private identity.
The effect would be to cast a chill over people - a "feeling that their private lives are the subject of constant surveillance", it said. The problem with this was that it suppressed people's freedom of expression.
It would encourage them to conform with the narrow world view conceived by social actuaries in Home Office uniforms, that they knitted into algorithms that describe a pattern of officially-sanctioned human behaviour. That is just the effect of the data being collected. That is even before the authorities put their collected data to use.Sex and drugs
The court ruling implied a need for dissent to be possible. It implied a time when homosexuality was still illegal, or when women were still denied the vote. It implied those boundary regions of social comprehension where in recent decades transsexuals have made a stand against discrimination. Or those margins of the law where ongoing widespread transgression may yet force the state to concede people should be granted the liberties they already take for themselves, such as sex workers and drug takers. It is currently possible that drugs and sex work will be decriminalised, but only because large numbers of people persist in defying the law: as persistent illegality eventually beat those oppressors of homosexuality - because people persisted in defying the law, because they could.
The court ruling implied prejudices that are yet so ingrained we cannot see them. It implied perhaps the gross inequalities of wealth that might yet only be redressed if enough people use their their freedoms to defy the state, in open dissent. But only if they can.
"You do think sometimes that society learns lessons. But that isn't so. All there is is the ability to be constantly alert that all the danger signs are there," said Pierce, the human rights solicitor, on the Radio 4 programme
dedicated to her last week.
Asked about her work defending victims of wrongful imprisonment such as the Guildford Four and the Birmingham Six, she drew parallels between the internment
(and torture) by British police of Irish Catholics
in the 1970s and again of terror suspects in Belmarsh
Prison in Woolwich, South East London, in the last decade.
"... internment ... Having said it would never be used again in this country, we lock people up indefinitely without trial, and our government lawyers argued in addition that the government should be allowed to rely on evidence derived from torture.
"In the 21st Century, we were having to argue that it shouldn't be used, against our government's lawyers," she said.
That is not to tar the entire establishment with the same brush. Nor should the exception(s) prove the rule. But Pierce's career demonstrates how those with power cannot always be trusted.Us and us
The data retention legislation admitted only that those without
power cannot be trusted. It gave police granular insight into our lives while subjecting them only to the most cursory oversight. The legislation itself makes us into us and them and them.
The court ruling implied a right for us to protect ourselves from them. But the home secretary's justification of her legislation, set out in her Mansion House speech
in June, was based on a desire to protect them from us.
Most of the reasons she gave for doing more powerful comms surveillance were the threats of terrorism that had been caused or worsened in the first place by their
military invasions and their
The legislation was necessary to protect us, she told her City of London audience, from the disaffected people of "Iraq.. Afghanistan.. Syria.. Yemen.. Pakistan.. Libya".
Blinded by this idea that she was with us against them, the home secretary gave operational oversight of comms data snooping to the same police and intelligence agencies doing the snooping.
These were the police authorities in whose custody 17 people died
last year. These were the intelligence agencies who helped fabricate
the case for war against Iraq. These were the overseers who held the British flag aloft while people were kidnapped
And this has been overseen, in confidence, by a house of parliament who gave us gross inequality, expenses fiddling, cash for questions, cash for honours, cash for lobbying, cash for policy, the surveillance state, an alleged paedophile-ring cover up, back-room security deals, an imperial legacy, and war, war and more war.
It all comes down to trust ultimately. Police like to be trusted. As people, like us, they deserve to be trusted. It is safe to assume that like us, they mostly mean well.
Even those police who have already abused their data access powers were most likely trying to do good. Like those
tabloid journalists at News of the World who hacked the mobile phone of Milly Dowler
, the 13-year old who was abducted and murdered on her way home from school in 2002. Anyone who knows journalism knows those journalists were doing their bit, in this case at least, to look for the murderer. They just let their crusading zeal carry them over the line.Carried away
The home secretary may have been similarly carried away when she laid out her justifications for claiming comms data surveillance powers.
She illustrated her case by citing how police had arrested 2,500 people on terrorism charges between September 2001 and 2014. This was supposed to warrant more unwarranted snooping. But she forgot to say the conviction rate
for arrested terrorist suspects was
just 16 per cent. In 2009 it was
4 per cent. In comparison, the average UK conviction rate for all reported crimes - from theft to murder - since 2003 was
80 per cent.
Similarly, May and her supporters in parliament - such as shadow home secretary Yvette Cooper, who backed the emergency legislation eagerly - used
misleading information to persuade other MPs to accept it.
They conflated the warrants police must seek before placing a wire tap so closely with their case for warrantless comms data searches that it effectively reassured parliament that warrants would cover all when they would not.
Confusingly, they also made a case for warrantless comms data searches that was morally impregnable but factually false. That was life or death emergencies.
They used the example of a child who told a telephone helpline he was going to end his life. Police got his internet address and arrived at his house just in time to cut the rope before he died.
But they forgot to mention that police already have powers to act on their own discretion in an emergency. They can even tap someone's comms without a warrant in an emergency. They just need verbal confirmation. In fact, police used
their emergency powers to search people's comms records 42,293 times last year. That was roughly the same number of all comms accesses done by US and French police in an entire year. They would have the same discretion to search comms data in an emergency even under a warranted system.Spurious evidence
Another spurious fact the Home Secretary used
to sell her legislation was that police had used comms data in 95 per cent of all serious crime cases. Therefore, the argument went, it was vital.
But police have had unwarranted access to this data since 2009. Of course they will have used it in the majority of cases. The behaviour does not justify itself. May's justification was simply because they can.
Take middle-aged women, for example. They earn
between 18 and 30 per cent less than men in similar jobs. The gap was about the same 10 years ago.
Overall wage inequality has meanwhile been worsening
for 30 years. Inequality is rife. But that does not justify inequality. Those with power permit inequality because they enjoy its fruits. Because they can.
So while a rough correlation between the number of police comms accesses and the number of serious and organised crimes might suggest their snooping was roughly in the line of duty, it seems foolish to allow the police to take unbridled power, and to allow them to wield it in secrecy.
But if 500,000 comms data snoops is the product of reasonable policing, that raises a question about how to do oversight at such a large scale.
Of course, as police have had the resources to manually vet their own snoops, then it stands to reason that magistrates' offices can find the manpower to vet them as the court intended. Unless it has been more feasible for police to do the vetting because the volumes are too high for manual oversight to be done properly. Computers would then be the obvious answer, for oversight of high volume snoops on a high volume of information age comms conducted by a high population of people. The oversight would have to be risk assessed to make it manageable - to make it comprehensible to computer. It might even give the public the same powers of oversight over police as visa versa: a public watch; and more stringent on any police officer ever associated with a death in custody, say, or a complaint about racial prejudice or a shoot-to-kill: like Big Brother, only in reverse.
Some police activity is already risk assessed, behind the scenes at the Home Office, by the same systems that determine who is suspect enough to be pulled up at airport check-in, or who has their communications records retained for longer than anyone else in the first place.
Such systems would know the degree of certainty with which a particular person or group has been targeted by surveillance. The statistical measure of risk that might determine that known anarchists within spitting distance of Whitehall, say, or known paedophiles within sniffing distance of a school, were a greater risk than visa versa: that measure would also signify the degree to which that estimate could be trusted. So the data accesses might end up justifying themselves: by the probability of the risk estimate. The Home Office might think therefore that it need be answerable to no-one. But the public will only have proper oversight when they are told where the lines of discrimination have been drawn and with what degree of certainty, and when they are permitted to scrutinize and contest the intelligence systems and the algorithms themselves.