Conservatives erase Internet history

| 12 Comments
| More
The Conservative Party has attempted to erase a 10-year backlog of speeches from the internet, including pledges for a new kind of transparent politics the prime minister and chancellor made when they were campaigning for election.

Prime minister David Cameron and chancellor George Osborne campaigned on a promise to democratise information held by those in power, so people could hold them to account. They wanted to use the internet transform politics.

But the Conservative Party has removed the archive from its public facing website, erasing records of speeches and press releases going back to the year 2000 and up until it was elected in May 2010.

It also struck the record of their past speeches off internet engines including Google, which had been a role model for Cameron and Osborne's "open source politics".

And it erased the official record of their speeches from the Internet Archive, the public record of the net - with an effect as alarming as sending Men in Black to strip history books from a public library and burn them in the car park.

Sometime after 5 October, when Computer Weekly last took a snapshot of a Conservative speech from the Internet Archive, the Tory speech and news archive was eradicated.

Conservatives posted a robot blocker on their website, which told search engines and the Internet Archive they were no longer permitted to keep a record of the Conservative Party web archive.

The Internet Archive was unavailable for comment. But a fortnight after Computer Weekly started asking its San Francisco HQ for an explanation, the Conservative speeches have begun reappearing on its site.

CW had asked the Internet Archive to explain how the historic record of the lead party in the coalition that holds power in the UK could simply be erased.

The Conservative Party's robot blocker forced the Internet Archive to remove the entire record of speeches and news it had collected, in 1,158 snapshots it took of the Conservative website since 8 May 1999.

The Conservative bot blocker listed all the pages barred for public consumption thus (excerpt):

Disallow: /News/News_stories/2000/
Disallow: /News/News_stories/2001/
Disallow: /News/News_stories/2002/
Disallow: /News/News_stories/2003/
Disallow: /News/News_stories/2004/
Disallow: /News/News_stories/2005/
Disallow: /News/News_stories/2006/
Disallow: /News/News_stories/2007/
Disallow: /News/News_stories/2008/
Disallow: /News/News_stories/2009/
Disallow: /News/News_stories/2010/01/
Disallow: /News/News_stories/2010/02/
Disallow: /News/News_stories/2010/03/
Disallow: /News/News_stories/2010/04/
Disallow: /News/News_stories/2010/05/
Disallow: /News/Speeches/2000/
Disallow: /News/Speeches/2001/
Disallow: /News/Speeches/2002/
Disallow: /News/Speeches/2003/
Disallow: /News/Speeches/2004/
Disallow: /News/Speeches/2005/
Disallow: /News/Speeches/2006/
Disallow: /News/Speeches/2007/
Disallow: /News/Speeches/2008/
Disallow: /News/Speeches/2009/
Disallow: /News/Speeches/2010/01/
Disallow: /News/Speeches/2010/02/
Disallow: /News/Speeches/2010/03/
Disallow: /News/Speeches/2010/04/
Disallow: /News/Speeches/2010/05/
Disallow: /News/Articles/2000/
Disallow: /News/Articles/2001/
Disallow: /News/Articles/2002/
Disallow: /News/Articles/2003/
Disallow: /News/Articles/2004/
Disallow: /News/Articles/2005/
Disallow: /News/Articles/2006/
Disallow: /News/Articles/2007/
Disallow: /News/Articles/2008/
Disallow: /News/Articles/2009/
Disallow: /News/Articles/2010/01/
Disallow: /News/Articles/2010/02/
Disallow: /News/Articles/2010/03/
Disallow: /News/Articles/2010/04/
Disallow: /News/Articles/2010/05/
For pages at these addresses, the Internet Archive reported: "Page cannot be crawled or displayed due to robots.txt".

An administrator at the Internet Archive HQ in San Francisco said its guidance for lawyers explained the mechanism. That was that if a website, like Conservatives.com, put up a robot blocker, those pages it blocked would simply be erased from the record as a matter of etiquette.

The erasure had the effect of hiding Conservative speeches in a secretive corner of the internet like those that shelter the military, secret services, gangsters and paedophiles.

The Conservative Party HQ was unavailable for comment. A spokesman said he had referred the matter to a "website guy", who was out of the office.

It wasn't always going to be like this.

Such as when the prime minister first floated his groovy idea that the democratisation of information would transform politics, at the Google Zeitgeist Europe Conference, on 22 May 2006.

"You've begun the process of democratising the world's information," he told the Googlers. "Democratising is the right word to use because by making more information available to more people, you're giving them more power.

"Above all, the power for anyone to hold to account those who in the past might have had a monopoly of power - whether it's government, big business, or the traditional media," said Cameron, who was then campaigning for power as leader of the Conservative opposition.

Cameron was going to make sure the information revolution would hold people like prime ministers to account, he said another speech on 11 October 2007, at the Google Zeitgeist Conference in San Francisco.

"It's clear to me that political leaders will have to learn to let go," he said then. "Let go of the information that we've guarded so jealously."

Transparency would make public officials accountable to the people, said Cameron then. He was riding at the front of the wave that would wash us into a new world, and a new age.

Likewise the chancellor, who on delivering his landmark "Open Source Politics" speech at the Royal Society of Arts on 8 March 2007, declared his ambition was "to recast the political settlement for the digital age".

"We need to harness the Internet to help us become more accountable, more transparent and more accessible - and so bridge the gap between government and governed," said Osborne.

"The democratization of access to information... is eroding traditional power and informational imbalances.

"No longer is there an asymmetry of information between the individual and the state, or between the layperson and the expert," said the Chancellor when he was campaigning for election.

If the Conservative Party had moved its speeches and news archive to a more convenient location it had managed to do it in a way that hid it from the search engines. It might before long end up at the Oxford University's Bodleian Library, which keeps the official Conservative Party archive of really old stuff like speeches from the days before the internet.

The robot blocker - a robots.txt file - tells software bots run by sites like Google and the Internet Archive to bog off. The bots grab web pages for the benefit of plebs like those Cameron and Osborne claimed to be speaking for in those years before they were elected. The bots were what made the democratization of information possible. It was bots that inspired Cameron and Osborne. It was bots that were going to free us from serfdom in the way they said we would be. Without the bots you just had pockets of power and privilege for those in the know. Without the bots you just had the same old concentration of wealth and power there had always been, since long before the Internet Archive started taking snapshots of the Conservative website in 1999.

12 Comments

  • I find this utterly scandalous! I hope other news outlets pick up on it because citizens deserve to be informed about this.

  • Wasn't something similar done in Orwell's 1984?
    Big brother is here and doesn't want you to remember what it promised you. They are gradually leading us into a completely Orwellian society, hoping people don't notice.

  • The Tories may not have been as clever as they think on deleting data from the Internet. They seem to have forgotten that the British Library has been archiving a selection of the UK's Web since the mid-2000s. Not sure whether every speech will be in BL archive but they have certainly taken snapshots of Conservative Party site on a regular basis.

    See: http://www.webarchive.org.uk/ukwa/target/101940/source/alpha

    Paul Anderson @pdanderson

  • Poor Sarah, you are very easily scandalised, aren't you. This merely affects the Conservative's own online record - all the material is readily available online from all the news outlets that published the speeches in the first place.

    In office, the Conservatives have actually delivered on their intent to open up government information more widely. There is now a vast amount of information available that was previously hidden or inaccessible. I'll give you one example - local crime maps. Whether or not all this material is useful to the citizen is another matter.

  • Good to see Computer weekly ensures it's content isn't written with a clear tone of political bias then. I'd have expected at the very least an article written this strongly to have established the intent of those accused, rather than as is far more likely, that someone removed old content from the website, and simply didn't consider that anyone would want it.
    Perhaps it would be more helpful to ask the Conservatives to send copies of their archive to the Bodleian library (if they want it).

  • Yet again Computer Weekly has shown the quality of their reporting by finding out about and publicising this important issue

  • Is it just me?

    This article grossly misrepresents why sysadmins would use a robots.txt file and it doesn't compare with the Labour website where you'll find content before 2010 also not available.

    I'm not surprised the mainstream media is screaming about Conservative robots deleting the internets but I'd have expected better here. More fool me I guess.

  • From what I understand the Bodleian Library, the official archivists of the Conservative Party, have copies of the speeches for research purposes. The Labour Party's archives are held at the People's History Museum and they have also confirmed they have copies of old Labour Party speeches after similar concerns were raised about the deletion of some past Labour speeches on the website.

  • This article gives an explanation for why the Tories would do this - https://theconversation.com/tories-forget-how-the-internet-works-deletegate-reminds-them-20323

  • Re Labor & Conservatives trying to erase embarrassing history - that's just in-character behavior from lying politicians (pardon the redundancy). But for the Internet Archive to voluntarily allow retroactive erasure of pages by mere manipulation of robots.txt on the current web page? That is inexcusable. IA has thereby rendered their site pointless. What would prevent me from buying rights to domain names with lapsed subscriptions for the sole purpose of making content I don't like disappear?

    VG

    • We have similar problems with some fundamentalist schools in the UK who removed embarrassing/illegal content after it was highlighted, and ensured the content was also not available on the Internet Archive.

  • Leave a comment

    Subscribe to blog feed

    Archives

    -- Advertisement --