Political parties open sites to search engines after Tories' 2013 deletion

The Conservative Party’s deletion of its archive of speeches from internet search engines last October was unique among political parties, according to an analysis of 180,000 web archive records.

But the Tories have since opened the website up to almost full public visibility in the run up to the European elections today.

At the same time, all major UK political parties opened their websites to internet search engines almost entirely, marking the first time an election has not caused parties to obscure some news and policy areas of their websites from public web searches.

The Conservative Party routinely obstructed blogs, PDFs and its online activist centre from search engines and archives since October 2009 when prime minister David Cameron was campaigning for the May 2010 general election on a ticket of political transparency.

His party continued to obscure these parts of its website when it was in power, only removing the block in April this year as the European election loomed.

In 2008, when the Conservative transparency campaign was in full swing, it had a permissive policy, which can be seen now in archived records of the “robots.txt” exclusion scripts that website publishers use to restrict or permit visibility to a wide audience through search engines.

Robots exclusions tell search engines and archives what portions of a website they are permitted to catalogue, and what therefore the public can discover easily. The Conservative Party's original permissive policy meant anybody could find any information from any part of its website.

Computer Weekly's analysis of 34,177 Conservative Party exclusion scripts recorded by the Internet Archive since 1999 determined that it had a transparent policy in place between the London Mayoral election of May 2008 and October 2009.

Its October 2013 purge of past speeches and news from search engines and archives was a unique event, according to our analysis.

A spokesman for the Conservatives said: "If we were trying to hide old speeches we wouldn't have put a link to the British Library's archive of our website going back to 2004 on the bottom right of every single page of our website."

But the UK web archive also obscures its records from search engines using its own robots.txt script, so the effect of excluding them from web searches is the same.

Political parties have used the exclusion script to obscure portions of their website that advocates of transparent politics might insist should be made easier to find by the public.

The Conservative Party barred search engines from seeing its press releases for most of 2004. The Green Party has had a standing block against search engines finding its video and audio since May 2004.

The Labour Party has had a permissive robots policy since 2010, after the last general election, but appeared not to use the facility when it was in power. The Liberal Democrats had a permissive policy between April 2005 and June 2008. But have otherwise not used the facility either.

The Scottish National Party has had a permissive policy since 2011. Plaid Cymru, the Democratic Unionist Party and Sinn Féin have not used exclusion scripts at all, according to archive records.

Ukip had a block against search engines finding its documents and media from late 2013 but changed this in March, removing all restrictions.

The British National Party similarly blocked documents, media and video between July 2009 and the May 2010 general election. But it has had a permissive policy since then.

Of other fringe parties, The Communists put a total block in force in late 2013, the first time it ever made use of the facility. The Peace Party, Pirate Party and Monster Raving Loony Party have all had permissive policies or none at all, while the Worker's Revolutionary Party and Respect Party have not used robots exclusions.

Computer Weekly generated this report by analysing more than 180,000 robots.txt files recorded in use by UK political parties since 1999 by the Internet Archive. It used an archive query language first to obtain data files with references to each record of such robots exclusion files the archive recorded whenever it made copies of public websites. Computer Weekly programmatically downloaded those files and matched them to determine the unique types of robots exclusion used by each party in the last 16 years.