The technical challenges of increasing transparency through open data

The coalition government has been busy making headline announcements over the past few weeks, many of them relying on or directly affecting IT the public sector.

The coalition government has been busy making headline announcements over the past few weeks, many of them relying on or directly affecting IT the public sector.

David Cameron expects IT budgets to play a central role in the £6bn cuts he wants to make, and has also written to departments asking them to upload large amounts of data within fairly strict deadlines.

The open letter to departments and local authorities sets out requirements for the publication of data sets this year, and work is ongoing to ensure that performance data (such as crime statistics in different council zones) is uploaded regularly.

No problem

The announcements are likely to bring plenty of challenges for IT professionals - not least the ongoing battle to show that technology should be part of the solution, rather than a problem or cost that can be cut back easily.

The immediate work, alongside the reviews of big projects, will involve uploading performance data and data sets.

A lot of data is already online and few argue with the principle of increasing openness, but challenges will arise around making sure data is usable and meets certain standards, according to public sector Ovum analyst Sarah Burnett.

"There will need to be a standardised framework for reporting the data, because you have to know that what you are reading is an accurate representation of what's going on," she said.

"There will need to be ways of ensuring that organisations don't have to upload the information manually, and of checking the data is good quality," she added.

"A lot of it will already be available, but different authorities need to have agreements between them. You need to automate the process as much as possible, otherwise staff will have to spend their time doing it, and that's not particularly efficient."

Sensitive data

David Wilde, chief information officer at Westminster City Council and a former central government CIO, said another hurdle will be to make sure you don't upload or share data that could be sensitive. "With contracts that are up for renewal and re-tender, there are some difficult issues around what we share and when."

Councils must publish every transaction over £500, and Jos Creese, president at the society of IT management, Socitm, added that there might also be sensitive data around purchases relating to adult social care or fostering children.

The sheer amount of data that will need to be published is overwhelming, and Creese who is also CIO at Hampshire County Council, said it will take some time to automate the whole process.

"We estimate we have 250,000 payments a year, and there's no automatic reporting mechanism in the standard financial systems. Maybe that's something suppliers will need to provide in the future. At the moment we are talking about putting stuff up in a raw format."

Councils whose IT is outsourced could face further issues and costs, he added.

"The concerns are around the technical and resource challenges," he said. "We don't quite know how big a job it's going to be at this stage. The sooner we scope it out the sooner we'll know about inherent challenges."

The next stage, Wilde added, is what to do once the data is up.

"The next question is how much money and time does the government spend making the data usable - or whether we leave that to other people. Revealing the data might not cost much - what could be expensive is interpreting it, and delivering it in formats that people want."

What the public sector must share:

  • Historic data on the Combined Online Information System (Coins) database, which lists every item of central government spending.
  • All new central government IT contracts.
  • All tender documents for contracts over £10,000. They must be published on a single website from September 2010.
  • New items of central government spending over £25,000 must be online from November 2010.
  • All new central government contracts online in full from January 2011.
  • Full information on international development projects over £500, including financial information and project documentation.
  • Local government to publish new items of spending over £500, and contracts and tender documents for anything over £500, from January next year.
  • Local crime data to be published from 2011. The pay rates of the top-earning civil servants have already been published.

Read more on IT legislation and regulation