The National Audit Office has just reported on the quality of the data systems used to measure progress against the hundred or so public service agreements that were set in the 2004 spending review.
The Silicon.com headline for the story reads “A third of gov’t data systems need improvement”. The NAO report carries the opaque title “Fourth Validation Compendium Report” and you would not find or recognise it if you did not know where to look. It is not easy reading but the recommendations in the management summary are clear, simple and rarely followed. In fact, half the systems were fit for purpose. A further third were classified “broadly appropriate but needs strengthening” and 6% “Too early to tell”. Only 8% were not fit for purpose and the remaining 1% were “not established”.
The Public Service Agreements that accompany the 2007 spending review show some attempt to learn the lessons but while departments have taken steps to improve the quality of existing monitoring systems they “have been less than successful at transferring the learning to the development of new data systems”.
Over the next few months we will see growing debate over the measures contained in the 2007 Public Service Agreements – such as measuring the success of University research programmes designed to meet industrial needs by the quantity and prestige of academic citations.
But those who think that the ICT industry has much to tell government about how to measure delivery would do well to read the NAO report first.
Some years ago I was involved with an attempt by a major market research organisation to apply to the ICT industry the techniques they used to measure delivered quality of service for suppliers in other industries.
There was no interest on the part of the main ICT suppliers, outside financial services where delivered service, including speed and reliability of response, was crucial to business survival, not just success.
Instead there was a widespread belief that technical measures would do what was necessary. Not only have these hardly ever been rigorously deployed outside financial services, but the problems known in the early 1990s have not yet been overcome: witness, for example, disputes over broadband speeds, let alone over response times for services delivered over the Internet.
This leads to me to the blackest ICT Christmas Cracker Joke that I have received to date:
Question: What does a hospital call an out-patient whose home life support system depends on an always-on Internet connection?
Answer: Dead on arrival