Tony Collins looks back at some of the successes and failures of central government computing over several decades and asks: what has happened to the public money that has been spent so far?
By submitting your personal information, you agree that TechTarget and its partners may contact you regarding relevant content, products and special offers.
In 1969 the UK civil service began experimenting with large and complex schemes to use computers to standardise the running of central departments the results are still keenly awaited.
Learning few of the lessons from the 1970s and 1980s, namely keep IT simple, ministers and civil servants have continued to launch ambitious and subsequently notorious schemes to help manage payments of child support, tax credits and farming subsidies, the issuing of passports, collating intelligence for the Ministry of Defence, and the handling of police suspects.
Market researcher Kable said that £15bn a year is spent on public sector IT, £2.6bn of it by central government in 2005/2006. Over the 40 years since central government has used computers in earnest, the money spent on IT is thought to be far in excess of £25bn - about £400 for every man, woman and child in the UK. This huge spend has served a few major suppliers well.
One recipient of Whitehall's largesse is EDS, which signed what was in 1994 the government's biggest IT outsourcing deal, when the Inland Revenue transferred its ICL mainframes and 2,000 computer specialists to the supplier. With extra work, the £1bn contract turned into one costing about £2bn.
By the time Capgemini took over from EDS in 2004, the estimated cost of running the enlarged department's IT systems had doubled again, to about £4bn over 10 years.
Consultancy firm Ovum reported recently that Capgemini makes 44% of its profit on the Revenue deal from "essentially add-on work over and above the original contract".
A third supplier, Accenture, bid £45m to build and run replacement national insurance systems, and was given extra work worth between £70m and £144m.
Another company, Fujitsu, won a £184m contract in 1998 - Labour's first private finance initiative deal - to supply a standard national system which would allow staff in magistrates' courts to manage cases electronically.
It was unable to deliver the case management system, but was still paid more than the original contract price - a final amount of £232m.
In 1996, a fifth supplier, Siemens, was similarly contracted to deliver a £77m system to help immigration officers manage their case workload. It was unable to deliver the system and was paid instead for helping immigration officers improve the way they did things manually.
However, suppliers have not always had their own way. The Department of Health in 2002 appointed Richard Granger as director general of NHS IT, and he has managed to stop suppliers beaming all the way to the bank whether they delivered or not.
There are other positive developments in the way civil servants have managed projects in central government over the past four decades.
The Driver and Vehicle Licensing Agency has made it possible to renew car tax online without having to go to a post office with the correct paperwork. This useful innovation has required a linking of information from insurance companies and MoT garages. It was a project with a simple, clear purpose which had the strong support of all stakeholders.
Civil servants, with their supplier EDS, have had an IT success with Pension Credits, a system which gives top-up payments to some needy over-60s.
There are a plethora of other successes in which IT directors have made a little money go a long way. The Land Registry has made valuable information easily accessible online, as have many local councils.
But the innumerable, unsung successes are dwarfed by Whitehall's taste for the dark side of computing: the overly large and complex projects which have limited support from potential end-users, and which trudge on for years without hope of justifying their cost. Over four decades these oversized projects have given central government a reputation for serial mismanagement of IT.
It is not simply the failed projects that have soiled Whitehall's reputation. Some central departments and agencies have spent billions on systems, consultants and services, and still struggle with antiquated equipment, giving the public a poor service.
No administration has yet introduced a standardised means of recording an individual's name and address in one place hence many of us have our names spelt in different ways on different systems. Our registered addresses on some systems are sometimes decades out of date, and an unknown number of us have different national insurance numbers on different systems.
Whitehall still has different e-mail systems, some of which cannot exchange messages because they each have different levels of security. Three departments which have overseas offices, the Foreign and Commonwealth Office, the Ministry of Defence and the Department for International Development, have e-mail systems on different security levels which do not adequately communicate.
And the advent of an ID card, or at least the possibility of one, has raised a new question for Whitehall's chief information officers: should we be identified as citizens on government systems by a national insurance number - even though these numbers cannot always be relied on - or by a national ID number, even though the national ID systems may eventually prove unworkable?
Experienced chief information officers from the private sector have been recruited to end Whitehall's reactionary culture. But they can appear to some as no more influential than a row of powerful fists trying to demolish a 1960s tower block.
If they need extra help they can call on the Office of Government Commerce which oversees computing in central government. But the OGC is sometimes of limited help, though it will advise the CIOs on new or revised methodologies for increasing the impact of fists on 1960s tower blocks.
Meanwhile, two central departments with the largest IT budgets also have some of the oldest systems. HM Revenue and Customs and the Department for Work and Pensions rely on Fujitsu VME equipment, key parts of which date back more than 20 years.
HMRC's disjointed PAYE systems, which collect more than £110bn a year, were criticised in a report of the National Audit Office this year.
"HMRC can have difficulty in ensuring that taxpayers with more than one source of income pay the correct amount of tax because it may not know about additional sources of income."
This department which has, for more than a decade, calculated its annual IT spend in hundreds of millions, does not have dependable information with which to manage tax affairs.
The report of the National Audit Office, dated 7 July 2006, said: "Deficiencies in management information have also made it difficult for HMRC to prevent or detect errors made by staff. And several times in recent years HMRC has diverted PAYE resources to other areas of work which it considered had higher operational priority, such as tax credits."
The report also noted that the "operation of PAYE has been compounded by inconsistent working practices within HMRC." The department's CIO Steve Lamey has described some of the department's IT equipment as of interest to technology historians.
So what has gone wrong? Why do some departments continue to spend billions of pounds without making much apparent progress?
One answer is buried in exchanges 22 years ago between MPs on the House of Commons' Public Accounts Committee and top civil servants to discuss the "management and control of the development of administrative computing in government departments".
The questions and answers on 28 March 1984 were pertinent because MPs wanted to know why IT projects were regularly failing, and civil servants were saying that there was a shortage of skilled staff, that things were improving, and that the right project methodologies and lessons were being learned.
Little has changed. Over the past 10 years there have been countless exchanges along the same lines between top civil servants and MPs over major projects in the NHS, Home Office, Department for Work and Pensions and HM Revenue and Customs, and the Ministry of Defence.
But civil servants change, and the ones who learn the lessons move to other jobs, and their enthusiastic successors embark on projects without a thorough knowledge of why past projects had to be buried.
In 1984, MPs were concerned about a proposed project called the Operational Strategy, the objective of which was to bring new advanced levels of automation to the payment of welfare benefits. They were questioning a director of the Central Computer and Telecommunicatons Agency, now the OGC. They also questioned a benighted Geoffrey Otton, a permanent secretary, and JWC Spackman, director of the Operational Strategy.
MPs were sceptical that Opstrat, as the Operational Strategy was called, would ever work or stay within budget. Camelot, a previous benefits project, had failed at a cost of £12m.
Otton told the committee that Camelot was an experiment: "We have, I hope, learned the lessons from Camelot. We have gone to a great deal of trouble to restructure our management processes. We have recruited Spackman who brings a great deal of technical expertise into the department. We are making extensive use of consultants we are on course very safely on the things we are trying."
But the lesson of not being too ambitious was not then fully understood. Camelot cost £12m and Opstrat would cost about 60 times as much - about £700m. And its actual cost would go way beyond this, setting a new record for overspent IT projects.
Back in 1984, the chairman of the Public Accounts Committee, Robert Sheldon, asked Spackman whether a project as ambitious as Opstrat had been undertaken successfully on a similar scale and complexity.
Spackman's reply was indistinguishable from the assurances that have been given to MPs decades later on the ID cards scheme, for example. "What is possibly unique in this one is its size," said Spackman.
"I do not know of a project that actually involves quite as large a number of online terminals but it is fairly straightforward if it were being done on a much smaller system it would indeed be a pedestrian and fairly straightforward project. It is its size which makes it complicated."
The then MP Eric Deakins asked Spackman whether the projected costs of Opstrat of about £700m were likely to rise in real terms. In good faith Spackman replied: "No, they are unlikely to rise in real terms the equipment costs if anything are going to come down."
Ten years later, on 5 May 1994, social security minister Nicholas Scott confirmed to the House of Commons that the costs of Opstrat had risen by £1.9bn - from £713m to £2.6bn. "The most recent - 1993 - estimated total cost in real terms of the operational strategy to 1998-1999 is £2.6bn, which includes an estimated £315m, 11.9%, for consultancy services.
"The total cost of the operational strategy was originally, in 1982, estimated at £713m. The current estimated cost exceeds the initial estimate by £1.9bn. The original specification did not embrace the substantial cost of subsequent changes in social security legislation."
But, as today, there was officially no such thing as an IT failure. Scott made the £1.9bn overspend seem insignificant when he claimed that "estimates of total savings over the period of the operational strategy have also been revised from an initial estimate of some £1.9bn to £3.3bn."
This portrayed the distended Opstrat project as an unqualified success, saving hundreds of millions of pounds over its cost. But Scott's announcement, based on a briefing by his department, needs to be put into context.
Indeed, there is no evidence that Opstrat has achieved the savings claimed for it. On 12 January 1989, the National Audit Office had published a report on Opstrat. It suggested that the government had no real idea how much Opstrat had saved, or would save.
The report said the project had been hit by a shortage of skilled staff, the department had "seriously underestimated" the cost of installing systems in local offices, and it had overestimated savings. So poor was the way costs were recorded that double counting of savings "could not be identified".
The project exemplifies the problems with computing in 21st century Whitehall. Failures do not officially exist.
In August 2006, the DWP, the successor of the organisation that launched Opstrat, secretly cancelled a £141m IT project to replace several benefits systems, dubbed the Benefits Processing Replacement Programme.
In a statement issued after news leaked out, the DWP said that the lessons had been learned, and that most of the work had been reused in other projects. The full truth did not emerge.
In the 24 years between Camelot and the Benefits Processing Replacement Programme little had changed.
One would like to think that the mistakes of the past few decades would have made ministers and mandarins paranoid about launching any more overambitious IT schemes. The opposite has happened.
In a report published on 19 September 2006, non-profit organisation The Work Foundation said that government executives were too gung ho about IT programmes and projects, and had lost millions of pounds in taxpayers' money. The report stated that executives need to lose their "reckless streak" and concentrate on what is practical.
The report claims that too many government IT projects are insufficiently piloted before being rolled out, are overly complex in design and ignore the advice of the staff who will use the systems.
Alexandra Jones, associate director at The Work Foundation and co-author of the report, said: "Too many government IT projects fail to deliver the promised benefits because public sector managers have a reckless streak - they become dazzled by the potential of the technology and lose sight of what is practically deliverable."
She said the government should not be about cutting edge innovation, but about serving citizens well and efficiently. "The private sector can afford the luxuries of innovating - in the public sector, IT needs to work."
Her words will doubtless be ignored. The Department of Health has launched the world's biggest non-military IT-based programme, the £12.4bn National Programme for IT. It has been marred by shortages of skilled staff, an underestimation of total costs and over-optimistic statements by ministers on when systems would be delivered. Officially it is already a success.
The private sector has its disasters - but over the decades one can see that corporate victims tend not to repeat major failures. If anything a large-scale failure encourages boards to think small next time.
It is unlikely that ministers and mandarins will ever enjoy thinking small when it comes to IT, not while a significant part of the IT industry depends so heavily on the public sector's love of high stakes gambling.