Government ministers and CIOs have identified the key causes of public-sector IT disasters. So why are mistakes repeated?, asks Tony Collins
Every few years, like cyclical bouts of unseasonably warm weather, a minister pledges an end to tackle the causes of systemic failure of IT-related projects in government. But do any of these announcements make any difference?
Jim Murphy is the latest minister to read the yellowing script entitled "How this government will tackle the generic causes of failure".
In a speech to a London conference on 10 October, Murphy promised that the government would use better management and monitoring of suppliers to "tackle the generic causes of failure". He said, "We have had high-profile mistakes in a minority of cases... We are putting a system in place now that should make that sort of systematic failure a thing of the past."
Five years earlier another Cabinet Office minister, Ian McCartney, announced, "We will not tolerate failure." He unveiled a Cabinet Office study on the lessons from IT project failures. The report was authoritative, the result of much work by experienced IT professionals, and contained many important do's and don'ts.
But it has not stopped flawed IT-related schemes being implemented. Audit reports show that projects that supported the Criminal Records Bureau, the Child Support Agency, tax credits, the national programme for IT in the NHS and National Air Traffic Services repeated mistakes from past project failures.
In several cases political imperatives, rushed planning and bad decisions clashed with a need for caution. Business processes were not fully understood or simplified before the systems were designed.
The safest option is not to launch at all, so suppliers and departments have balanced taking risks with not proceeding with implementation. But a lack of scrutiny and external oversight, coupled with exuberance and over-optimism, has led to an emphasis on risk taking rather than caution.
As a remedy for failure, ministers have announced repeatedly that they are avoiding big bang implementations, as if this were the only reason that projects are aborted, do not achieve the predicted benefits, soar above budget or are repeatedly delayed.
The former head of the Office of Government Commerce, Peter Gershon, highlighted this point to the Treasury Committee of the House of Commons in January 2004. He said, "One thing I think we have learned from these so-called big bang implementations, like the Criminal Records Bureau, like new tax credits, like the Passport OfficeÉ is that you have to test them very thoroughly."
He added, "The whole emphasis of the government's approach now to IT systems is to avoid big bang if we possibly can."
The result is that IT projects now run into serious problems for a variety of largely predictable reasons, with the advantage that big bang is not one of them.
For example, the NHS IT programme, the world's largest civil IT project, is suffering from protracted delays and deep disenchantment for reasons that had been predicted. But big bang implementation is not one of the reasons.
This week the Cabinet Office announces a new IT strategy, designed, in part, to consign failure to history. It is likely to be an impressive document, the work of government chief information officer Ian Watmore and his CIO colleagues from Whitehall departments.
Few in government have a better understanding and experience of complex IT-related programmes and projects than Watmore. He is one of an increasing number of private sector experts who have been recruited to tune the engines of government. Other far-sighted recruits include David Varney, chairman of HM Revenue and Customs, and his chief information officer, Steve Lamey.
But one wonders whether any of these talented individuals can make any real difference when so little has changed in more than a decade, despite a continuous flow of ministerial and departmental announcements of modernisation, improvements and new strategies and projects.
Billions has been effectively spent for the benefit of IT suppliers, and for some departments that have expanded their empires. In return for financing huge investments, taxpayers have sometimes suffered levels of service from some major departments that reached new lows.
Magistrates courts are installing "new" case management technology that was originally designed more than 12 years ago at a cost of nearly £400m - almost three times the original sum agreed under a contract with Fujitsu.
HM Revenue and Customs and the Department for Work and Pensions, the two largest departments in Whitehall, have spent more than £2bn on IT developments over the past 10 years. But they still do not have a single view of their customers. Information is scattered on different, obsolescent systems that do not adequately talk to each other.
HMRC, with 100,000 staff, is struggling to cope but nobody with influence seems able to tell ministers who are excited by the anticipation of a major new system: "Please don't launch new initiatives when the foundations are crumbling. Let us simplify business processes and the systems, get our own house in order, then we can build new storeys."
There are numerous successes in the public sector, but they tend to be smaller, tightly controlled projects with specific targets. For example, the Driver and Vehicle Licensing Agency has had several IT-related successes, as have many local authorities and some NHS trusts.
But too often ministers and suppliers encourage each other to think big: to launch something that is immeasurably more complex than what has gone before, to beat the rest of world, to do it quickly, but in stages, and with as little transparency as possible.
One result is that the national programme for IT in the NHS was launched without anyone having a clear idea of how much it would cost when local implementation was taken into account.
The ID cards scheme is going ahead without enough information being made available for assumptions to be rigorously tested.
Ministers say they are confident the schemes will be successful. And if the projects are not, one suspects they will still be declared successes. There is no reason to fear failure.
In short, it is difficult to see how any ministerial or departmental announcement or any new strategy will make much conspicuous difference to the way government is run, its costs, or the services to the public.
New IT strategies will add to the good works on common-sense practices. But sound advice and noble intentions will not make the difference between success and failure: countless parliamentary reports on the causes of IT-related failures show that several are caused, in the main, by non-adherence to known principles of good project management.
Little can be done to force adherence to good practice in the public sector other than by legislating for accountability and transparency, and allowing rigorous external scrutiny of the feasibility and progress of projects by disinterested parliamentarians, the media and taxpayers.
But this will not happen. Whitehall's culture of secrecy and less than frank internal and external communications is too strong.
The private sector has learned, sometimes the hard way, that openness, honesty and rigorous accountability are critical to the success of large and complex IT-related programmes.
In Whitehall, open and honest communications are all but non-existent. A senior executive at HMRC said last month that nobody in senior management could bring themselves to use the word "problem". Potential problems are spoken of as merely "issues", stepping stones on the path to inevitable success.
In the US, funding for large programmes has to be approved by Congress. This policy ensures external debate and rigorous scrutiny by representatives of taxpayers. There is even legislation that ensures accountability to Congress on major public sector IT-related programmes.
In the UK there is so little regular external scrutiny that some MPs resort to asking Computer Weekly for information on how IT-related projects are progressing. They say they do not discover that IT projects are failing until letters begin pouring in from distressed constituents.
Two weeks ago a Home Office minister, Tony McNulty, tried to make a real difference. He said in the Commons that Gateway reviews on ID cards would be published as far as they could be. The reviews are independent assessments of IT projects at six stages of their lifecyle by the Office of Government Commerce.
It was the first time a minister had given any undertaking to publish the reviews. His announcement threatened to lift Whitehall's curtain on the progress of IT-related projects.
Yet the next day the Home Office's press office issued a statement saying the reviews would not be published. A minister had been overruled by his civil servants. Nothing changes. It means that the ID cards scheme will continue without any rigorous external scrutiny, and with parliament being fed selective titbits of whatever information civil servants allow their ministers to reveal.
There are of course diligent IT leaders in the public sector, and determined efforts to bring about change will continue.
The new IT strategy for government announced this week is likely to emphasise the need for professionalism and bringing together human resources and finance divisions of various departments, so-called shared services.
It makes sense to amalgamate departments that are doing the same thing with different staff and systems.
But seasoned observers of the government machine may wonder if suppliers, unable to see e-government as a continued source of revenue, have lobbied for shared services as a replacement initiative.
If the new schemes fail, the suppliers will still be paid; and heads of departments will have nothing to fear because the chances are that parliament will not be told of any failures.
So there is little incentive for departments to succeed. That is how it has been for decades, and there is no reason to believe things will change now.
The BBC programme Yes Minister is often quoted in parliament to explain how the engines of government are powered. In one episode the archetypal civil servant, Sir Humphrey, expounds on the benefits to taxpayers of a fully-staffed and highly efficient NHS hospital which, incidentally, has no patients.
But Yes Minister lacked one important touch of realism. When it comes to the management of IT-related projects and programmes, life in government is Yes Minister without the smiles.