How an independent review could make the difference between success and failure

When consultancy Arthur D Little was appointed to investigate a potential IT disaster at the Swanwick air traffic control centre...

When consultancy Arthur D Little was appointed to investigate a potential IT disaster at the Swanwick air traffic control centre in Hampshire, its independent review was welcomed by all - except the organisation being audited.

In oral and written evidence to the Transport Select Committee of the House of Commons, Computer Weekly had called for the audit after years of delays and cost overruns jeopardised a project to deliver a £337m air traffic control system. Senior management at National Air Traffic Services (Nats), who ran the project, put up strong arguments against our recommendation.

Executives at Nats said an independent audit was unnecessary, as the project and organisation were already subject to independent audits.

These arguments have been echoed by senior managers at the Department of Health's national programme for IT (NPfIT) in response to calls by Computer Weekly for an independent audit. Our call has been backed by the Conservative Party.

A spokesman for the NPfIT said the national programme is already subject to Gateway reviews, which are carried out by the Office of Government Commerce. He said there are also internal audits and the NPfIT is being investigated by public spending watchdog the National Audit Office.

Nats' managers went further than the NPfIT spokesman in opposing an external independent audit. They told the Transport Select Committee that an external audit would delay the Swanwick project further and distract the organisation's managers.

But Computer Weekly gave evidence to the Transport Committee that an independent review could save the Nats project. We argued that such an audit could assess whether the scheme was under the control of managers and whether the systems would eventually work satisfactorily.

The Transport Committee agreed with our arguments - as did the government. On the recommendation of the Transport Committee, ministers ordered two audits: one technical and another managerial and financial. The second audit, for which Arthur D Little was appointed after an open competitive tender, was particularly valuable.

Besides studies by the National Audit Office, the inquiry by Arthur D Little in 1998 remains one of the deepest reaching investigations into a major government IT project.

The Transport Committee took the bold step of publishing the consultancy's 86-page report - having been given a choice by the government to keep it secret.

Arthur D Little's report was not better than those published by the NAO on failures of IT projects, but it was very different.

In the case of the Swanwick system, the NAO could not have investigated. It is excluded by statute from examining some public organisations, including the Civil Aviation Authority, of which Nats was then a part.

Even so, the difference between the NAO's remit and that of Arthur D Little was stark. The NAO assesses whether public money is being spent wisely. Although it does much more, it mainly looks back on whether projects have met their targets, how and why they have diverted from original plans, the cost implications and extracts the lessons learnt for other departments.

It is widely regarded as having unparalleled expertise in extracting facts, putting them into the context of the department's work and making complex situations easy to understand.

But there are constraints. The facts in NAO reports have to be agreed by departments, and this process can delay publication. It avoids being seen to question government policy.

The NAO rarely investigates ongoing themes, and although it has made an exception with the NPfIT, it usually avoids being seen to intervene as a project manager. Its recommendations are often more general than biting.

For example, the NAO reported last month on Choose and Book, a key part of the NPfIT, which aims to enable doctors and patients to book outpatient appointments electronically at a time and hospital to suit.

Its report was widely publicised because it was authoritative, made assertions which were fully supported by facts and highlighted fundamental flaws in the scheme, such as the lack of support and engagement by clinicians.

The NAO pointed out all the main risks and challenges but its auditors were not charged with looking forward and giving a view on whether Choose and Book, as now configured, would work satisfactorily, or whether it was too complex to control. Few of its main recommendations were so specific that MPs, the media and the public could check whether and how the Department of Health has put them into action.

"The department needs urgently to address the low level of GP support for its plans for implementing choice at referral," said the report.

Another of its mild recommendations was to "consider whether further action is needed to secure the required level of GP support, once GPs are fully informed on what choice at referral involves". The department should also "complete its planned benefits realisation plan for choice at referral by the summer of 2005, along with a monitoring mechanism and quantified targets".

And it ends on a comforting note. "As noted throughout this report, the department is taking action to address the challenges it and the NHS face in implementing choice." But it does not comment on whether that action is enough to prevent a failure.

NAO reports typically raise good questions for MPs on the Public Accounts Committee to ask, but over many years it has proved easy for the government to ignore many of the committee's recommendations as they have been made by MPs - inexpert external scrutineers - not professional auditors.

The result is that NAO reports rock boats on the day they are published and also during hearings of the Public Accounts Committee, but the waters soon revert to their previous calm.

Too often the limited shelf-life of NAO reports does not do justice to the depth of their findings. This may be because the recommendations are usually tactful and consensual: they do not require difficult decisions on halting a project.

Nor do the NAO's recommendations require cancelling parts of a project, removing some complexity in software, or opening up a secretive culture. The NAO also avoids criticism of a department's management style, though it will highlight systemic weaknesses if these are historical.

After the failure of the Libra project for magistrates courts, the NAO published a report which extracted the wider lessons. One of these was that, "Departments should have up-to-date contingency plans ready on all major contracts so there is a fall-back position if and when a contract goes wrong."

This was a valuable observation, but the NAO did not give a view on any systemic failures in the management of the project.

The Office of Government Commerce's Gateway review process has proved valuable across the public sector. But these "snapshots" of progress are never published, so there is no opportunity for the project's wider stakeholders to comment on the findings, or hold the department to the recommendations of the Gateway review.

By contrast, the independent audit at Swanwick carried out by Arthur D Little was replete with recommendations which had an impact on government policy. Indeed, part of its remit was to "review the government handling of the project".

It recommended, for example, changes in government policy to improve transparency over the full costs of the New En Route Centre (Nerc) systems project.

The consultancy had no need to seek agreement and consensus on its recommendations. They were written to save the project and did not shy away from pointing out deep-rooted flaws in the management of Nats.

It found that the messages to the board of Nats about the state of the project were "mild" compared to the severity of the risks the scheme was facing.

"Almost without exception, any negative statement about the Nerc systems project is balanced by a positive one without an objective measure being provided of the severity of the risks or their potential impact."

The Arthur D Little report concluded that the key underlying causes of management problems at Nats were in areas of "project strategy, organisational interfaces and cultural barriers within Nats at middle and senior management levels".

A dispassionate analysis of the differences between the report of Arthur D Little and the findings of the NAO on IT failures, or the efforts of the Office of Government Commerce through the Gateway review process shows that they are different and complementary.

After Nats and the government acted on most of the recommendations of Arthur D Little, the Swanwick system eventually went live in January 2002.

The cost of an independent audit on the NPfIT is tiny compared to the amount already being spent. If the audit cost £2m, that would be about one ten-thousandth of the projected total cost of the national programme, and it could help make the difference between a celebrated success and the world's worst IT failure.

Finding from the National Air traffic Services report speak volumes about difficult IT projects >>

Lack of transparency limits value of Gateway reviews inot NHS plan >>


Read more on IT legislation and regulation