When consultancy Arthur D Little was appointed to
investigate a potential IT disaster at the Swanwick air traffic
control centre in Hampshire, its independent review was welcomed by
all - except the organisation being audited.
In oral and written evidence to the Transport Select Committee of
the House of Commons, Computer Weekly had called for the audit
after years of delays and cost overruns jeopardised a project to
deliver a £337m air traffic control system. Senior management at
National Air Traffic Services (Nats), who ran the project, put up
strong arguments against our recommendation.
Executives at Nats said an independent audit was unnecessary, as
the project and organisation were already subject to independent
audits.
These arguments have been echoed by senior managers at the
Department of Health's national programme for IT (NPfIT) in
response to calls by Computer Weekly for an independent audit. Our
call has been backed by the Conservative Party.
A spokesman for the NPfIT said the national programme is already
subject to Gateway reviews, which are carried out by the Office of
Government Commerce. He said there are also internal audits and the
NPfIT is being investigated by public spending watchdog the
National Audit Office.
Nats' managers went further than the NPfIT spokesman in opposing an
external independent audit. They told the Transport Select
Committee that an external audit would delay the Swanwick project
further and distract the organisation's managers.
But Computer Weekly gave evidence to the Transport Committee that
an independent review could save the Nats project. We argued that
such an audit could assess whether the scheme was under the control
of managers and whether the systems would eventually work
satisfactorily.
The Transport Committee agreed with our arguments - as did the
government. On the recommendation of the Transport Committee,
ministers ordered two audits: one technical and another managerial
and financial. The second audit, for which Arthur D Little was
appointed after an open competitive tender, was particularly
valuable.
Besides studies by the National Audit Office, the inquiry by Arthur
D Little in 1998 remains one of the deepest reaching investigations
into a major government IT project.
The Transport Committee took the bold step of publishing the
consultancy's 86-page report - having been given a choice by the
government to keep it secret.
Arthur D Little's report was not better than those published by the
NAO on failures of IT projects, but it was very different.
In the case of the Swanwick system, the NAO could not have
investigated. It is excluded by statute from examining some public
organisations, including the Civil Aviation Authority, of which
Nats was then a part.
Even so, the difference between the NAO's remit and that of Arthur
D Little was stark. The NAO assesses whether public money is being
spent wisely. Although it does much more, it mainly looks back on
whether projects have met their targets, how and why they have
diverted from original plans, the cost implications and extracts
the lessons learnt for other departments.
It is widely regarded as having unparalleled expertise in
extracting facts, putting them into the context of the department's
work and making complex situations easy to understand.
But there are constraints. The facts in NAO reports have to be
agreed by departments, and this process can delay publication. It
avoids being seen to question government policy.
The NAO rarely investigates ongoing themes, and although it has
made an exception with the NPfIT, it usually avoids being seen to
intervene as a project manager. Its recommendations are often more
general than biting.
For example, the NAO reported last month on Choose and Book, a key
part of the NPfIT, which aims to enable doctors and patients to
book outpatient appointments electronically at a time and hospital
to suit.
Its report was widely publicised because it was authoritative, made
assertions which were fully supported by facts and highlighted
fundamental flaws in the scheme, such as the lack of support and
engagement by clinicians.
The NAO pointed out all the main risks and challenges but its
auditors were not charged with looking forward and giving a view on
whether Choose and Book, as now configured, would work
satisfactorily, or whether it was too complex to control. Few of
its main recommendations were so specific that MPs, the media and
the public could check whether and how the Department of Health has
put them into action.
"The department needs urgently to address the low level of GP
support for its plans for implementing choice at referral," said
the report.
Another of its mild recommendations was to "consider whether
further action is needed to secure the required level of GP
support, once GPs are fully informed on what choice at referral
involves". The department should also "complete its planned
benefits realisation plan for choice at referral by the summer of
2005, along with a monitoring mechanism and quantified
targets".
And it ends on a comforting note. "As noted throughout this report,
the department is taking action to address the challenges it and
the NHS face in implementing choice." But it does not comment on
whether that action is enough to prevent a failure.
NAO reports typically raise good questions for MPs on the Public
Accounts Committee to ask, but over many years it has proved easy
for the government to ignore many of the committee's
recommendations as they have been made by MPs - inexpert external
scrutineers - not professional auditors.
The result is that NAO reports rock boats on the day they are
published and also during hearings of the Public Accounts
Committee, but the waters soon revert to their previous calm.
Too often the limited shelf-life of NAO reports does not do justice
to the depth of their findings. This may be because the
recommendations are usually tactful and consensual: they do not
require difficult decisions on halting a project.
Nor do the NAO's recommendations require cancelling parts of a
project, removing some complexity in software, or opening up a
secretive culture. The NAO also avoids criticism of a department's
management style, though it will highlight systemic weaknesses if
these are historical.
After the failure of the Libra project for magistrates courts, the
NAO published a report which extracted the wider lessons. One of
these was that, "Departments should have up-to-date contingency
plans ready on all major contracts so there is a fall-back position
if and when a contract goes wrong."
This was a valuable observation, but the NAO did not give a view on
any systemic failures in the management of the project.
The Office of Government Commerce's Gateway review process has
proved valuable across the public sector. But these "snapshots" of
progress are never published, so there is no opportunity for the
project's wider stakeholders to comment on the findings, or hold
the department to the recommendations of the Gateway review.
By contrast, the independent audit at Swanwick carried out by
Arthur D Little was replete with recommendations which had an
impact on government policy. Indeed, part of its remit was to
"review the government handling of the project".
It recommended, for example, changes in government policy to
improve transparency over the full costs of the New En Route Centre
(Nerc) systems project.
The consultancy had no need to seek agreement and consensus on its
recommendations. They were written to save the project and did not
shy away from pointing out deep-rooted flaws in the management of
Nats.
It found that the messages to the board of Nats about the state of
the project were "mild" compared to the severity of the risks the
scheme was facing.
"Almost without exception, any negative statement about the Nerc
systems project is balanced by a positive one without an objective
measure being provided of the severity of the risks or their
potential impact."
The Arthur D Little report concluded that the key underlying causes
of management problems at Nats were in areas of "project strategy,
organisational interfaces and cultural barriers within Nats at
middle and senior management levels".
A dispassionate analysis of the differences between the report of
Arthur D Little and the findings of the NAO on IT failures, or the
efforts of the Office of Government Commerce through the Gateway
review process shows that they are different and
complementary.
After Nats and the government acted on most of the recommendations
of Arthur D Little, the Swanwick system eventually went live in
January 2002.
The cost of an independent audit on the NPfIT is tiny compared to
the amount already being spent. If the audit cost £2m, that would
be about one ten-thousandth of the projected total cost of the
national programme, and it could help make the difference between a
celebrated success and the world's worst IT failure.
Finding from the National Air traffic Services report speak volumes
about difficult IT projects >>
Lack of transparency limits value of Gateway reviews inot NHS plan
>>