NAO report: a journey from criticism to praise

News Analysis

NAO report: a journey from criticism to praise

Tony Collins

filename

When a report was published in June by the National Audit Office into the NHS's National Programme for IT (NPfIT), it was seen by ministers as a vindication of the UK's decision to spend £12.4bn on the world's largest civil computer scheme.

The report was strongly supportive of the scheme and replete with praise for the Department of Health and NHS Connecting for Health, its agency which runs the NPfIT. But earlier drafts seen by Computer Weekly tell a different story to the final NAO report.

Comparing the earlier drafts against the final version of the NAO's report shows that there has been a cover-up, with passages critical of the programme removed or substantially altered. Among the key changes are:

  • The omission of a report - commissioned by the NAO - which suggested the UK may be out of step with the US and the rest of Europe. Other countries are moving towards linking locally-maintained systems through technical standards and messaging facilities - in Germany, Sweden, and The Netherlands, for example - rather than adopting an arguably more risky centralised patient record.

The report by Cambridge University's Foundation for Information Policy Research also found that, as a result of the NPfIT, England spends some 30% of the European total spend on healthcare IT.

  • The omission of passages from the draft report which show the negative effect on NHS trusts from delays in the delivery of core software by the NPfIT's local service providers. The final report of the NAO suggests that suppliers are absorbing the effects of delays in the delivery of their software.

It is common practice for NAO reports to pass through several drafts to correct and update facts, but in this case the changes between the drafting and publication of the final NAO report do something more. The draft reports on the NPfIT have been reshaped to tell a different, and much more positive, story.

The final report is so complimentary about the Department of Health, Connecting for Health, and the NPfIT that MP Greg Clarke, a member of the House of Commons Public Accounts Committee, commented on its tone at a hearing on 26 June.

"In a year on the committee I have read 62 NAO reports. This is easily the most gushing," he said.

MP Richard Bacon, a member of the Public Accounts Committee who has followed the NPfIT closely and refers to it regularly in Commons speeches, says the report on the scheme is not up to the NAO's usual high standards. He has sent dozens of written questions to the NAO and the Department of Health. He hopes the answers will help inform the committee before it publishes its report.

Now that three draft reports on the NPfIT are available - released to Computer Weekly under the Freedom of Information Act - it is possible to analyse what has changed, the extent of the changes, and their effect on the final report. This is the first time a series of draft NAO reports have been released.

What jumps out from the final report, in comparison with the drafts, is the aggregation of changes that are seemingly minor when considered singly, but which have changed the tone and tenor of the NAO's findings.

The first draft in January 2006 notes that the NPfIT - the UK's largest IT investment, costing £2bn more than the Channel Tunnel - has strong support from ministers and senior management in the medical professions, and there is some praise for the work of Connecting for Health. But the draft also shows that the NPfIT has deep-rooted weaknesses that put the programme's success in jeopardy.

The January draft refers to concerns among NHS trusts about the affordability of the programme it quotes surveys as suggesting that there is considerable distrust and cynicism about the scheme, which the Department of Health and Connecting for Health are a long way from overcoming it also reveals that trainers in trusts are already working to capacity, and that hospitals and the programme's suppliers are trying to recruit from the same scarce pool, which is driving up salaries. By the final report, all these points have been omitted.

The final version is longer than the first draft, despite the deletion of lengthy passages of criticisms. The extra space is given over largely to comment, explanation, clarification, and speculative statements in praise of the programme or Connecting for Health.

In comparison with the unembellished January draft, much of the final report published six months later reads as if it were a defence of the programme composed by the Department of Health.

This is not normal.

The NAO has a reputation for producing factual reports that keep comment and speculation to a minimum. It audits the annual accounts of major government departments such as the Department of Health, the Department for Work and Pensions and HM Revenue and Customs. It also publishes value-for-money reports on particular projects, of which the NPfIT is one.

The NAO has a special place among publicly-funded organisations: a protected independence from government. It is accountable to parliament's Public Accounts Committee, and has its budget set by the Public Accounts Commission of the House of Commons. In this way it is set up to be immune to pressure from government or from permanent secretaries who run Whitehall departments.

Its special role reflects the pivotal position it holds in the democratic process: its job is to report objectively to parliament on the £800bn government spends and receives each year.

But the changes made to this report on the NPfIT prompt questions about whether the NAO's independence has been undermined.

Before NAO reports are published, convention requires that the NAO agree the factual content of its reports with the departmental heads before publication. This process was particularly lengthy - six months - in the case of the NPfIT report.

As the delay in publication dragged on, the NAO came under pressure to agree its report with the Department of Health - but Whitehall officials did not have to sign off an NAO report they were not content to be published.

The resulting document defends the adoption of a centralised programme and depicts Connecting for Health as an exemplary manager of the UK's largest IT investment.

It is also an official vindication of the programme in the face of criticisms of some key facets of the NPfIT from within the NHS, academia and members of organisations such as the British Computer Society.

But the final publication does not represent the concerns expressed by doctors and IT executives in the NHS - or the NAO's auditors in their draft reports.

In the first draft, there are 19 section headings, of which five are critical, including concerns about possible structural weaknesses in the programme. One, for example, says that "not all contractual arrangements have worked".

By the final report there are 21 section headings and the negative ones have disappeared, except one which is only mildly critical: it says that national leadership of parts of the programme has changed a number of times. The words "not all contractual arrangements have worked" have gone.

Some of the praise in the final version - not included in the first draft - introduces discrepancies into the final report figures for savings have been revised upwards by billions of pounds, words and construction of sentences have been changed to remove negative connotations, and some changes go further by turning a neutral or critical statement into a positive one.

The January 2006 draft, for example, has a neutral recommendation advising Connecting for Health to "continue to closely monitor suppliers' performance". The same recommendation in the final report praises Connecting for Health. It recommends that the agency "continue its strong management of suppliers' performance".

And between the draft and final reports, a negative comment about Cable & Wireless is turned into a positive one.

The comment refers to the company's contract with Connecting for Health to provide a secure e-mail service for the NHS. The draft says plainly that Cable & Wireless did not achieve go-live and availability targets between October 2004 and March 2005.

The final version of the NAO report wraps this sentence in a longer one that ends on a positive note. It says that although Cable & Wireless did not achieve go-live and availability targets between October 2004 and March 2005, these were not termination grade failures and the number of registered users of the service increased to 167,946 and the number of active users increased to 80,183.

Some negative findings by the NAO are not disputed by even the NPfIT's most ardent supporters, but still they are omitted from the final report.

For example, the draft says, "The programme represents the largest single IT investment in the UK and it is important for taxpayers and patients that it pays off. However, the programme has attracted criticism from many sources, which, whatever the basis of this criticism, emphasises the need for the programme to be well managed and open to public scrutiny."

The final version leaves out any reference to the sentence about the programme's attracting criticism. It says that it is "important for taxpayers and patients that the investment pays off, and for the programme to be well managed and open to public scrutiny".

In the summary of the final report, the change of a single word from the draft makes the NAO complicit in the controversial view that the quick signing of £6.2bn worth of contracts with local service providers was good news. In the draft, the NAO says that procurement of the contracts was completed "very quickly". By the final version, the procurement was described as having been completed "commendably quickly".

Other changes between the drafts and the final report are more obvious, and all have the effect of showing the NPfIT in a better light. According to the draft report, deals negotiated by Connecting for Health with Microsoft and other suppliers are expected to save about £470m. By the final report, the estimated savings have risen to £860m.

Other figures have been revised upwards. The draft in May 2006 says that Connecting for Health commissioned a report from analyst firm Ovum. The report compared the prices achieved by Connecting for Health with estimates of the prices that could have been achieved by individual organisations buying the same services separately. "On this basis it estimated a saving of some £3.8bn from central procurement by NHS Connecting for Health."

By the final NAO report, the saving had increased. "On this basis it estimated a saving of some £4.5bn from central procurement by Connecting for Health."

Similarly, the draft says that Connecting for Health "achieved significant price reductions from the eight prime contractors, the difference between their initial and final bids totalling £4.5bn".

This figure of £4.5bn remained in the drafts dated May and 6 June, two weeks before the final report was published. By the final report the saving had increased by more than £2bn. The final report said Connecting for Health had achieved "significant price reductions from the eight prime contractors, the difference between their initial and final bids totalling £6.8bn".

The final report's praise is not always accurate. It says the programme is the "only major public sector IT project in the UK where the responsible body has a dedicated website to provide information on the progress of the project". This is not correct. There is a dedicated website on the progress of billions of pounds worth of criminal justice information systems.

But the most obvious changes between drafts and the final version are the omissions.

The June draft, for example, has a reference to a failed implementation of an x-ray Picture Archiving and Communications system (Pacs) supplied by CSC, one of four local service providers appointed by Connecting for Health to deliver centrally-specified systems.

The June draft says, "The Royal Liverpool University Hospitals NHS Trust rejected CSC's Pacs solution for its Broadgreen Hospital after an abortive three-month implementation effort." This is not mentioned in the final report.

The June draft also refers to Norfolk and Norwich University Hospitals Trust. It said it had "decided that it could no longer continue with delays in the delivery of the implementation of the new local service provider solution and it suspended the implementation of iSoft's Patient Administration System to be supplied by Accenture as it was not confident that it would be delivered to the timescale it needed". Again, this does not make the final report.

A draft also expresses uncertainty over whether trusts will set aside enough money to implement national systems locally. "Money will only be available if trusts give IT spending priority over other demands on their budgets," says the January draft. This concern is omitted from the final version.

An entire research paper published in August 2005 by the London School of Hygiene and Tropical Medicine is excluded from the final report. An NAO draft says the survey "reported that uncertainty and poor communication were leading to a lowering of morale amongst NHS staff responsible for implementation". The researchers had looked at the processes and outcomes of implementing an electronic care record in four acute hospital trusts.

Another omission in the final report is an NAO criticism about the output-based specification for the NPfIT. "We encountered a widespread view that there had been a lack of consultation about the development of the output-based specification."

There are too many significant changes and omissions to list here. Had the January draft been published, with allowances for minor corrections, it would given the impression that the NPfIT had much right with it, and much wrong. As it is, the final report gives the impression that the programme is one of the most outstanding successes in government IT.

The most satisfactory explanation for the differences between the draft reports and the final publication is that the NAO has been self-censoring: that it saw its January draft as too plain speaking, too unembroidered, too hard on the programme. So it decided to change the wording, the tone, and the impression created by the draft, not forgetting to add more pages of amplification, clarification and praise for the programme and its managers.

But it is more likely that every sentence, every phrase, was fought over in the effort to get sign-off, with the NAO losing every skirmish. Next week Computer Weekly will publish evidence that Connecting for Health put the NAO on the defensive more than a year before the final report was published, and that the NAO ended up writing a long deferential letter to the agency.

The NAO says that the changes between the drafts and final report on the NHS IT programme represent nothing more than the normal process of clearance that precedes every major report it publishes. But it is not normal practice for the NAO to delay a report for a year and spend six months agreeing its drafts.

The NAO also says in response to accusations of a cover-up that the conclusions and recommendations of the drafts have not changed. This is not true either, as in the main text there are subtle but important differences.

Given the apparent enfeeblement of the final report, questions may now be asked about a process which means the NAO has to obtain clearance from the departments it studies. And some may also question whether the NAO can always be relied on by parliament to conduct objective studies on major IT projects within central government.

Spot the difference

Some of the most serious criticisms of the National Programme for IT in the NHS were removed between the draft and final reports of the National Audit Office. These are some examples:

  • Not all contractual arrangements have worked.
  • Connecting for Health has a long way to go to… overcome a considerable rump of distrust and cynicism about the programme.
  • The department has been slow in securing the engagement and commitment of the NHS to the programme.
  • The programme has attracted criticism from many sources.
  • Changing deployment timetables have also hampered trusts’ ability to plan the resources for hardware and training needed to help ensure smooth transition and acceptance of the new systems.
  • Local service providers’ slow development of their systems has led some NHS organisations to look elsewhere.
  • Relations with GPs were also damaged by concerns that they will be forced to give up their existing IT systems.
  • Money will only be available if trusts give IT spending priority over other demands on their budgets.
  • Within the NHS existing trainers in trusts are already working to capacity.

Read comment: Unhealthy tale of NAO report


Vote for your IT greats

Who have been the most influential people in IT in the past 40 years? The greatest organisations? The best hardware and software technologies? As part of Computer Weekly’s 40th anniversary celebrations, we are asking our readers who and what has really made a difference?

Vote now at: www.computerweekly.com/ITgreats


Email Alerts

Register now to receive ComputerWeekly.com IT-related news, guides and more, delivered to your inbox.
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
 

COMMENTS powered by Disqus  //  Commenting policy