Daniel - stock.adobe.com
At first glance, the report by members of Parliament (MPs) of the PublicAccounts Committee (PAC) tells the story of flawed engine control software on the Chinook Mk2 helicopter.
The report finds the helicopter’s poorly-functioning Full Authority Digital Engine Control (Fadec) system could not be ruled out as a factor in the RAF’s worst peacetime accident, when Chinook ZD576 crashed into the Mull of Kintyre in June 1994, killing four crew and 25 intelligence specialists.
The PAC’s findings vindicates a campaign by Computer Weekly and others who have sought to establish that pilot negligence may not have the cause of the accident.
However, the wider issues raised by the PAC’s report will be familiar to those who have studied IT problems across all government departments. In the report, there is evidence that the Ministry of Defence (MoD) suppressed evidence that could have led to its own political embarrassment, at the expense of an open investigation.
Parts of the PAC report conflict in tone and substance to statements made by prime minister Tony Blair, defence secretary Geoffrey Hoon, various Labour ministers including John Spellar, Lewis Moonie, Doug Henderson and John Reid, and the permanent secretary at the MoD, Kevin Tebbit.
At various times over the past three years, Blair, his ministers and Tebbit have upheld the decision of two air marshals. In 1995, they found that the pilots of Chinook ZD576, flight lieutenants Rick Cook and Jonathan Tapper were, without any doubt whatsoever, grossly negligent. The pilots flew too low and too fast into a cloud-covered area on the Mull of Kintyre, said ministers on the basis of briefings by officials.
In the previous Tory government, ministers made similar comments at the time, endorsing the findings of the two air marshals. Again, their knowledge of the crash and circumstances surrounding it was based on the strength on their briefings by officials.
The Defence Select Committee, also based on briefings from officials, came to views that conflicted with today’s PAC report.
In 1998, the Defence Committee, for example, attached no safety significance to the concerns over the Fadec expressed by the MoD’s software assessors at Boscombe Down. Defence MPs went further. They attacked Boscombe Down for its “failure” to give final approval to the Fadec. They said this was a “management failure”, adding, “We are persuaded by the evidence that this absence of approval raises no safety-critical questions”.
But today’s PAC report finds that Fadec could have caused the crash – nobody knows for certain whether it did or did not.
Differences in evidence
It is highly unusual for the PAC’s findings to be in conflict with another select committee, but there is a worrying difference in the way the two committees and ministers have set out gathering and evaluating the evidence.
In an investigation into the circumstances of the Mull crash that has lasted several months, the PAC took evidence from eclectic specialist sources. These included the MoD, Computer Weekly and Malcolm Perks, who was the MoD’s expert witness in the department’s legal action against Textron Lycoming, the supplier of the Chinook’s Fadec.
Further technical evidence was given to the committee by a Chinook unit test pilot, squadron leader Robert Burke, who said the Fadec system was so unreliable at the time of the Mull crash that systems were being regularly replaced on the Chinook Mk2.
In contrast, Blair, his ministers, Tebbit and the Defence Committee, based their findings on briefings that were almost entirely from MoD officials and military staffs.
Yet evidence of the MoD’s lack of objectivity over matters related to the Mull crash is copious. In July, based on briefings by officials, Blair wrote to independent MP Martin Bell assuring him that the government was being “open” about the circumstances of the crash and the conduct of the inquiry.
Blair may have been unaware that, a few weeks earlier, the MoD had refused a request by the families of the dead pilots for a copy of Boscombe Down’s official assessment of the Chinook Mk2 prior to its introduction into service.
The MoD has also refused to disclose, even to the PAC, details of the Fadec-related incidents before the crash.
In addition, the MoD has declined to publish the independent assessment of the Fadec software by IT defence contractor EDS-Scicon.
The MoD has told MP Robert Key that it is withholding a “whitepaper” by Fadec supplier Textron Lycoming, which responds to the concerns expressed by EDS.
Blair also told Bell, on the advice of officials, that the government was being “straightforward” in matters related to the crash. But Key recently received a parliamentary reply which quoted liberally from internal MoD legal papers on the litigation against Textron Lycoming, while another MP was refused any access to the same documents because the MoD said they were confidential.
The confusion over what was fact and what was not, in the MoD’s eyes, was particularly noticeable at a hearing of the Defence Committee in 1998.
MPs on the committee were told in a MoD statement that “Boeing did not consider the Fadec to be flight safety critical because the engines on a Chinook are not considered to be safety critical”. The Defence Committee accepted this. It concluded that “Fadec was not regarded as safety critical”.
But nearly a year after the Defence Committee had finished its deliberations on the lessons learned from the Mull crash, the MoD wrote to Key: “You asked if Boeing treated Fadec as flight safety critical. They did so, but assessed that the risk of catastrophic failure was mitigated by the design of the Fadec system.”
Despite this clear, but perhaps inadvertent, misleading of the Defence Committee, the MoD has continued to insist that the Fadec is not safety critical by the department’s own standards.
In a letter to Key, the MoD said: “The department assesses that Fadec is not safety critical by the standards which MoD authorities work, namely that failure ‘would’ lead to catastrophe, as opposed to the US definition ‘could’ so result.”
This is not what is said in the main software defence standard to which MoD authorities work, namely MoD’s “00-55”, which sets a benchmark for safety-related software in defence equipment.
In its introductory paragraph (published on the MoD’s Web site) the 00-55 standard says: “Safety-critical software is software that relates to a safety-critical function or system, therefore software of the highest safety integrity level (S4), the failure of which could [our emphasis] cause the highest risk to human life.”
There is no room to list here the many other examples of ministers and MPs receiving incorrect and conflicting information from the MoD.
But it is not only the MoD that has been inattentive to the facts when facing a potential political embarrassment.
Department in denial?
Computer Weekly has published many articles about different government departments that have given incorrect or misleading answers to parliamentary questions on matters related to computer problems.
These problems, including those at the Passport Agency, Post Office and National Air Traffic Control Services, have resulted in losses of tens of millions of pounds or projects that failed to deliver value for money.
In the case of the Chinook Mk2’s Fadec and the crash on the Mull of Kintyre, there is no evidence of any conspiracy to misinform.
It appears that the MoD has simply sought to protect itself from possible criticism that it may have rushed the Fadec into operational service amid a hasty dismissal of expert concerns, with possible tragic consequences. This could be a classic symptom of a department in denial, a problem that has tended to characterise unsuccessful IT projects in the past.
Any department is likely to seek to preserve its own interests over the public interest if there is any conflict between the two. It is then the job of watchdogs such as the Public Accounts Committee to hold the departments accountable.
Indeed, Computer Weekly argued, in evidence to the Cabinet Office earlier this year, that many major government computer disasters could be avoided if they were subject throughout the procurement and afterwards to rigorous and truly independent scrutiny.
But no department likes independent auditors looking over its shoulder. In the case of the Chinook’s multimillion-pound Fadec programme, the accounts committee report has come about only because its MPs and the chairman David Davis decided to launch an in-depth independent inquiry.
But it comes after a Scottish Fatal Accident Inquiry cleared the names of the dead pilots; a 140-page report by Computer Weekly which said there had been a cover-up of Fadec software problems; and the formation of the Mull of Kintyre pressure group led by Lord Chalfont and comprising some of the most senior MPs and peers in parliament.
It also comes after an investigation by pilots in the Royal Aeronautical Society, who questioned the safety of the Chinook Mk2, and an inquiry by the Lord Advocate’s office in Scotland, which studied Computer Weekly’s evidence and found that the Fadec problems could lend weight to suggestions that the pilots were not necessarily to blame.
In addition, there has been an early-day motion signed by 89 MPs calling for a new inquiry, numerous media articles, campaigning by the families of the dead pilots and their representative Georgie Vestey and a series of documentaries by Channel 4 News. All of this has taken six years.
If this is what is meant by departmental accountability, there is a fundamental flaw in the process of government. It could also mean that government departments are destined to repeat the computer failures of the past.
Key points of Computer Weekly evidence given to and published by the Public Accounts Committee
Ultimately, there are two issues. Did the Fadec pass all the tests set for it by the procurement team, which rightly included Boscombe Down and EDS-Scicon? Or, for a host of operational reasons, was Fadec rushed into service amid a hasty dismissal of expert concerns, with possible tragic consequences?
- The MoD has failed its own defence team in the arbitration hearing between the MoD and the Fadec’s supplier Textron Lycoming. Once details of the MoD’s successful case against Textron were leaked by the media, the MoD attacked its own evidence and its expert witness Malcolm Perks.
- The wider issues include what if any checks exist to stop departments sidelining any independent advice that goes against the grain, as the MoD did with the EDS-Scicon report.
- Will software ever be found to be a definite cause of a major fatal accident? Only manufacturers understand their software enough to say if it caused or contributed to a crash, but will they tell?
- As in this case, and often after a major accident, departments will seek to protect an IT supplier from criticism rather than allow some of the opprobrium to settle on the department’s lap.
- The Fadec was procured without open competition and with the RAF kept at a distance from the development process. Studies of IT disasters show the need for the work of vigilant software’s developers to be scrutinised almost constantly by users.
- The software should not be financed by the developer but by an independent company that will audit the work and the results. In the case of the Fadec, the project was financed initially by the subcontractors who were also the developers.
Fadec problems – the history
1985: Almost at the start of the development, there were unrealistic expectations. Delivery of the Fadec was promised within 23 months and was delivered several years late.
1986: RAF specialists complain of secrecy over Fadec project.
1989: After four years of development, Fadec has its first series of tests fitted to an MoD Chinook. In a disaster that is described in an MoD report as “potentially catastrophic”, a Chinook is nearly destroyed by a Fadec-related engine surge.
1989: Fadec is modified. A letter to a subcontractor from the Fadec’s supplier Textron Lycoming says: “... the eyes of the world ... Boeing/ RAF/MoD, will be focusing on Fadec. We absolutely cannot afford another problem if we are retain Fadec’s technical credibility.”
1990: The MoD begins legal action against Textron over the 1989 incident. The first paragraph of the writ says the engine surge was “caused by respondent Textron’s faulty design of a computerised engine fuel control device Fadec”.
1993: An assessment on the modified Fadec by contractor EDS-Scicon is abandoned because of the large number of anomalies found – 485 after an analysis of less than 18% of the code. EDS-Scicon says a potential flaw in the Fadec’s main computer “may cause incorrect operation of the Fadec”.
1993: Boscombe Down, the MoD’s airworthiness assessors refuse to give the Fadec an unqualified approval unless the Fadec software is rewritten. But MoD and RAF over-rule Boscombe Down and put the Chinook Mk2 into operational service without software rewrite.
Jan to May 1994: MoD procurement executive raises “safety case issues” over the Fadec. Chinook pilots experience “flight critical” problems including unexpected engine surges, engine run-downs and cockpit warning lights. Boscombe Down suspends trials flights because of Fadec concerns.
1 June 1994: For the second time in five months, Boscombe Down suspends trials flights over Fadec concerns.
2 June 1994: Chinook ZD576 crashes on the Mull of Kintyre.
3 June 1994: Boscombe Down says in a memo that the Fadec has been shown to be “unacceptable” and is “unsuitable for its intended purpose”.
1994: A top RAF officer, in a draft memo, attacks Boscombe Down’s Fadec reservations as “quite incredible”. Boscombe Down’s “ongoing stance towards the Mk2 contrasts sharply with the considerable efforts being made by the front line to bring the aircraft into service and maintain a capability”.
Late 1994: Improvements to Fadec by Textron include replacement of the system’s central computer after the concern expressed by EDS-Scicon.
1995: MoD procurement executive memo says that Fadec suffered a “series of problems between February and July 1994” – the period covered by the Mull crash.
1995: MoD wins $3m damages against Textron over the 1989 incident.
1995: Two air marshals over-rule the inconclusive report of an RAF Board of Inquiry and find that the pilots of ZD576 were grossly negligent.
1996: A Scottish Fatal Accident Inquiry says there is not enough evidence to blame the pilots.
1998: Defence Committee rejects concerns over Fadec.
1999: Computer Weekly publishes RAF Justice, a 140-page report on a cover-up of Fadec problems. Nearly 90 MPs sign an early-day motion for an investigation into Computer Weekly’s findings.
2000: Senior ministers including Tony Blair reject calls for new inquiry. Public Accounts Committee publishes damning report on Fadec.
How readers see the Chinook scandal
Computer Weekly has received hundreds of emails and letters giving support for the campaign for an independent inquiry into the circumstances of the Mull of Kintyre crash. Here are some of them:
Gordon Johnston, former flight lieutenant: May I offer my wholehearted support for your campaign to overturn the gross negligence verdict of Flight Lieutenants Cook and Tapper. As an ex-RAF pilot and current systems programmer, I feel I am in a position to make a balanced judgement of the case. It is inconceivable that the pilots flew into the Mull due to negligence. Something serious must have happened to cause the accident, I believe a Fadec malfunction is highly likely. Keep up your campaign.
Mic L Porter, consulting ergonomist, Tyne and Wear: Good luck with your campaign – not only for this specific case, but for the correct investigation and reporting of “invisible”/counter-intuitive software behavioural problems.
Ciaran Brady, technical director, Freehand: After reading your report, I find myself appalled at the injustice of the decision to blame the two pilots. The larger picture becomes one of mission-critical systems suppliers getting off the hook should there be a failure. The users would get the blame, which is absolutely astonishing.
You have raised trade journalism to investigative journalism at its finest, and you should be very proud indeed.
David Bird, executive director, London: I should like to register my disquiet and disgust at the way in which this whole tragic incident has been handled by the government and to urge them to re-open the inquiry to establish the truth of the matter. It is grossly unfair that the pilots should have been blamed with no evidence to support such a finding, and so much evidence that contradicts it.
Robert Lawson, European Technical Training: I am extremely grateful to your publication for continuing to burn the candle for Rick [Cook] and John [Tapper] and earnestly hope that in doing so you manage to clear their names and expose those who should hold ultimate responsibility for their deaths.
Mike Hancock MP: Computer Weekly is to be commended on its coverage of the crash of Chinook ZD576 on the Mull of Kintyre, which has brought to light how unsafe the verdict of gross negligence against the pilots is.
In order to restore their reputation, I have tabled an Early Day Motion in Parliament which calls for the RAF Board of Inquiry to be reopened – I urge all readers of Computer Weekly who have followed this campaign to contact their MP and ask them to sign EDM 796. If you do not know your MP’s name, you can find it at www.parliament.uk.
Steven J Wooff, software engineer, Surrey: I am a senior software engineer with many years’ experience of aviation-related and avionic systems software. After reading the information published by Computer Weekly, I would like to add my own voice to those demanding a reinvestigation of the crash. Although I am in general an advocate of computer-controlled systems, I am aware of a number of reported instances where the failure of such systems has resulted in serious incident or actual aircraft losses.
It does indeed appear that these systems are now of such complexity that analysis of contributing factors by non-expert investigators or by substantially partial manufacturers does the industry and air transport in general no service.