At first glance, today's report by MPs of the PublicAccounts
Committee (PAC) tells the story of flawed engine control software
on the Chinook Mk2 helicopter.
The report finds the helicopter's poorly-functioning Full
Authority Digital Engine Control (Fadec) system could not be ruled
out as a factor in the RAF's worst peacetime accident, when Chinook
ZD576 crashed into the Mull of Kintyre in June 1994, killing four
crew and 25 intelligence specialists.
The PAC's findings vindicates a campaign by Computer
Weekly and others who have sought to establish that pilot
negligence may not have the cause of the accident.
However the wider issues raised by the PAC's report will be
familiar to those who have studied IT problems across all
government departments. In the report there is evidence that the
MoD has suppressed evidence that could have led to its own
political embarrassment, at the expense of an open
investigation.
Parts of the PAC report conflict in tone and substance to
statements made by Prime Minister Tony Blair, Defence Secretary
Geoffrey Hoon, various Labour ministers including John Spellar,
Lewis Moonie, Doug Henderson and John Reid, and the permanent
secretary at the MoD, Kevin Tebbit.
At various times over the past three years, Blair, his ministers
and Tebbit have upheld the decision of two air marshals. In 1995
they found that the pilots of Chinook ZD576, flight lieutenants
Rick Cook and Jonathan Tapper were, without any doubt whatsoever,
grossly negligent. The pilots flew too low and too fast into a
cloud-covered area on the Mull of Kintyre, said ministers on the
basis of briefings by officials.
In the previous Tory Government, ministers made similar comments
at the time, endorsing the findings of the two air marshals. Again,
their knowledge of the crash and circumstances surrounding it, was
based on the strength on their briefings by officials.
The Defence Select Committee, too, based on briefings from
officials, came to views that conflicted with today's PAC
report.
In 1998 the Defence Committee, for example, attached no safety
significance to the concerns over the Fadec expressed by the MoD's
software assessors at Boscombe Down. Defence MPs went further. They
attacked Boscombe Down for its "failure" to give final approval to
the Fadec. They said this was a "management failure," adding, "We
are persuaded by the evidence that this absence of approval raises
no safety-critical questions."
But today's PAC report finds that Fadec could have caused the
crash - nobody knows for certain whether it did or did not.
It is highly unusual for the PAC's findings to be in conflict
with another select committee. But there is a worrying difference
in the way the two committees, and ministers, have set out
gathering and evaluating the evidence.
In an investigation into the circumstances of the Mull crash
that has lasted several months, the PAC took evidence from eclectic
specialist sources. These included the MoD, Computer Weekly,
and Malcolm Perks, the MoD's expert witness in the department's
legal action against Textron Lycoming, the supplier of the
Chinook's Fadec.
Further technical evidence was given to the committee by a
Chinook unit test pilot, Squadron Leader Robert Burke, who said the
Fadec system was so unreliable at the time of the Mull crash that
systems were being regularly replaced on the Chinook Mk2.
In contrast, Blair, his ministers, Tebbit and the Defence
Committee, based their findings on briefings that were almost
entirely from MoD officials and military staffs.
Yet evidence of the MoD's lack of objectivity over matters
related to the Mull crash is copious. In July, based on briefings
by officials, Blair wrote to independent MP Martin Bell assuring
him that the Government was being "open" about the circumstances of
the crash and the conduct of the inquiry. Blair may have been
unaware that, a few weeks earlier, the MoD had refused a request by
the families of the dead pilots for a copy of Boscombe Down's
official assessment of the Chinook Mk2 prior to its introduction
into service.
The MoD has also refused to disclose, even to the PAC, details
of the Fadec-related incidents before the crash.
In addition, the MoD has declined to publish the independent
assessment of the Fadec software by IT defence contractor
EDS-Scicon.
And the MoD has told MP Robert Key that it is withholding a
"white paper" by Fadec supplier Textron Lycoming which responds to
the concerns expressed by EDS.
Blair also told Bell, on the advice of officials, that the
Government was being "straightforward" in matters related to the
crash. But Key recently received a parliamentary reply which quoted
liberally from internal MoD legal papers on the litigation against
Textron Lycoming, while another MP was refused any access to the
same documents because the MoD said they were confidential.
The confusion over what was fact and what was not, in the MoD's
eyes, was particularly noticeable at a hearing of the Defence
Committee in 1998.
MPs on the committee were told in a MoD statement that "Boeing
did not consider the Fadec to be flight safety critical because the
engines on a Chinook are not considered to be safety critical". The
Defence Committee accepted this. It concluded that "Fadec was not
regarded as safety critical".
But nearly a year after the Defence Committee had finished its
deliberations on the lessons learned from the Mull crash, the MoD
wrote to Key saying, "You asked if Boeing treated Fadec as flight
safety critical. They did soÉ but assessed that the risk of
catastrophic failure was mitigated by the design of the Fadec
system".
Despite this clear, but perhaps inadvertent misleading of the
Defence Committee, the MoD has continued to insist that the Fadec
is not safety critical by the department's own standards.
In a letter to Key the MoD said, "Éthe department assesses that
Fadec is not safety critical by the standards which MoD authorities
work, namely that failure 'would' lead to catastrophe, as opposed
to the US definition 'could' so result".
This is not what is said in the main software defence standard
to which MoD authorities work, namely MoD's "00-55" which sets a
benchmark for safety related software in defence equipment.
In its introductory paragraph (published on the MoD's Web site)
the 00-55 standard says, "Safety critical software is software that
relates to a safety critical function or system, therefore software
of the highest safety integrity level (S4), the failure of which
could [our emphasis] cause the highest risk to human life".
There is no room to list here the many other examples of
ministers and MPs receiving incorrect and conflicting information
from the MoD.
But it is not only the MoD that has been inattentive to the
facts when facing a potential political embarrassment.
Computer Weekly has published many articles about
different government departments that have given incorrect or
misleading answers to parliamentary questions on matters related to
computer problems. These problems, including those at the Passport
Agency, Post Office and National Air Traffic Control Services, have
resulted in losses of tens of millions of pounds or projects that
failed to deliver value for money.
In the case of the Chinook Mk2's Fadec and the crash on the Mull
of Kintyre, there is no evidence of any conspiracy to
misinform.
It appears that the MoD has simply sought to protect itself from
possible criticism that it may have rushed the Fadec into
operational service amid a hasty dismissal of expert concerns, with
possible tragic consequences. This could be a classic symptom of a
department in denial, a problem that has tended to characterise
unsuccessful IT projects in the past.
Any department is likely to seek to preserve its own interests
over the public interest if there is any conflict between the two.
It is then the job of watchdogs such as the public accounts
committee to hold the departments accountable.
Indeed Computer Weekly argued, in evidence to the Cabinet
Office earlier this year, that many major government computer
disasters could be avoided if they were subject throughout the
procurement and afterwards to rigorous and truly independent
scrutiny.
But no department likes independent auditors looking over its
shoulder. In the case of the Chinook's multimillion pound Fadec
programme, the accounts committee report has come about only
because its MPs and the chairman David Davis decided to launch an
in-depth independent inquiry.
But it comes after a Scottish Fatal Accident Inquiry cleared the
names of the dead pilots; a 140-page report by Computer
Weekly which said there had been a cover-up of Fadec software
problems; and the formation of the Mull of Kintyre pressure group
led by Lord Chalfont and comprising some of the most senior MPs and
peers in parliament.
It also comes after an investigation by pilots in the Royal
Aeronautical Society, who questioned the safety of the Chinook Mk2,
and an inquiry by the Lord Advocate's office in Scotland which
studied Computer Weekly's evidence and found that the Fadec
problems could lend weight to suggestions that the pilots were not
necessarily to blame. In addition, there has been an early-day
motion signed by 89 MPs calling for a new inquiry, numerous media
articles, campaigning by the families of the dead pilots and their
representative Georgie Vestey and a series of documentaries by
Channel 4 News. All this has taken six years.
If this is what is meant by departmental accountability, there
is a fundamental flaw in the process of government. It could also
mean that government departments are destined to repeat the
computer failures of the past.
Key points of Computer Weekly evidence
Key points of the Computer Weekly evidence given to and
published by the Public Accounts Committee:
Ultimately there are two issues: Did the Fadec pass all
the tests set for it by the procurement team which rightly included
Boscombe Down and EDS-Scicon? Or, for a host of operational
reasons, was Fadec rushed into service amid a hasty dismissal of
expert concerns, with possible tragic consequences?
- The MoD has failed its own defence team in the arbitration
hearing between the MoD and the Fadec's supplier Textron Lycoming.
Once details of the MoD's successful case against Textron were
leaked by the media, the MoD attacked its own evidence and its
expert witness Malcolm Perks.
- The wider issues include what if any checks exist to stop
departments sidelining any independent advice that goes against the
grain, as the MoD did with the EDS-Scicon report.
- Will software ever be found to be a definite cause of a major
fatal accident? Only manufacturers understand their software enough
to say if it caused or contributed to a crash - but will they
tell?
- As in this case, and often after a major accident, departments
will seek to protect an IT supplier from criticism rather than
allow some of the opprobrium to settle on the department's
lap.
- The Fadec was procured without open competition, and with the
RAF kept at a distance from the development process. Studies of IT
disasters show the need for the work of vigilant software's
developers to be scrutinised almost constantly by
end-users.
- The software should not be financed by the developer but by an
independent company that will audit the work and the results. In
the case of the Fadec, the project was financed initially by the
subcontractors who were also the developers.
Fadec problems - the history
1985: Almost at the start of the development there were
unrealistic expectations. Delivery of the Fadec was promised within
23 months and was delivered several years late.
1986: RAF specialists complain of secrecy over Fadec
project.
1989: After four years of development, Fadec has its
first series of tests fitted to an MoD Chinook. In a disaster that
is described in an MoD report as "potentially catastrophic," a
Chinook is nearly destroyed by a Fadec-related engine surge.
1989: Fadec is modified. A letter to a subcontractor from
the Fadec's supplier Textron Lycoming says: "... the eyes of the
world ... Boeing/ RAF/MoD, will be focusing on Fadec. We absolutely
cannot afford another problem if we are retain Fadec's technical
credibility".
1990: The MoD begins legal action against Textron over
the 1989 incident. The first paragraph of the writ says the engine
surge was "caused by respondent Textron's faulty design of a
computerised engine fuel control device Fadec".
1993: An assessment on the modified Fadec by contractor
EDS-Scicon is abandoned because of the large number of anomalies
found - 485 after an analysis of less than 18% of the code.
EDS-Scicon says a potential flaw in the Fadec's main computer "may
cause incorrect operation of the Fadec".
1993: Boscombe Down, the MoD's airworthiness assessors
refuse to give the Fadec an unqualified approval unless the Fadec
software is rewritten. But MoD and RAF over-rule Boscombe Down and
put the Chinook Mk2 into operational service without software
rewrite.
Jan to May 1994: MoD procurement executive raises "safety
case issues" over the Fadec. Chinook pilots experience "flight
critical" problems including unexpected engine surges, engine
run-downs and cockpit warning lights. Boscombe Down suspends trials
flights because of Fadec concerns.
1 June 1994: For the second time in five months, Boscombe
Down suspends trials flights over Fadec concerns.
2 June 1994: Chinook ZD576 crashes on the Mull of
Kintyre.
3 June 1994: Boscombe Down says in a memo that the Fadec
has been shown to be "unacceptable" and is "unsuitable for its
intended purpose".
1994: A top RAF officer, in a draft memo, attacks
Boscombe Down's Fadec reservations as "quite incredible". Boscombe
Down's "ongoing stance towards the Mk2 contrasts sharply with the
considerable efforts being made by the front line to bring the
aircraft into service and maintain a capability".
Late 1994: Improvements to Fadec by Textron include
replacement of the system's central computer after the concern
expressed by EDS-Scicon.
1995: MoD procurement executive memo says that Fadec
suffered a "series of problems between February and July 1994" -
the period covered by the Mull crash.
1995: MoD wins $3m damages against Textron over the 1989
incident.
1995: Two air marshals over-rule the inconclusive report
of an RAF Board of Inquiry and find that the pilots of ZD576 were
grossly negligent.
1996: A Scottish Fatal Accident Inquiry says not enough
evidence to blame the pilots.
1998: Defence Committee rejects concerns over Fadec.
1999:Computer Weekly publishes RAF Justice, a
140-page report on a cover-up of Fadec problems. Nearly 90 MPs sign
an early-day motion for an investigation into Computer
Weekly's findings.
2000: Senior ministers including Tony Blair reject calls
for new inquiry. Public Accounts Committee publishes damning report
on Fadec.
How readers see the Chinook scandal
Computer Weekly has received hundreds of e-mails and
letters giving support for the campaign for an independent inquiry
into the circumstances of the Mull of Kintyre crash. Here are some
of them:
"May I offer my wholehearted support for your campaign to
overturn the gross negligence verdict of Flight Lieutenants Cook
and Tapper. As an ex-RAF pilot and current systems programmer, I
feel I am in a position to make a balanced judgement of the case.
It is inconceivable that the pilots flew into the Mull due to
negligence. Something serious must have happened to cause the
accident I believe a Fadec malfunction is highly likely. Keep up
your campaign.
Gordon Johnston, former flight lieutenant
Good luck with your campaign - not only for this specific
case but for the correct investigation and reporting of
"invisible"/counter-intuitive software behavioural
problems.
Mic L Porter, consulting ergonomist, Tyne and Wear
After reading your report, I find myself appalled at the
injustice of the decision to blame the two pilots. The larger
picture becomes one of mission-critical systems suppliers getting
off the hook should there be a failure. The users would get the
blame, which is absolutely astonishing.
You have raised trade journalism to investigative journalism
at its finest, and you should be very proud indeed.
Ciaran Brady, technical director, Freehand
I should like to register my disquiet and disgust at the way
in which this whole tragic incident has been handled by the
Government and to urge them to re-open the inquiry to establish the
truth of the matter. It is grossly unfair that the pilots should
have been blamed with no evidence to support such a finding, and so
much evidence that contradicts it.
David Bird, executive director, London
I am extremely grateful to your publication for continuing to
burn the candle for Rick [Cook] and John [Tapper] and earnestly
hope that in doing so you manage to clear their names once and for
all and expose those who should hold ultimate responsibility for
their deaths.
Robert Lawson, European Technical Training
Computer Weekly is to be commended on its coverage of
the crash of Chinook ZD576 on the Mull of Kintyre, which has
brought to light how unsafe the verdict of gross negligence against
the pilots is.
In order to restore their reputation, I have tabled an Early
Day Motion in Parliament which calls for the RAF Board of Inquiry
to be reopened É I urge all readers of Computer Weekly who
have followed this campaign to contact their MP and ask them to
sign EDM 796. If you do not know your MP's name, you can find it
atwww.parliament.uk
Mike Hancock MP
I am a senior software engineer with many years' experience
of aviation-related and avionic systems software. After reading the
information published by Computer Weekly I would like to add
my own voice to those demanding a reinvestigation of the crash.
Although I am in general an advocate of computer-controlled
systems, I am aware of a number of reported instances where the
failure of such systems has resulted in serious incident or actual
aircraft losses.
It does indeed appear that these systems are now of such
complexity that analysis of contributing factors by non-expert
investigators or by substantially partial manufacturers does the
industry and air transport in general no service.
Steven J Wooff, software engineer, Surrey
Leader