Letters from software experts to Computer Weekly show the Ministry
of Defence omitted to mention it sanctioned the very testing
methods it criticised its own IT assessors for using. Tony Collins
reports
Ever since the crash of a Chinook helicopter on the Mull of
Kintyre in 1994, the Ministry of Defence has defended the quality
of the Chinook helicopter's safety-critical engine control software
by adopting a twin-track approach.
First, it has staunchly defended its decision to put the
Chinook, with a new Full Authority Digital Engine Control (Fadec)
system into service.
Secondly, it has attacked a technique used by its IT and
airworthiness assessors at Boscombe Down for validating
safety-critical software, known as static code analysis, which
found a high incidence of flaws in the Fadec software.
Over the past three years, the MoD has told two Parliamentary
committees, ministers and the public spending watchdogs - the
National Audit Office - that static code analysis was inappropriate
for testing the Chinook Mk2's Fadec. The technique, said the MoD,
was used by the nuclear power industry rather than the defence
community.
Now IT specialists have written to Computer Weekly
revealing that the MoD was wrong in its repeated public statements
concerning static code analysis. A branch of the MoD in fact
developed the disputed methodology. And it was developed
specifically to provide an independent method of validating
safety-critical software, the quality of which may otherwise have
been impossible to ascertain.
This means that one of the main justifications for the decision
to blame the pilots for the crash on the Mull of Kintyre - that
there was no evidence of any software problems - has now been
undermined.
Specialists say the principal reason that static code analysis
is a recommended technique of the ministry is that it detects
anomalies that may be missed by other validation methods -
anomalies that may cause the software to behave unpredictably.
In various public statements, the ministry has argued that
Boscombe Down was unable to validate the Fadec software because of
its use of static code analysis. If this argument is accepted then
Boscombe Down, as well as the independent contractor EDS-Scicon,
were using the wrong technique to assess the software.
Were this true it would disaffirm the findings of Boscombe Down
that the software was unacceptable. It would also belittle the
evaluation by EDS-Scicon that found a high incidence of anomalies
in the software.
it is now clear that static code analysis is not only a defence
standard but is MoD best practice for validating safety-critical
software in aviation systems. This establishes that the techniques
used by Boscombe Down and EDS were the correct ones after all.
Therefore, Boscombe Down's findings have a credibility that has
been wrongly undermined by the MoD.
The ministry's incorrect statements on static code analysis may
also leave it open to the accusation that it has misled the
National Audit Office, and Parliament.
On the basis of its MoD briefings, the National Audit Office
(NAO) reported on the anomalies found by EDS-Scicon, but added that
the contractor had used static code analysis, which it said was an
"internal Boscombe Down policy, not supported by defence
standards".
At a hearing of the Public Accounts Committee earlier this
month, the MoD's top civil servant, Permanent Under-Secretary Kevin
Tebbit, said he agreed with the NAO's statement on static code
analysis.
He said static code analysis was Boscombe Down's "preferred"
method of analysing the software. He also claimed that it was
"inappropriate" for verifying or validating Fadec.
When asked by the committee why Boscombe Down was using an
inappropriate method of testing the software, Tebbit said, "I do
know that it was used in the nuclear power industry and was applied
in this context".
If the ministry has inadvertently rather than deliberately
denigrated Boscombe Down, it has done so consistently.
In 1998, MPs on the Commons' Defence Committee were told by the
Ministry of Defence that, "static code analysis is... a requirement
placed by British Nuclear Fuels on the safety of a nuclear
system".
The Defence Committee concluded that Boscombe Down's "failure"
to give final approval to the Fadec software was a "management
failure".
In July last year, a senior civil servant at the Secretariat
(Air Staff) of the Ministry of Defence wrote in a letter that,
"Static code analysis does not validate the performance of the
software and the department therefore had no requirement for
it".
In August last year, the Ministry wrote to Defence Committee MP
Michael Hancock saying that "Boscombe Down's preferred method of
examination is static code analysis, a system of verification not
widely in use but employed in the nuclear industry."
Also last year, the House of Lords was told by the ministry
that, "Boscombe Down indicated a wish to assess the design of the
Fadec software using static code analysis - a methodology used by
the nuclear industry."
None of this gave a true impression of the importance to the MoD
of static code analysis. Computer Weekly has a 1991 version of
defence standard 00-55 which specifies static analysis for the
validation of safety-critical software used in aircraft and other
defence equipment.
And a letter to Computer Weekly from Martyn Thomas, one
of the UK's most respected independent specialists in the field of
safety-critical software, says that static code analysis was
developed by the Royal Signals and Radar Estab-lishment, now the
Defence Evaluation Research Agency, an agency of the Ministry of
Defence.
"RSRE developed the secret technology so that they could verify
security-critical software," he says in a letter.
"Work on static analysis was declassified as a matter of public
policy, precisely so that it could be used on safety-critical
software - such as the Chinook Fadec," he adds.
Another detailed letter to Computer Weekly from a
software specialist says that the Lockheed C130J Hercules aircraft
is undergoing static code analysis for the purposes of UK flight
certification.
And separately, Bath-based Praxis Critical Systems, which
develops software for the defence, banking and other industries,
told Computer Weekly that static code analysis has been used to
validate safety-critical software in aircraft such as the Tornado
F3 and the Eurofighter.
It is also used in safety-critical aircraft support functions,
such as the Sholis system that helps helicopters to land safely on
ships. In addition, it has been used by the Government's
communications centre GCHQ to spot viruses in software.
Praxis said the importance of static code analysis, which
involves testing code without executing it, lies in its ability to
highlight anomalies and faults that could remain undetected by
dynamic testing.
It said dynamic testing, which involves executing the code, can
highlight only a small number of potential problems. This is
because it does not check paths through the software that can be
taken by executable code.
But it was not only in the area of static code analysis that the
Public Accounts Committee was given the wrong impression by the
ministry. Tebbit mistakenly told the committee that the Fadec
software was not designated as safety-critical, although Boeing had
said it was.
The ministry also told the committee that Fadec had undergone
70,000 hours of testing. But it did not say that this was the
cumulative total for a number a different versions of Fadec. And
the 70,000 hours related mainly to a version of Fadec used by the
US Army. This was not an identical system to Fadec as used in RAF
Chinook helicopters.
But even if the ministry may have omitted to tell MPs the whole
story of Fadec, and what it has told them may have sometimes been
misleading, does this matter?
Some people believe it does. If Parliament accepts incorrect
statements from the MoD, it suggests that nobody can ever be held
accountable for the decision to put 25 of the UK's top intelligence
officers into a helicopter which had known flaws in its
safety-critical software.
The technique, said the MoD, was used by the nuclear power
industry rather than the defence community
A branch of the MoD in fact developed the disputed
methodology
This establishes that the techniques used by Boscombe Down and
EDS were the correct ones after all
More Chinook news