Hospital software failure shows Y2K bug can still bite

Bill Goodwin reports on the NHS IT inquiry that holds a warning for those who think they can forget Y2K

Bill Goodwin reports on the NHS IT inquiry that holds a warning for those who think they can forget Y2K

An inquiry into a Y2K problem at a Sheffield hospital has turned the spotlight on the qualifications of IT staff in NHS laboratories, who carry out vital work for thousands of surgeons and GPs.

The investigation into the error at Sheffield's Northern General Hospital highlighted concerns about the home-grown nature of the IT support in NHS laboratories.

Many laboratories prefer to appoint IT managers from staff familiar with laboratory procedures, rather than rely on the hospital's central IT department, the inquiry found. The practice had "undoubtedly denied them access to staff with training in formal methods of system management, system testing and change control", it said.

This was one of the factors that lead to 158 women at the hospital being given inaccurate test results. They were wrongly told that there was little chance that their babies would have Down's syndrome.

With the hospital facing legal action, the question is how the error was able to slip through the NHS' carefully managed Y2K programme.

Sheffield began using a computer model to analyse prenatal tests for Down's syndrome in 1998. Researchers had devised a complex calculation that could predict the likelihood of Down's syndrome occurring from a blood test. They used a desktop PC and a specially-written Basic program, to work out the chances of the babies having the condition.

Later, researchers made significant improvements to the algorithm by factoring the weight of the patient into the calculation. It was this work that raised the first questions about the way IT was managed in the pathology laboratory. A patient complained when the results of two Down's syndrome tests came back with wildly differing results. An internal investigation identified an error in the testing software. The program should have used an average weight if the actual weight of the patient was unknown. Instead it assumed a weight of zero.

The hospital's IT department began a thorough review of the immunology department's IT systems. Its report made several recommendations including that new software should be loaded into a demonstration system and tested before use; all software changes should be logged; and change control procedures be developed.

A follow-up report by the hospital's internal auditors two years later reached similar conclusions. It called for full testing of software when modifications were made and improvements in documentation. It also raised concerns that the immunology department relied on one member of staff for its hardware and Unix support. There were no contingency plans if he was not available.

In 1994 the pathology department moved the Down's screening program to another computer system, called Pathlan, originally supplied and maintained by ICL, and run on a DRS 6000 computer. ICL had discontinued support for the system and Sheffield had passed its support contract to another hospital trust, Hartlepool and Peterlee. Hartlepool subcontracted the work to a self-taught hospital IT specialist, referred to in the report as Mr W.

Mr W had been given formal training by ICL in the Apogee programming language used by the Pathlan system. Although he did not understand how the existing Basic algorithm worked, Sheffield commissioned him to convert the code to run on Pathlan. The program he developed used existing code with a two-digit date field to calculate the age of patients. Although the software was built just six years before 2000, it was not millennium compliant.

The pathology laboratory was jointly funded by the NHS and the University of Sheffield and tended to operate at arm's length from the rest of the Northern General Hospital Trust. The trust's IT department maintained the laboratory's PCs, but the Pathlan system was the responsibility of the computer services manager in the lab, Mr R. The laboratory did not follow the hospital's normal IT procurement procedures. The department had an informal relationship with Hartlepool, and, according to the inquiry, this extended to software testing and acceptance of enhancements and upgrades.

When Sheffield Northern General Trust began its Y2K programme in 1996 the Pathlan system fell outside its remit, and Mr R assumed responsibility for its compliance. He wrote to ICL, the original supplier and received assurances that the hardware and operating system were compliant. Hartlepool gave verbal assurances that there were unlikely to be any problems.

Sheffield had signed a contract to buy a replacement system for Pathlan, the Telepath 2000, supplied by CDS. The system was Y2K compliant and was due to go live on 23 July 1999. But as the programme slipped more and more behind schedule and the department realised it would need to keep Pathlan online.

It became clear to the hospital that Pathlan was not as complaint as the suppliers had first thought. In May 1999, the pathology department agreed to a shut-down to install a millennium compliant version of the Unix operating system.

Hartlepool had also identified a use of Apogee language, which meant the way it calculated dates was not millennium compliant. Mr W informed the pathology lab that 138 program files in Pathlan would need to be upgraded. In late November the new versions of the files were added to the Pathlan system.

Mr R rolled the clock on Pathlan forward as a final check. All seemed well but, crucially, the test failed to assess the impact of the date changes to the women's age calculations. Mr R did not realise that Hartlepool had not had the time or the resources to check the Down's screening module.

"The Pathlan system is Y2K compliant for practical purposes," he wrote in a draft report to his manager, adding that it was possible there could be "some minor problems not yet uncovered due to the system not being fully tested". The manager accepted the draft report and filed it. The project was not formally signed off and no copy was sent to the hospital's Y2K board for consideration.

On 4 January 2000 further checks were carried out by one of the lab workers. She was asked to recalculate a few results to see whether they were the same as the year before. She was given no guidance how to do this but picked five tests at random and changed the babies' gestation times. She told her supervisor that there was no change but the test failed to double-check the age calculation.

Throughout January laboratory staff discovered a series of minor date errors in Pathlan. The changes were logged and fixed, but none was thought to be serious. At the end of January, the laboratory staff considered the system Y2K compliant.

But some of the midwives were not sure. That month a sister at one of Sheffield's client hospitals raised concerns that the number of positive tests for Down's syndrome was lower than she expected. In April, she phoned the department again to express her concern but staff assured her that there were no problems. A report by the NHS quality assurance department, also went unnoticed by the laboratory workers, who were battling with chronic staff shortages. The figures in the report might have shown them a discrepancy between Sheffield's results and those of other labs.

Later that month, one of the department's senior staff happened to glance through the department's log book. The results did not look right. He asked a medical officer to audit the log book and report back, but the officer was busy with other work and the report never arrived.

In May, a midwife at another hospital became concerned at the low number of high-risk test results she was receiving. She spoke to a medical officer. He did not consider it important enough to refer to his manager. Two weeks later, the midwife phoned again and spoke to the manager of the pathology department.

By 23 May the manager had the log book audit on his desk. It was clear that something was wrong. Hartlepool's Mr W tapped into the Pathlan system via remote modem and quickly found the error. Nearly 158 pregnant women had been wrongly told there was a low risk that their babies had Down's syndrome. The program had mistakenly calculated their ages as negative numbers.

The hospital quickly set about contacting and re-testing all of the women concerned. As a result of the problem, however, crucial amniotic tests, which should have been performed early in pregnancies, were delayed. The hospital now faces legal action.

An independent inquiry identified a catalogue of management and technical problems that contributed to the error. Its findings will have ramifications for IT professionals throughout laboratory systems in the NHS, and beyond.

Like Sheffield, the inquiry concluded that many NHS labs rely on home-grown IT staff for support rather than professionally qualified staff from their hospitals' IT departments. Ideally, it says, laboratory systems managers should have professional qualifications in biosciences and IT. This is the case in other professions such as engineering and finance.

The report found that the original screening software was poorly written, even by the standards of the time. It lacked internal error-trapping that could have detected problems at an early stage. The failures were compounded by a poor user interface, which meant that intermediate calculations which would have alerted staff immediately were not displayed.

Crucially the department had failed to learn from the earlier weight calculation error, despite two internal reports, and reassurances by the hospital's chief executive officer. It was still dependent on a single technician with expert knowledge about the system at Hartlepool hospital. And although Sheffield gave assurances that it would introduce validation checks to improve software change controls, this instruction was never passed on to Hartlepool.

Hartlepool said, "We carried out a check of all our systems as part of the Y2K strategy - and we believed that they were all Y2K compliant. We accept that this was not the case and there have been failings on the part of the trust."

Andrew Cash, chief executive of the Sheffield Teaching Hospitals Trust which manages the Northern General Hospital, has apologised to the families affected. "The recommendations in the report and the steps we are taking will ensure to the greatest possible extent that errors of this nature cannot happen again," he said.

Read more on Business applications