Multiple failures at a datacentre run by CSC left hospital trusts in the North West and West Midlands without access...
to patient administration systems for up to five days.
The failure, which affected 72 primary care trusts and eight acute care trusts, was one of the worst IT crashes ever to hit the NHS.
All computer systems supplied by CSC to the NHS failed simultaneously at 10am on 31 July, following a problem at the supplier's Maidstone datacentre. The failsafe switchover to the company's back-up datacentre in Tunbridge Wells did not operate correctly, leaving trusts without primary or back-up systems.
CSC and Connecting for Health said the incident was caused by a failure in storage area network (San) equipment due to a power failure. Back-up power systems failed to kick in at the main datacentre.
"A number of the multiple UPS [uninterruptible power supply] systems in use at the datacentre were down for essential maintenance and during that scheduled downtime the incident occurred," a CSC spokesman said.
Five days after the outage, access to the system was being provided via a San at the back-up datacentre.
"It will remain the operational San until the investigation at the primary datacentre is finalised," CSC said.
Producing failsafe non-stop systems is a key objective for the National Programme for IT in the NHS, and the investigation into the outage will raise questions over how a single incident could lead to such a severe problem and whether there was a single point of failure.
A San expert who spoke to Computer Weekly said, "Sans are the basis of business continuity plans. When you design a San it should have built-in redundancy, so if you lose a switch you do not lose the entire access to the network."
Read article: Question time for NHS
Read article: NHS staff bear the brunt
Vote for your IT greats
Who have been the most influential people in IT in the past 40 years? The greatest organisations? The best hardware and software technologies? As part of Computer Weekly’s 40th anniversary celebrations, we are asking our readers who and what has really made a difference?
Vote now at: www.computerweekly.com/ITgreats