Multiple failures at a datacentre run by CSC left
hospital trusts in the North West and West Midlands without access
to patient administration systems for up to five days.
The failure, which affected 72 primary care trusts and eight
acute care trusts, was one of the worst IT crashes ever to hit the
NHS.
All computer systems supplied by CSC to the NHS failed
simultaneously at 10am on 31 July, following a problem at the
supplier's Maidstone datacentre. The failsafe switchover to the
company's back-up datacentre in Tunbridge Wells did not operate
correctly, leaving trusts without primary or back-up systems.
CSC and Connecting for Health said the incident was caused by a
failure in storage area network (San) equipment due to a power
failure. Back-up power systems failed to kick in at the main
datacentre.
"A number of the multiple UPS [uninterruptible power supply]
systems in use at the datacentre were down for essential
maintenance and during that scheduled downtime the incident
occurred," a CSC spokesman said.
Five days after the outage, access to the system was being
provided via a San at the back-up datacentre.
"It will remain the operational San until the investigation at
the primary datacentre is finalised," CSC said.
Producing failsafe non-stop systems is a key objective for the
National Programme for IT in the NHS, and the investigation into
the outage will raise questions over how a single incident could
lead to such a severe problem and whether there was a single point
of failure.
A San expert who spoke to Computer Weekly said, "Sans are the
basis of business continuity plans. When you design a San it should
have built-in redundancy, so if you lose a switch you do not lose
the entire access to the network."
Read article:
Question time for NHS
Read article:
NHS staff bear the brunt
Vote for your IT greats
Who have been the most influential people in IT in the past 40
years? The greatest organisations? The best hardware and software
technologies? As part of Computer Weekly’s 40th anniversary
celebrations, we are asking our readers who and what has really
made a difference?
Vote now at:
www.computerweekly.com/ITgreats