Asian bank DBS has revealed that a seven-hour period where banking services were unavailable was caused by a mistake...
made by an IBM employee while carrying out routine work on a storage system.
The problems arose earlier this month and this week DBS chief executive Piyush Gupta revealed the reaseon in a message to customers.
He said, "The outage last week was triggered during a routine repair job on a component within the disk storage subsystem connected to our mainframe. This component was emitting alert messages, indicating that there could be an intermittent problem. As our IT environment is highly resilient and as the banking system was still fully functional, the problem was classified as low severity.
"Unfortunately, while IBM was conducting this routine replacement, under the guidance of their Asia Pacific team, a procedural error inadvertently triggered a malfunction in the multiple layers of systems redundancies, which led to the outage."
He said an outdated procedure was used.
IBM said, "IBM software monitoring tools detected instability in a communications link within the storage system, while the system remained fully functional. A recovery process was undertaken by IBM to address the instability. Unfortunately, a failure to apply the correct procedure inadvertently caused the service outage."
IBM said it has taken steps to enhance training of personnel "related to current procedures" and brought in experts from its global team to help.
Daniel Kushner, VP E-Business at application service automation platform maker Nolio said: "Unfortunately outdated procedures abound in IT Operations for both hardware and more so in software where countless software applications updates are managed by complex processes."