3dmentat - Fotolia
ING Bank has apologised to customers in Romania after a fire suppression system test knocked out its datacentre for 10 hours, leaving them unable to access their accounts and make payments over the weekend.
In a letter to customers, Daniel Llano, head of ING Bank’s retail division, said a planned test of its gas-based fire suppression system caused unforeseen problems for the datacentre’s servers and storage kit, located in Bucharest, on Saturday 10 September.
A source, quoted by Motherboard, said it was the sound generated by the release of the inert gas from the cylinders containing it, rather than the substance itself, that caused the datacentre hardware to fail.
The noise levels are reported to have exceeded 130dB during the test, causing the hard disk drives inside the datacentre’s storage systems to vibrate and malfunction.
As a result, people were left unable to carry out card transactions and access the firm’s internet banking services, and ING’s ability to notify customers about the situation was hampered by its email and text systems being knocked out.
The fire suppression system used by ING relies on a mix of nitrogen, argon and carbon dioxide to extinguish flames and is favoured by datacentre operators because it does not affect the performance of nearby IT equipment when in use.
Despite activating the company’s disaster recovery plans, Llano said the “magnitude and complexity” of the problems meant it took longer than expected to get the company’s systems back up and running.
“I wish to assure you this incident is exceptional and unprecedented in the history of ING Romania,” Llano added.
Read more about datacentre outages
- SSP confirms to Computer Weekly it has no clear idea when all brokers will be able to continue doing business via its SaaS platform, two weeks on from Solihull datacentre power outage.
- Power supply issues at Docklands datacentre could be behind loss of internet access for more than 5,500 BT broadband users.
Speaking to Computer Weekly, Rob Garbutt, CEO at colocation provider LDeX Group, said he had never encountered another case where excess noise had caused datacentre hardware to fail.
“This is certainly the first time I’ve heard of sound waves from a gas fire suppression system creating damage to hard disk drives (and so server failure), and this is probably highly unlikely to happen again,” he said.
“However, it really underlines how important multiple lines of defence are in datacentre environments. We have a ‘double knock’ system so that smoke particles are required to be detected in two zones to trigger our gas fire suppression system, reducing the risk of a false positive.”