3dmentat - Fotolia

ING Bank datacentre fire suppression system test knocks banking services offline

Bank apologises after planned test of datacentre fire suppression system knocks out servers and storage, leaving Romanian customers unable to access funds

ING Bank has apologised to customers in Romania after a fire suppression system test knocked out its datacentre for 10 hours, leaving them unable to access their accounts and make payments over the weekend.

In a letter to customers, Daniel Llano, head of ING Bank’s retail division, said a planned test of its gas-based fire suppression system caused unforeseen problems for the datacentre’s servers and storage kit, located in Bucharest, on Saturday 10 September. 

A source, quoted by Motherboard, said it was the sound generated by the release of the inert gas from the cylinders containing it, rather than the substance itself, that caused the datacentre hardware to fail.

The noise levels are reported to have exceeded 130dB during the test, causing the hard disk drives inside the datacentre’s storage systems to vibrate and malfunction.

As a result, people were left unable to carry out card transactions and access the firm’s internet banking services, and ING’s ability to notify customers about the situation was hampered by its email and text systems being knocked out.

The fire suppression system used by ING relies on a mix of nitrogen, argon and carbon dioxide to extinguish flames and is favoured by datacentre operators because it does not affect the performance of nearby IT equipment when in use.   

Despite activating the company’s disaster recovery plans, Llano said the “magnitude and complexity” of the problems meant it took longer than expected to get the company’s systems back up and running.

“I wish to assure you this incident is exceptional and unprecedented in the history of ING Romania,” Llano added.

Read more about datacentre outages

Speaking to Computer Weekly, Rob Garbutt, CEO at colocation provider LDeX Group, said he had never encountered another case where excess noise had caused datacentre hardware to fail.

“This is certainly the first time I’ve heard of sound waves from a gas fire suppression system creating damage to hard disk drives (and so server failure), and this is probably highly unlikely to happen again,” he said.

“However, it really underlines how important multiple lines of defence are in datacentre environments. We have a ‘double knock’ system so that smoke particles are required to be detected in two zones to trigger our gas fire suppression system, reducing the risk of a false positive.”

Read more on Datacentre disaster recovery and security