As firms count cost of oil depot blast, what are the disaster recovery lessons?

How companies coped in the aftermath of the Buncefield inferno.


How companies coped in the aftermath of the Buncefield inferno.

IT services companies and hardware suppliers are among the businesses counting the cost of the fire at the Buncefield oil depot last month.

Explosions and heat from the fire caused severe damage to more than 80 buildings on the industrial estates surrounding the terminal, and some were demolished by the blasts.

Initial estimates from Hertfordshire Chamber of Commerce put the cost of the damage at between £500m and £1bn.

IT software and services firm Northgate Information Solutions was the closest business to the fire, when a blast ripped through the oil depot in the early hours of Sunday 11 December.

The explosion rolled across Northgate's car park and into its 150,000 square-foot building, starting a fire which left the firm's UK headquarters an empty shell.

Other companies that suffered severe damage include printer manufacturer Epson, imaging company Fuji, Waverley, the distribution arm of Scottish and Newcastle Breweries, online retailer Asos, IT services company Steria, networks equipment supplier 3Com and eye care manufacturer Alcon.

The incident was as dramatic a test of business continuity planning that any firm could expect to face. The larger firms responded quickly and were able to resume trading by the following day. But there are lessons to learn, particularly for smaller businesses.

"Some of the large companies, such as Northgate and Fuji, were able to get up and running fairly quickly, but a number of smaller businesses were caught on the hop," said Tim Hutchings, chief executive of Hertfordshire Chamber of Commerce.

Digby Jones, director general of the Confederation of British Industry, urged firms to make business continuity planning their new year's resolution, following the Buncefield fire.

"It was a miracle that no one was killed, but more than anything it showed that the unexpected can happen at any time. Please, take a long hard look at how you safeguard your workplace and employees," he said.

It is too early to talk of business casualties yet, but at least two small companies are in "intensive care". Others owe their survival more to luck than good planning, according to Hutchings. If they had been denied access to their premises for more than 24 hours, their survival would have been "extremely questionable", he said.

"Evidence suggests that not as many companies had computer back-ups and other financial records stored off-site, as they should," he added.

Case study: Asos weighs the risks

Online clothing retailer Asos opened its 85,000 square-foot warehouse on the Buncefield site just months before the blast.

The building and clothing with a retail value of £5.5m suffered extensive damage, effectively shutting down the business.

The firm suspended its shares on 23 December and has refunded 19,000 customers who ordered goods over the web for Christmas.

"We are heavily seasonable, so it could not have happened at a worse time," said Asos finance director Jon Kamaluddin.

One hundred warehouse staff are now cleaning up and sorting the good stock from the damaged, and the company is confident that it will be up and running by the end of January. It said insurance cover will mean that its predicted end-of-year profits will not be affected.

"We have always known that operating from one site would be an Achilles' heel for us, but where do you draw the line? Do you set yourself up in two warehouses and incur the cost of running across two operations, or do you accept that this is a risk in your business model and take out insurance to cover that risk? That is the decision we made," said Kamaluddin.

Case study: Steria meets council paycheck deadline despite control centre loss

John Torrie, chief executive at IT services firm Steria, was shaken out of bed by the explosion and was at the site within half an hour.

The blast had devastated the back of the Steria building, destroying a newly installed control centre used for remote monitoring of customers' IT systems.

Torrie convened an emergency board meeting at the firm's Sunbury office at 10am that morning, while managers raced to secure temporary offices in Hemel Hempstead before they were snapped up by other firms affected by the blast.

"We had the operational issue of recovering customer services and we still had to keep the business running. We had a major contract bid due in on Friday [16 December]," he said. The IT department was able to rebuild the bid team's infrastructure in the head office, and they were functioning by lunchtime, said Torrie.

IT staff worked around the clock to rebuild the control centre.

"We had to bring down one system completely. Fortunately, it was a batch processing system. We rebuilt the system image in another datacentre on standby equipment. And that was up and running by 8.30 on Monday morning," said Torrie.

Three customers experienced slight delays in their IT services, he said, but the firm managed to get its systems up and running in time to ensure that a local authority was able to pay its 75,000 staff on time.

Case study: back-up and recovery systems enable Northgate to survive datacentre devastation

Chris Stone, chief executive at IT services firm Northgate Information Solutions, was phoned at 6:25 on Sunday morning by his head of security. Northgate's UK head office, which directly faced the oil depot, had been completely destroyed.

"The explosion rolled across our car park, which is all that separated us from the oil depot. It came in at the end of the building and it ripped up and out, setting the building on fire," said Stone.

Stone's first thought was the safety of staff. "There was a moment of fear when you think, well, all the people you were expecting to be there were accounted for, but in the IT world, some of our more talented programmers are prone to have good ideas at curious times of day," he said.

In the event, only four people were in the building. Three suffered cuts and bruises from flying debris and were taken to hospital.

The blast devastated the company's state-of-the-art datacentre - more than 20,000 square feet containing servers belonging to both Northgate and its customers.

The timing of the explosion was unfortunate. Night shift operators had just finished preparing the back-up tapes, containing a day's worth of data. The tapes were due to be collected an hour later and taken to a secure off-site store by courier.

By 8am, after a series of conference calls with Northgate's other directors, Stone had invoked the disaster recovery plan. Managers were able to communicate with staff by sending SMS messages to their mobile phones, instructing them not to come into work while the firm found alternative premises.

Northgate's disaster recovery supplier, Sungard, began firing up back-up systems. It was able to provide duplicate hardware for customers who had taken up a disaster recovery option. Other customers had to wait while Northgate engineers bought in and commissioned new hardware. "It was a good end of year for our suppliers," said Stone.

"We had our first engineers arrive at the Sungard facility at about 10am on Monday. Then we retrieved the back-up tapes from the off-site store. The back-ups had been completed on Friday night."

Northgate is still attempting to recover the missing 24 hours worth of data. "We have recovered a lot of it already. We have been able to recover the assets and to retrieve the data from the drives," said Stone.

"All customers are now operating effectively. Well over half the customers are back to where they were before the explosion."

Read more on IT for small and medium-sized enterprises (SME)