US telco to spend £1bn overcoming 11 September terror attack

US telecom giant Verizon expects to spend £1bn on recovering from the 11 September terror attacks that wiped out key company...

US telecom giant Verizon expects to spend £1bn on recovering from the 11 September terror attacks that wiped out key company facilitates in New York.

It will take another 12 to 15 months before the company will have fully recovered from the attack according to Paul Lacoutre, president of network services at Verizon, speaking at last week's CIO Summit organised by US magazine Business Week in Boston.

Steven Ruegnitz, managing director of enterprise applications security and telecommunications for Morgan Stanley Dean Witter's institutional securities business, told the conference his bank was spending an extra £20m a year on business continuity planning this year, on top of its regular budget.

"We will face this again," Ruegnitz told the conference. "We hope not at this magnitude."

Ruegnitz and Lacoutre described how their organisation's IT and telecoms infrastructure and emergency response plans saved lives and kept their businesses functioning.

Both agreed that the most important factor was that staff at both companies knew they had power to take action before disaster struck.

Lacouture stressed the need for field operations leaders and communications leaders who can devote their full attention to those tasks.

Ruegnitz was convinced that the quick action of leaders saved employees' lives. "We didn't talk about it," he said. "We kicked into gear and we activated."

Using internal charts and graphs, Ruegnitz showed how exactly one minute after the first plane hit the north tower of the World Trade Centre at 8.48am, Morgan Stanley's security staff began evacuating its 3,828 employees from the south tower. By 9.03am, the second plane hit the south tower.

At 9.15am, senior management was moved to a command facility elsewhere in New York. Ten minutes later, Morgan Stanley's backup system was activated and its New Jersey offices started buying emergency IT equipment.

At 9.50am, Morgan Stanley evacuated its last employee from the south tower. Nine minutes later, the south tower had collapsed, said Ruegnitz.

Morgan Stanley lost six employees in the disaster, four of whom had gone back in to make sure everyone was out.

Both companies credited their speedy recovery to underlying telecommunications infrastructures, said Ruegnitz and Lacouture.

Two years ago, Morgan Stanley changed its telecommunications policy to emphasise disaster readiness over cost efficiency. That meant the company's telecommunications infrastructure was built to survive daily disruptions, said Ruegnitz.

Of Morgan Stanley's entire technology infrastructure, telephone service was most critical just after the attacks, said Ruegnitz. The company had its own emergency telephone lines installed so employees could get outside lines.

The company also had four Internet service providers in New York alone, and was able to stay online throughout the disaster, However, three of the four lines went down, something Ruegnitz said was a little too "white knuckle" for him. He has since signed on with a fifth ISP.

Lacouture and Ruegnitz's key lessons

  • Turn customer call centres into emergency contact centres. Morgan Stanley, for instance, had call centre reps call all of its World Trade Centre employees.

  • Ensure access to emergency areas.

  • Prioritise which systems to restore first.

  • Maintain constant communication/co-ordination to update priorities.

  • Upgrade physical security at key locations. Companies may have secure rooms with pass-code entry, but make sure the code is not openly available.

  • Keep equipment up to date. Discontinued equipment is more difficult to replace, a fact that can serve as a helpful tool in rationalising upgrades to chief exective officers, said Lacouture.

  • Design next-generation networks with different hub locations.

  • Keep records backed up. Without back up, restoration time can double or triple.

  • Make sure employee assistance programs can expand resources in emergencies.

  • Use diverse office locations to help employees get back to work quickly.

Read more on IT risk management