Strategic Storage: DR planning blueprint

Executives resist spending money on anything that does not generate revenue. In the case of disaster recovery (DR), that could be a fatal mistake.

DR provides a blueprint for resuming operations in the event of unanticipated disruptions -- whether from hurricanes, hackers, terrorists, power outages or any number of other risks. Included are arrangements for data replication and storage, auxiliary power sources, redundant computer systems and alternative work locations for employees.

International Data Corp. estimates that companies lose an average of $84,000 for every hour of downtime. Strategic Research puts the figure closer to $90,000 an hour. Yet many enterprises put off implementing enterprise-wide DR plans until it is too late. "Their attitude is, 'It's never going to happen to me,'" says Ed Broderick, an analyst with the IT consultancy Robert Frances Group in Westport, Conn.

Experts advise that there are seven key components to DR planning: Business impact assessment, discovery, budget, role-based teams, data protection, logistics and semiannual tests. What follows is a look at each of these areas, along with solid advice from storage experts and IT pros in the trenches.

1. Business impact assessment

A business impact assessment (BIA) should answer two questions pivotal to your DR planning: How soon do you need to resume operations? And how much information would be lost, including its financial impact?

Risks may vary according to geography, industry and other factors, but they generally fall into one of four distinct categories: financial loss, damage to reputation, regulatory penalties or operational disruption. Carl Claunch, a research vice president with Gartner Inc., says a BIA articulates each risk in a financial context for chief executives.

"There are all sorts of ways an outage can affect a business, and having a sense for how those costs mount up and when they start getting steep 'gives' you a reasonable target" on how much to spend on DR planning, Claunch says.

2. Discovery

Bill Peldzus, an analyst with Glasshouse Technologies Inc. in Framingham, Mass., says companies often jump right to the information technology aspect of DR without first knowing their business drivers.

"For each application, you have to understand the network infrastructure, the systems infrastructure and the storage infrastructure. If you don't have that information, it's nearly impossible to begin discussing technology options for your DR architecture," Peldzus says.

Segment data according to "tiers of criticality" based on its relevance, according to Broderick. Make sure you cover any and all enterprise-critical applications, as well as any line-of-business-critical applications.

"Lots of companies do an adequate job of protecting the corporate data center but fail to adequately protect (individual) data centers that run their lines of business," Broderick says.

3. Budget

Not every risk is worth offsetting. If your business lies in a floodplain or a region prone to hurricanes, weigh those probable disruptions ahead of less likely scenarios. Financial institutions may face severe fines for failing to meet government regulations. Health care companies similarly are at risk if sensitive patient data gets exposed. A retailer may lose customers should its system be hacked in credit-card scams.

Broderick suggests a phased approach to implementation to spread the cost over short cycles of time. Include at least one person appointed by the CEO on your DR team and begin by drafting a preliminary 90-day budget.

"The first task during the first 90 days is to build a six-month second-phase budget," Broderick says.

4. Role-based teams

Rather than relying on one or two people, it is widely considered preferable to appoint a team of different employees to manage and maintain the plan.

"Your DR plan should be documented so that different people can execute it and you're not dependent on any specific individuals. Don't depend on knowledge that is in somebody's head," Claunch says.

Boston Capital, a financial services company with about 240 employees, launched DR planning about a year ago to comply with regulations from the National Association of Securities Dealers.

"We're definitely trying to make our DR plan role-based. We've identified a business continuity team of nine people who have been here a long time, represent a good cross-section of 'departments' and whom we feel would display good leadership qualities in the event of a crisis," says Brian Madden, the company's IT director.

Team members usually need specific technical competencies to execute the plan, although they don't need to understand the intricacies of your applications. "You want the plan to be detailed and role based so that competent people in different areas can look at it and 'know how' to recover your applications," Madden says.

Among its responsibilities, a DR team should establish the sequence in which key systems and applications are recovered. "If you fall behind on one system that's not as critical, you can let it slip and work on 'restoring' systems that are most critical to the business first," Claunch says. "The other reason this is important is that sometimes system B depends on system A."

5. Data protection

Perhaps no component is as crucial as technology. Aside from people, information is the single most important asset for any organization. Yet companies often neglect simple measures such as storing multiple copies of crucial business files and applications at backup locations.

Miami-based Biscayne Aquaculture designs life-support systems for aquatic animals in zoos and public aquaria. The company began institutionalizing its DR practices last year as hurricane Frances bore down on Florida. Biscayne participated in a beta test of a remote data backup software, using it to store 3D drawings, computer-aided design, customer information, financial and other vital operational data.

"The most critical thing is getting data secured, because equipment and buildings can be replaced," says Jim Post, Biscayne's director of project development.

Automated, remote backup services also are available from competitors Iron Mountain, Comdisco, Amerivault and numerous smaller vendors. Other enterprises pay companies like SunGuard Disaster Recovery, Electronic Data Systems or IBM to manage remote "hot sites," or alternate offices that contain computers, equipment and furniture that allow for continuous operations after a disaster.

"Whether 'delivered' by truck or by automation, you've got to have multiple copies of data stored off site and a long, long way from your primary data center. And you must have a plan to get it back that is tested, doable and practical," Broderick says.

Include the same architecture in your DR site as in your primary data center, Peldzus says. If you use a WAN, the Internet, an intranet portal and telephones to take customer orders, build the same infrastructure at your backup facility. "If you can't get users to your DR site at a similar level of performance 'as at' your production site, it's almost not worth having."

6. Logistics

In DR planning, it pays to sweat the small stuff. Following the Sept. 11, 2001 terrorist attacks, Broderick says some companies neglected to deal with logistical details, such as employees' transportation to backup facilities. Still other firms neglected to stock their DR site with everyday items like desks, pads and pens.

"Where a lot of companies stubbed their toe on 9/11 was simple logistical details," Broderick says.

Along those same lines, he recommends giving some members of your DR team spending authority in the event you have to temporarily house and feed employees.

7. Semiannual tests

The conventional wisdom is that DR plans should be tested at least twice a year, perhaps more depending on the degree of change within your environment. Moving servers in and out of production, reworking a database or launching new applications are all significant environmental changes that need to be documented in your DR plan. This could be accomplished by making DR plan updates the responsibility of application development or database administration, Broderick says.

Don't be discouraged when you discover points of failure. The most successful tests point up the most problems because they help "improve the quality of the plan so it's more likely to work in an actual disaster," Claunch says.