Reduce risks of disaster recovery testing

Every IT manager knows that a disaster recovery plan is crucial, but for many organizations that plan is little more than a document, untested and unpracticed.

While there are several variables involved in safeguarding your server farm, the ultimate resting place for your data in a disaster is a remote hot site where you might have little control. In practice, do you know how your hot site will react to a failure?

New disaster recovery testing software may help data center pros get a better view of their business continuity operations.

Organizations test backup systems rarely, if ever, because of the risks involved -- creating an interruption while trying to prevent one.

Bringing down a live data center to test a hot site often runs the possibility of losing data and revenue, according to Harvey J. Hindin, vice president and lead analyst of business continuity for Port Chester, N.Y.-based Ideas International Ltd.

"Most people don't do real-world testing because they don't want to shut down revenues," Hindin said. "Taking live production systems off takes a lot of planning. If it fails, it's not a career enhancing move."

That's why some companies are looking for software to work around that risk.

Ben Weinberger, director of IT for Fort Lauderdale, Fla.-based law firm of Ruden McClosky, faced this exact situation. He is in the middle of a project to collocate his main data center in Fort Lauderdale with a hot site in Chicago -- and he's in a race against time to finish with a tropical storm season already under way.

"All of our offices are in the state of Florida. We're moving the hot site to Chicago to get it completely away from the potential for hurricanes. Last year we had four [hurricanes] hit the state," Weinberger said.

The most important aspect of the business continuity for Weinberger is the data. His firm's lawyers are often working on very tight deadlines and wait up to the last minute to file briefs.

"If our documents are unavailable, one of our attorneys could miss a filing deadline," Weinberger said. "There's nothing worse than a lawyer that can't file a document. Just because there's a natural disaster, it doesn't change court deadlines."

Weinberger put the hot site in place and reconfigured the network in such a way that if Fort Lauderdale went down, attorneys could connect directly to Chicago. But he didn't have a way to make sure that the database in Chicago would be available if the main site was down. So he turned to a disaster recovery software vendor, Waltham, Mass.-based XOsoft, for a testing product.

XOsoft's Assured Recovery software allows Weinberger to test his replicated data in Chicago by actually starting up application services on a replica system and performing whatever operations are required, including database updates.

During normal operations, data is copied from the master server to the backup. During testing with Assured Recovery, replication on the master server is suspended and data comes up on the backup server. Testing is done on the backup database, and then the master server is notified of the successful test. Snapshots of the backup are taken, and then the database is returned to its pretest state. After the test is complete, spooled backup is copied and the replication from main site to hot site resumes.

According to Weinberger, these tests can take place without any end-user interruption.

"We're bringing up our Sequel database and synchronizing directly," Weinberger said. "The other option would be to shut down the main site and do it out of hours. Being able to test during business hours saves money and gives us access to more resources."

Launched in May, Assured Recovery enables companies to test production environments every day, instead of once or twice a year, according to XOsoft.

Other companies offer similar products, such as Veritas Software (now owned by Cupertino, Calif.-based Symantec Corp.) Veritas' Cluster Server software has a feature called the Disaster Recovery Fire Drill that will make a carbon copy of a live production environment that enables users to test failover scenarios and events. Veritas said IT pros can test any configuration's readiness by mimicking a failover -- without disrupting the applications at the primary data center.

But Hindin said the word "test" means different things to different people, and that the database is only one piece of the business continuity puzzle. Hindin recommends researching testing providers and sticking with known players.

"If you're the business continuity manager at a large company, you're going to deal with IBM, Sun or Hewlett-Packard," Hindin said.

But Weinberger faced difficulties with major players. According to Weinberger, his organization went to Dell several times and asked it to implement a disaster recovery program exactly the way he wanted it, but the price was beyond his budget, so the project didn't get implemented until he went with an outside vendor.

"We were in a bad position last year," Weinberger said. "It took over 30 hours to bring our backup data across the area wide network. Today, our level of backup isn't as high as NASDAQ, but our lawyers around the globe can now connect to Chicago if there's a problem."

Let us know what you think about the story; e-mail: Matt Stansberry, News Editor

This article originally appeared on SearchDataCenter.com.