Independent Bank is using virtualization from the desktop to the data center to eliminate some of the most common problems found with traditional disaster recovery (DR) plans.
Missing was a recovery environment that was in sync with the production environment. Before introducing virtualization disaster recovery, configurations between the production environment at the bank's headquarters a hot site 14 miles away, drifted.
"We had identical hardware in the hot site and the production site, which operationally was a lot of work, and we might not get to a [configuration or data] change for a few months," said Tom McKowen, enterprise architect with Independent Bank, which has total assets of about $US3 billion and 100 offices. "As a result, we might miss a few steps [between sites]," he said.
With Citrix Systems' XenServer Provisioning Services for managing the OS and configurations for Citrix's XenApp application virtualization technology, the bank can now stream workloads (OS and applications) to physical or virtual servers. "The provisioning service guarantees that the configuration in the DR environment matches the production environment -- where before we may have been three revs behind, now we have exactly the same image at both sites," McKowen said.
At the desktop layer, Independent Bank uses XenDesktop to virtualize entire desktops and stream them to users wherever they may be. The desktop images are hosted centrally on virtualized data center servers, and can be provisioned or deprovisioned on the fly. XenApp is also used as a means to separate applications from underlying hardware, in turn allowing applications to be streamed to users at home or in other remote locations regardless of the end device they are using. Applications can be accessed online or offline if the network goes down.
Independent Bank uses two virtual server platforms, Citrix's XenServer and VMware ESX, but it is VMware that comes into play as part of the bank's data replication strategy between the production environment and the hot site.
EMC provides the foundational components for recovery and replication between the two sites: a CLARiiON storage area network(SAN), Centera for data archiving and a RecoverPoint appliance that sits between the facilities. VMware's vCenter Site Recovery Manager uses these components to script the failover and restart order of all the virtual machines protected within the VMware vSphere environment. "We now have bidirectional replication; if there is a disaster recovery situation, we press a button and a virtual machine comes up in ESX at the backup data center," McKowen said.
VMware's ESX servers are used to house and recover the bank's Microsoft Exchange Server and SQL Server and Sybase Inc. databases.
The RecoverPoint appliance also bookmarks for point-in-time recovery. If current data is not clean in the hot site, the appliance allows IT to retrieve bookmarked data from several days earlier.
By combining VMware and Citrix virtualization technologies and EMC backup and replication technologies between sites, the bank can now take virtual desktop and server snapshots or maintain "golden OS and desktop images" and store them on a SAN. The snapshots can be restored at either site.
"We are much more able to standardize the process for recovery across both sites," said Pete Graves, CIO of Independent Bank. "We didn't have that a year ago, so in the last year we've come up in a huge way in terms of automating recovery. Recovery times have dropped from a couple days to a couple hours, and sometimes minutes he said.
Reducing hardware costs
In addition to reducing recovery times, the bank has been able to reduce its hardware costs by about $US1 million using application and server virtualization. The bank was maintaining hardware for specific business-unit Citrix configurations, as well as applications that were used infrequently. Since application virtualization creates a hardware-agnostic workspace, the bank could do away with the hardware needed to support multiple configurations and applications, and instead put the workspaces on a virtual machine running on any type of server.
It is not uncommon for enterprises to gain cost savings as a result of using virtualization disaster recovery, said Nelson Ruest, principal at consultancy Resolutions Enterprises .
"We had a very large customer that was taking up three floors in its backup data center, with each floor housing duplicate servers for every physical server in their production environment," Ruest said. "As they moved to a virtual disaster recovery strategy, they were able to get rid of a lot of servers and realize about a 60% to 70% savings per year for their whole DR infrastructure, he said."
Preparing for a pandemic, meeting federal regulations
Today Independent Bank uses XenApp and Citrix Access Gateway to deliver 120 financial and office productivity applications to 100 branch offices and about 1,300 employees.
About 1,000 employees can get access to the applications remotely, but the gateway could allow for capacity of as many as 2,500 employees, CIO Graves said. To prepare for a disaster situation -- from a pandemic to a power outage -- and the additional strain it would put on the network, the bank now uses Multiprotocol Label Switching Ethernet at each data center. From a wide area network perspective, the bank moved from a legacy network using frame relay over Asynchronous Transfer Mode technology to AT&T Corp.'s MPLS network.
"We can't plan for everything, but virtualization has given us flexibility in how we prepare for a disaster and meet regulatory requirements for disaster recovery situations, like a pandemic where we would have to separate employees," Graves said.
Next up: load balancing
With a virtualization mind-set already in place -- such as the ability to redistribute resources as needed -- load balancing between sites is a natural progression for the bank. This quarter, the hot site is expected to be shut down and replaced with a new backup data center.
"We have to recover quickly, and virtualization has brought us closer to that," Graves said. "We want to keep moving closer to our goal of recovery in minutes, and load balancing will help us achieve an always-up situation."