For Bill Jankowski, senior director of IT for the Baltimore Ravens, it all started with a presentation by representatives from the New Orleans Saints team at the most recent National Football League (NFL) offseason meeting on what they'd been through with Hurricane Katrina. The Ravens had also been affected by weather in 2003 when Hurricane Isabel drove the players and coaching staff to their opponents' practice facilities in San Diego a week before their Sept. 21 game, as the east coast of Maryland flooded and power outages spread across the state.
Previously, the Ravens had backed up some 200 GB of administrative files on a set of 20 tapes, half of which rotated through the office on a weekly basis for incremental backups, and the other half of which were stored every six months in offsite archives. It was an economical plan. Jankowski estimated it cost between $2,000 and $3,000 per year, most of it the cost of replacing tape media.
The problem was money. It may seem as though NFL teams would have no shortage of funds, "but 62% of our money from TV goes to paying player salaries," Jankowski said.
Ultimately, Jankowski settled on disk-based onsite backup, as well as offsite disaster recovery services from AmeriVault Corp. Right now, the Ravens back up their data using AmeriVault's ServerVault device at their offices in Baltimore. The data is then replicated nightly over the wire to Boston, and within 36 hours it is also transferred to a second disaster recovery site in Chicago.
It costs 10 times as much as the tape system -- around $2,600 per month. But Jankowski said Ravens management was swayed by this approach because it would save substantially on capital equipment and staffing costs over the next three years.
"If you factor in adding new hardware every three years and at least one more staff person, plus replication products, renting a secondary location, new backup software and bandwidth, you're talking tens of thousands of dollars over the approximately $30,000 we pay per year for AmeriVault's services," Jankowski said.
The other side -- Keeping things in-house while cutting costs
Meanwhile, Telecom provider Cbeyond faced many of the same issues as the Ravens when considering a new backup plan late last year for 170 terabytes (TB) of storage. According to Allen Thomas, senior director of systems operations for Cbeyond, despite having a much larger storage environment than the Ravens, Cbeyond is just as cost conscious.
"We were founded just six years ago," Thomas said. "We've grown up being used to cutting costs and behaving as a startup." Cbeyond already had a secondary data center in Dallas; that and a complex mix of server operating systems, including Solaris 8, Red Hat Enterprise Linux 4 and Microsoft Windows 2000 and 2003, made outsourcing less practical, though Thomas said he considered the option. Cbeyond also has a mix of storage protocols in its environment. It has 12 TB stored on 3PARdata Inc.'s InServ Fibre Channel array for Tier-1 storage, most of it Oracle databases containing customer and billing information and 100 TB of Tier-2 data, including back office file shares and email stores on Nexsan Technologies' SATABeast iSCSI arrays.
Ultimately, Cbeyond kept backups in-house but found that staging backups to disk before writing to tape not only helped centralize the backup infrastructure, but it could be done cost effectively using iSCSI.
In order to preserve performance for the primary data on Fibre Channel but cut costs, Cbeyond attached iSCSI connections to the front ends of each of its servers and performed host-based backup over IP, rather than backing up from the back end of the Fibre Channel storage arrays. The company is also staging its backups to another Nexsan SATABeast array accessed using iSCSI before backing it up to tape using BakBone's NetVault.
BakBone was also chosen, according to Thomas, because of cost considerations -- the software uses the same agent for any operating system. "Cost was a big issue, but the consistency of the BakBone product across different environments was what made it attractive to us," he said
Because the tiered storage and heterogeneous server environments are centralized and run over IP, Thomas said he estimated that his administrators spend just 30 minutes a week administering the company's backups, and the company has been able to stick with eight IT engineers in total.
He isn't permitted to disclose specific numbers with regard to his cost savings, Thomas said, but taken together, he estimated that by networking with IP rather than Fibre Channel, using iSCSI storage for staging backups and centralizing server backups using BakBone, the company slashed backup costs in half over its previous system. Before, it had a mix of direct attached storage (DAS), a Dell rebranded EMC CX200 and network attached storage (NAS), including Network Appliances Inc.s' NetApp 740s backed up with Veritas NetBackup. Under this setup backups took the administrators between three and five hours per week to manage, he said.
Ultimately, both users did end up making an investment over what they had been doing previously, but according to Greg Schulz, founder and analyst with the StorageIO Group, they were wise to find ways to consolidate and simplify their environments.
"It's interesting that people continue to focus on disk space in terms of cost, yet it's probably the area where we continue to see the biggest declines in pricing -- and over a period of time, its the people and software costs to manage storage that add up," Schulz said. "Users should look to improve effective service for a given amount of dollars spent, as opposed to just focusing on cost."
Other companies in the outsourced backup space include Asigra Inc., EVault, Yosemite Technologies Inc., CommVault Systems Inc., Intradyn Inc. and Iron Mountain Inc.'s subsidiary LiveVault. Asigra's Televaulting product, also available for users to purchase and manage themselves, has flexibility similar to BakBone's when it comes to multiple operating systems, as it does not use software agents.