AT&T subsidiary talks real-world storage virtualisation

3PAR's storage virtualisation helps AT&T's managed services arm to stay competitive over EMC and NetApp, though there are still some kinks to be worked out.

Virtualised storage from 3PARdata has helped USinternetworking (USi), the managed services arm of AT&T, to stay competitive in the face of Web-based Software as a Service (SaaS) competition, though the company is also pushing 3PAR for more advanced snapshot and tiered storage features.

According to Tobias Ford, chief technology officer of USi, in a presentation to Storage Decisions attendees this week, the company previously found EMC  storage too costly and had a support issue with Network Appliance Inc. (NetApp) two years ago that soured them on the company.

"EMC locked in prices and we couldn't negotiate on a regular basis, so costs weren't in line with the market after we had owned the [EMC Symmetrix] frame for awhile," Ford said.

More on storage virtualisation

Storage virtualisation overrated, says NetApp VP
NetApp's vice president of emerging products, Jay Kidd, discusses storage virtualisation, competition with Isilon and NetApp's current identity crisis

Storage virtualisation is no silver bullet, users say
Storage virtualisation is the hottest topic at SNW this week, but many users are still struggling with how to implement it

Storage Area Networks: getting the full value
What is a storage area network? What are the benefits and pitfalls of installation? How can a SAN fit into your organisation? These questions and more are answered in our Storage Area Network School

In 2004, USi inherited NetApp storage with an acquisition and liked the snapshot capabilities that came with the filers. However, Ford said, when the company's systems began to fill up in early 2005 and performance issues arose, he was unhappy with the support from NetApp.

"We had an outage, and we ended up paying back our clients because we violated service level agreements (SLA). NetApp took a week to give us a 'fix' of two new filers," he said. He advised attendees to consult with user references about a vendor's "support while you're under fire" before purchasing.

Though the company still has 200 terabytes (TB) each on EMC and NetApp storage on the floor, Ford said the company plans to migrate everything to 3PAR InServ Storage Server systems. USi purchased four arrays from 3PAR, two S800s and two S400, which together have a capacity of another 200 TB.

Thanks to thin provisioning, which helps control storage costs by deferring disk purchases, and 3PAR's back-end storage virtualisation, the company has been able to lower its prices and remain in business alongside ultra-cheap Web-based hosting services, such as Amason's S3.

"With our previous systems we were priced at $7 to $8 per gigabyte per month -- Amazon's charging 10 cents per gigabyte per month," Ford said. Adding the 3PAR system has allowed USi to lower its base pricing to $2.50 per gigabyte per month, more in line with market cost.

The company has converted some network-attached storage (NA) shares into LUNs on the new storage area network (SAN) and said customers have seen as much as a 50% performance improvement. The company can migrate data frequently and easily between three tiers of storage, consisting of 146 GB Fibre Channel drives in RAID-1 for Tier-1, in a RAID-5 configuration for Tier-2 and 500 GB SATA drives in a RAID-5 configuration for Tier-3.

Wish list: Snapshot features and automated migration

Still, in an environment that depends heavily on snapshots for backup, tiered storage migration and testing, Ford said he's still waiting to see 3PAR work out a few kinks.

Chief among them is the fact that 3PAR Recovery Manager for Exchange, a software product extension to 3PAR's Virtual Copy snapshot software specifically for snapshot images of Microsoft Exchange databases, struggles to synchronise his large Exchange mail stores, which can often run to 40,000-to-50,000 mailboxes at 1 GB to 2 GB each.

Snapshots will capture archive logs from the Exchange server, which Ford said is useful for getting back to particular points in time for restoring parts of the system. However, in a total outage or disaster recovery scenario, without being able to take a full snapshot of the mail store, the company won't be able to recover the whole system.

For now, Ford said the company is "suffering" with Symantec's NetBackup and tape, "and regularly blowing out the backup windows."

He admitted his Exchange capacity is unusual, but also said 3PAR has promised it is working on tighter integration with Microsoft's Visual SourceSafe (VSS). Meanwhile, Ford said he is looking to backing up email using a Microsoft tool, Data Protection Manager (DPM) 2.0, in order to cut out integration issues entirely.

According to Craig Nunes, vice president of marketing for 3PAR in an email to, "The testing [of this product] with [USi's] specific server stack combination has not yet been completed."

Another place where Ford said he'd like to see further development in 3PAR's snapshot software is the fact that on thin-provisioned volumes, it requires snapshots to be taken to the same tier of storage and then migrated manually to a lower tier. On baseline volumes, Ford said, snaps can be taken directly to a lower tier of storage. It's another thing he said 3PAR has promised in a future release.

"We support this capability today with one of our other Thin Copy technologies, 3PAR Remote Copy, and expect to do the same with 3PAR Virtual Copy," according to Nunes.

Ultimately, Ford said he'd like to see 3PAR do away with manual tiered storage migrations altogether by adding a feature like Compellent Technologies Inc.'s Data Progression, which migrates blocks automatically between tiers of disk according to frequency of access.

"We tested Compellent very vigorously, but we just couldn't get their array to perform well enough for our needs," Ford said. "If they could, I'd still buy them -- I'd find a niche for them in my environment."

According to Nunes, "this is a valuable technology and a part of [3PAR's] statement of direction."

Finally, Ford said thin provisioning is also not a perfected science. "You still absolutely must do careful capacity planning in order to make it work," he said. "Some applications also aren't compatible with thin provisioning because they will write data to an entire allocated volume." Others can be made to work, but it often takes training and testing with application administrators.

"It's like Wac-A-Mole," Ford said. "You solve one problem, then you move on to the next pain."

Read more on Integration software and middleware