Expect more dedupe innovation in 2009

The storage industry spent all of 2008 praising data deduplication. There's more to come - and products a-plenty - this year!

Even with all of the data deduplication product rollouts in 2008, we can expect plenty more throughout this year. CommVault got the ball rolling when it launched Simpana 8 with block-level deduplication and dedupe for data on tape and disk in January. Quantum Corp. brushed up the integration and replication capabilities of its DXi7500 dedupe backup box around the same time. While there was plenty of news and new products in 2008, we don't expect the onslaught to let up -- even more data deduplication products, including some for primary storage, are slated to roll out soon. The following is a roundup of dedupe products currently in the pipeline:

Dell. Last November, Dell said it would bring out dedupe products in 2009 that will integrate with Quantum's dedupe products. Quantum also licenses its target deduplication software to Dell's storage partner EMC Corp., so it's expected that Dell's products will be compatible with EMC's Disk Library, if not an outright OEM deal. Dell said it would also sell Quantum's replication software to copy deduped data for disaster recovery.

EMC's Celerra. EMC will offer primary storage dedupe for file systems with the next upgrade of its Celerra NAS platform, due early this year. EMC will use single instancing from its Avamar dedupe product and compression from its RecoverPoint appliance. The Celerra dedupe will compress files with low usage activity, and single-instance files to remove duplicates. EMC's NAS rival NetApp began offering dedupe for primary storage a few years back, but no other major storage vendors have followed suit until now.

Hewlett-Packard (HP). There have been no public announcements, but David Rogers, manager of product marketing for HP StorageWorks data protection, tells SearchStorage.com that the company will add replication on the firmware of its dedupe products, the StorageWorks D2D Backup System and StorageWorks Virtual Library System (VLS). Rogers says the replication was developed specifically for the dedupe products and will be a licensed feature.

In addition to these new products, expect enhancements to the dedupe products already out. For instance, FalconStor Software Inc. added a NAS interface to its virtual tape library (VTL) dedupe product in early December.

Next dedupe frontier: Archiving

Stephen Foskett, director of data practice at storage consultancy Contoural Inc., says dedupe will become a necessary technology for backup and take big steps into archiving in 2009. But dedupe isn't yet ready for many types of primary storage, to the chagrin of some of the storage customers he talks to.

"There's some general disappointment at this point that people can't use dedupe on primary storage, but they're excited about the potential it has on archiving," says Foskett. "I'll be shocked if every product in the archiving space doesn't have advanced deduplication pretty soon."

In brief: Dedupe in 2009

The following deduplication products have been unveiled or are expected to be rolled out this year.

Vendor info

What's notable


CommVault Simpana 8

Adds block dedupe and tape/disk dedupe

Available now

Dell Inc.

Dedupe systems based on Quantum technology

Not announced yet

EMC Celerra

File dedupe using single-instance storage and compression

Not announced yet


Added NAS interface to virtual tape library (VTL) dedupe product

Available now


Plans to add replication to current dedupe products

Not announced yet

Quantum DXi7500

Enhances replication and integration of DXi7500 dedupe system

Available now

Riverbed Technology Atlas

Applies some Steelhead technology to primary storage dedupe

Expected mid-2009


Frank Slootman, CEO at Data Domain, agrees dedupe will become more of an archiving play this year, but says dedupe products will change more in size and scope than in capability. He says they'll get bigger and faster on the high end, and smaller and cheaper on the low end.

"The technology is still developing, and will always be developing," says Slootman. "We're riding a relentless wave of microprocessor improvements, mostly on the Intel side. And that kind of stuff is manna from heaven for us."

CommVault took the lead on one new development. Besides adding block-level deduplication, Simpana 8 became the first product to allow writes to physical tape libraries without requiring re-inflation of deduplicated data.

But vendors are working on primary dedupe, too. Riverbed Technology Inc. is preparing a primary dedupe product, although it has been pushed out until 2010. Riverbed began alpha testing its Atlas device in September, with the expectation that it would ship around the middle of this year. But testing showed the product needs more work to make it easier to install and manage, so Riverbed will wait until next year. Atlas will use the deduplication technology that Riverbed employs in its Steelhead WAN optimization products to shrink primary data. Atlas' closest competitor is NetApp's deduplication software for primary data.

Eric Burgener, a senior analyst and consultant at Taneja Group, says Riverbed will also raise the bar with Atlas. "The scalability in a distributed environment is better than anything out there," he says.

Want more stories like this and access to White Papers from leading storage vendors? Then click here to join SearchStorage ANZ.

Read more on Data quality management and governance