jules - Fotolia
Advertising content production agency Hogarth is set to expand the volume of storage it holds in Cleversafe object storage to nearly 5PB when it deploys further capacity early in 2016. The upgrade comes two years after Hogarth rejected cloud storage as unsuited to its workloads and opted for in-house object storage as a disk archive repository.
Hogarth produces video, audio, print and digital content for advertising campaigns. These are usually global in nature but must be adapted to local markets. This workflow takes place across key locations in London, New York and Singapore, with many satellite sites and 43 offices in total. Offices are connected by private WAN connections with bandwidth of 10GbE between main sites.
When the first phases of production work are carried out to completion, media is archived. But files – with often many individual files associated with a project – must be fairly quickly accessible to update content.
Primary storage is on Quantum StorNext arrays at each site. These now operate as high-speed cache for working processes but originally provided all storage capacity for Hogarth. As these got closer to capacity Hogarth technology vice-president Mark Keller looked around for a suitable archive tier of storage.
Keller said: “We weren't archiving at all originally. The Quantum arrays replicated across locations. We were close to capacity and we could have expanded but you don't always want to, even if you can. We had to seriously consider an archive system.”
Initially he knew he didn't want tape. Keller said: “Tape wouldn't be compatible with the way we work. There are thousands of files associated with projects. That makes it hard to sift through tape. Projects are often long-lived and we need to open up the archive and make big restores of entire datasets.”
Public cloud was also looked at but rejected. “We looked at Amazon S3, Azure etc but these are more expensive per TB than Cleversafe if you go for the resilience options because they use replication so you duplicate your costs. Also, as we have our own network we would not get the bandwidth we need from a cloud provider. If you're not close to them you're in trouble. The bandwidth is not there to support the deltas we have every day. We have file sizes of 500GB up to TBs and tens of TBs and it's not trivial moving this amount of data around.”
He added: “We wanted a disk-based archive we could scan quickly and get to the datasets we needed.”
Keller's team initially deployed around 3PB of Cleversafe capacity. It has now ordered another 1.7PB that will be installed by January 2016.
Flat system offers reilience
The Cleversafe product set consists of three types of appliance: The dsNet Manager that performs management operations; Accesser appliances that front-end requests from applications, and; Slicestor storage nodes.
Slicestors house SATA disks. Access is via a Rest (representational state transfer) API and Amazon S3, but for traditional file protocols such as NFS and CIFS you need a third-party gateway. Hogarth uses Nasuni filers to front-end requests to the Cleversafe object storage.
Cleversafe is inherently multi-site, with data slices stored in multiple locations to aid resilience using erasure coding.
Object storage suppliers such as Cleversafe use the Rest protocol to store very large amounts of data in a flat system where files are identified by metadata.
This contrasts with traditional file systems that use a tree-like hierarchical structure. These place limits on file systems because performance overheads increase as the file system grows into the millions and billions of files.
Object storage aims to sidestep these performance difficulties with its flat structure. It protects data using a form of the erasure coding method, which splits objects up and stores parts in different places. The method can reconstruct data if any are lost.
Keller summed up the benefits thus: “We can lose eight of our 19 Slicestors and the system still runs. Overall Cleversafe is just the best value for money against other disk products and public cloud. Performance-wise we get close to linespeed with our 10 GbE networks. Some argue tape would be cheaper, sticking it in an old coalmine or something – but that would just amount to delete-to-tape.”
Read more about object storage
- All but one of the big six storage suppliers have object storage products that target public and private cloud environments and/or archiving use cases.
- Object storage from specialist suppliers aim at cloud, archiving, HPC and big data with products that vary by features and between hardware and software-only products.