idspopd - Fotolia
On-premise object storage: Key products and use cases
We run the rule over on-premise object storage, key use cases and suppliers, take a closer look at an organisation that has deployed object storage, and assess cloud compatibility
Object storage is the bedrock of cloud computing services. It’s the basis for services such as Dropbox, Facebook’s file system, and is the foundation Amazon’s S3 cloud storage; arguably the largest storage system in the world.
Object technology is usually associated with solving the specific challenges of massive public clouds: petabyte-scale data volumes, highly distributed systems and long-term data retention.
Object storage is known for its flexibility, not for its speed. Internet-based connections simply cannot compete with locally based flash, or even disk-based storage.
But object storage is now moving into the enterprise mainstream, and with that, to enterprise on-premise deployments.
According to IHS Markit, a research firm, 56% companies in North America to invest in object storage.
Interest in on-premise object storage is being driven by the need to store more data, suggests IHS analyst Dennis Hahn. Much of this is driven by the vast volumes data produced by the internet things (IoT) and artificial intelligence, although these are not the only use cases.
“Beyond better scalability and access from any location, object storage offers lower-cost storage management and improved cross-geography data protection,” says Hahn.
Moving to on-premise
Cloud-based storage systems use object technology because it is increasingly the only way to hold vast quantities of customer data, and to do so at a cost, and with the reliability, those customers demand.
Amazon, for example, claims 11-9s for S3. And as Hahn points out, businesses are now using the geographic flexibility of object storage – which separates the data from the metadata – to cut costs and build in more resilience.
Unsurprisingly, the early adopters of object on premises are companies with extremely large quantities of valuable data.
Media organisations are among the early adopters. MediaCloud, a South Africa based media service, is using Object Matrix’s technology to give its customers access to cloud-based and tapeless workflows for editing and archiving video content. DailyMotion, a consumer online video service, uses Scality’s Ring object storage to handle around 3.5 billion videos – or 30PB of data for both production and backup. The company is now using object as its primary storage technology.
Media, though, is not the only industry using object storage. In the UK, Telent provides mission critical data services to industries including defence, transport and law enforcement. Telent uses object storage to provide cloud-based digital evidence systems to UK police forces.
These examples may, for now, be at the extreme end of data storage requirements, but IHS expects enterprise data volumes to grow 33% annually. Nor are businesses making an either-or choice between on-premise and cloud based storage. IHS’s Hahn says they are expanding both.
Using object storage to combat crime
UK technology outsourcing company Telent operates a range of data-intensive, mission-critical systems. These range from defence, to the railway network and road traffic management.
But Telent also hosts applications for law enforcement. Operating as infrastructure-as-a-service, the company supplies digital evidence systems to police forces. This evidence can include CCTV and camera footage, or digital copies of interviews. With police forces using body-worn cameras and even drones, the demand for electronic storage is growing fast. The data is held on a Scality Ring system on HP Apollo hardware.
“We give police forces a platform to upload, validate and analyse evidence,” says Matt Latter, senior solutions architect. “All those assets have to be stored somewhere, so we use object storage for that. It is cost effective to scale, and the cost per gigabyte is really important for petabyte-scale projects. It is always on, and we use the resilient Ring architecture across multiple datacentres. We also don’t need to back it up; you can’t back up petabyte-scale data.”
Because the systems are custom designed, Telent has not faced the compatibility issues that come from marrying block and file-based applications with object storage.
“You don’t have the bells and whistles in object storage that you have in a file system, but S3 connectivity is becoming more prevalent. More applications are being written for S3,” he says. “And it’s transparent to the customer that they are on object storage.”
Moving to object storage, however, brings its own challenges. Moving the technology on premises will cut the latency associated with cloud-based storage, but other barriers remain.
“The biggest problem for object storage on-premise has been that most existing applications expect file or block storage. Object store developers have tried to meet those needs with gateways or NAS-type emulations, but with limited success,” says Bryan Betts, of industry analysts Freeform Dynamics.
“This is changing as object storage continues to coalesce around S3 as a de facto standard, and as newer apps and app architectures come on line using cloud delivery mechanisms and metaphors, and expecting cloud-like object storage.” If applications are S3 compatible, it is that much easier to run object storage in house.
Early applications for object storage include media stores and archives. These are applications where sheer volumes force IT architects to look beyond block and file, and in the case of archiving at least, where performance is less important than reliability.
Read more about object storage
- We take a look at the object storage market and products and find offerings made to build public and private clouds that often provide a link between the two.
- We look at the object storage services of the big three cloud providers – Amazon S3, Microsoft Azure Blob and Google Cloud Storage – and how customers can achieve compatibility between them.
In fact, few applications require object technology – unless driven there by data volumes.
“Very few [workloads] need object,” says Ian Richardson, head of technical sales at Tectrade, a storage value-added reseller. “Object is typically a viable option at scale, usually 500TB to 1PB-plus. This is where other systems are reaching their limits or you’re forced to use the top model of a product family.”
At that point, the advantages of object storage outweigh its complexity and compatibility limitations. But with more organisations looking at hybrid storage, or at least, the capability to move workloads and data to and from the cloud, on-premise object storage starts to make more sense.
“There’s growing acceptance of object storage as a software-defined storage 'underlay', which is then consumed mostly in the form of files and blocks,” says Freeform Dynamics’ Betts.
“You hear this kind of thing from within the skilled Ceph community with CephFS and RBD, for example, but naturally commercial suppliers are doing it too; adding native front-end file and block access to an object store, rather than relying on gateways or emulation.”
In this way, object technology becomes just another storage tool, on premises or off.
The object storage market
IDC’s 2018 MarketScape report lists 13 suppliers of on-premise object storage, split between software only, hardware and mixed suppliers, as well as specialists and established enterprise suppliers.
IBM: IBM Cloud Object Storage is available in the cloud, on a per-gigabyte basis, or as an on-premise system with software or appliances. The supplier claims virtually limitless storage with policy-based tiering and 10-9s reliability. IBM promotes object storage for archiving, backup and recovery and cloud-native application data, but also for AI and big data analytics. IBM’s technology works with objects up to 10GB in size.
NetApp: A long-established supplier of enterprise NAS systems, NetApp provides StorageGrid for storage tiering and cloud integration. StorageGrid is a software-defined object storage architecture which works on private and public clouds. Analysts highlight that NetApp has particularly good integration with Amazon S3. The supplier’s compatible appliances range from 48TB to 720TB capacity.
Cloudian: Cloudian is one of the “pure play” object storage suppliers with, it claims, technology that scales up to 100PB, or exabyte-scale, deployments. Customers can choose to use appliances or run software-defined storage on their own servers. Cloudian also works with Seagate to produce 1.5PB, HyperStore Xtreme nodes. These can be combined to put 18PB of storage in one standard rack.
Dell EMC: Dell EMC’s ECS is a hardware-based solution, with up to 8.6PB per rack unit based on standard 12TB disks. But the supplier stresses multiprotocol support, including file and Hadoop HDFS, as well as compatibility with Amazon S3. ECS works alongside the supplier’s Isilon scale-out NAS technology
Scality: Scality is a pure-play object storage supplier, but also works in the large enterprise market with HPE; HPE does not have its own object storage line. Scality Ring is software defined storage, and the supplier emphasises speed of deployment (it says it can be done in an hour) as well as point-and-click provisioning to Amazon S3 storage. NFS v4, SMB & LinuxFS (fuse). It claims it operates at up to 100PB of data.
Object storage and the cloud
Another reason organisations pick object storage is for compatibility with the cloud. As most cloud storage is object-based, using object technology on premises or on private cloud infrastructure should make for easier transitions between storage locations.
According to market research firm IHS Markit, by the end of this year 66% of organisations to deploy converged infrastructure, 56% hyper-converged infrastructure (HCI), and 66% hybrid cloud storage.
Much of this is being made possible by the de facto adoption of Amazon’s S3 standards and application programming interfaces for on-premise and hybrid storage, as well as for a growing range of applications.
“Major cloud suppliers, including Amazon Web Services, Microsoft Azure, and Google, all have their storage based upon their object storage technologies, and this is driving a lot of the development and innovation in object storage,” Ian Richardson, head of technical sales at Tectrade.
“One the limiting factors at the moment is they all run their own developed protocols for interfacing with their object storage.” On premises, though, AWS S3 is now well established, and applications increasingly support Azure Blobs and Google Cloud Storage, natively or through a translator.
For businesses, this allows data to move from on-premise to cloud storage, and back again, as the workload requires.
