Edelweiss - Fotolia

Why is the data storage industry locked in by hardware?

The data storage industry revolves around the coupling of storage hardware and software. Will it evolve to allow for the separation of the two?

When you stop and think about it, the storage systems we use are all made to a pretty simple formula. The SAN, the NAS, the disk backup and data deduplication hardware -- from the lowliest desktop NAS to the enterprise Fibre Channel behemoths, they are all made from two ingredients.

The first key ingredient is the thing that actually effects data storage: the hard drive, of which there are relatively few variations that are almost completely interchangeable. The second key ingredient is the controlling software, the operating system. This is the bit that enables storage vendors to make billions of dollars and to lock customers into their product eco-systems.

There are some hugely wealthy empires built on this structure, which in some senses appears quite fragile. After all, it isn't the hardware in a storage system that is unique but merely the software; it's literally a set of ideas about how systems should operate set down in coded form. It's a wonder that software-only products and, in particular, open source software have not taken off in storage to a greater extent than they have.

We've seen the revolutionary effect of a separation between hardware and software elsewhere in IT. In the world of servers, open source, specifically in the form of Linux, shook up a large chunk of the industry over the past decade or so. Look back to the turn of the millennium and the server scene wasn't dissimilar to what you'll find among the giants of storage today. There were several large players in the Unix server market, each with its own flavour of Unix that ran only on its specific version of the RISC processor: HP with HPUX, IBM with AIX, Sun with Solaris, Compaq with Tru64 and so on.

That world collapsed as Linux became a viable alternative as a server operating system for the type of jobs formerly the preserve of Unix and that, crucially, were able to run on commodity x86 servers. Suddenly, the existing in-built lock-ins of the Unix world evaporated, and Linux became an economic option for core enterprise apps such as SAP and database serving.

This separation of hardware and operating software by open source, and in particular its Linux manifestation, brought some huge changes in the IT landscape, but it didn't conquer the world. The desktop, for example, remained stubbornly resistant to Linux except among a tiny number of nerdy enthusiasts and was certainly never on the cards for the enterprise. The key lack of traction here was not only a result of the poor evolution of Linux as a desktop OS, but simply because there was no need to sidestep the processor/OS lock-in that existed with Unix servers.

Storage subsystems are, however, almost as far as you can get from the commodity nature of an organisation's desktop estate. SAN and NAS systems are often bought in pairs or clusters of multiple devices and are the biggest ticket items in the data centre. They also manifest a clear hardware/software lock-in. So, are they vulnerable to the type of wedge that open source drove into the server market?

Well, there is a small but significant current of open source storage available.

Sun's ZFS, for example, is a file system that is fully featured and scalable and, although not on general release, can be incorporated into Linux and the Free BSD operating systems and unified storage systems built from it using commodity hardware. Some small vendors, such as GreenBytes and Nexenta, have done precisely this, but it's a project theoretically within reach of an in-house IT department.

Meanwhile, Red Hat Enterprise Linux Advanced Platform provides shared-storage access, including support for CIFS, NFS, iSCSI, Fibre Channel and FCoE. And FreeNAS, which is built on FreeBSD Unix, supports CIFS, NFS, FTP, iSCSI, Rsync and AFP (Apple File Protocol).

So, the idea of an open source storage revolution, with an operating system and file system built into commodity hardware, isn't too great a stretch of the imagination. Illustrative of this is the fact that EMC, for example, uses versions of Linux in its Rainfinity, Data Domain, RecoverPoint, VPlex and Avamar appliances, albeit with code bases customised to those implementations.

And the potential separation of storage hardware and software is not restricted to the use of open source OSes. Vendors such as LeftHand and DataCore, for example, already offer their SAN products as software, while the likes of Nasuni and TwinStrata sell virtual NAS devices that only exist as software too.

Logic seems to suggest that one day it will be common for customers in businesses of all sizes to buy storage operating systems, processing power and disk hardware separately. But when that day will come is open to speculation. All kinds of commercial interests and practical challenges lie between here and there, and that is a subject for reflection in a future editorial.

Next Steps

The fure of the data storage industry in 2017

Read more on Storage fabric, switches and networks