This article is part of our Essential Guide: Guide to storage performance and specs

Storage performance metrics: How suppliers spin performance specifications

Array makers’ storage performance specs are not always what they seem. Storage analysts explain how suppliers spin spec sheet figures

Caveat emptor – let the buyer beware – is as true in IT as any walk of life; maybe more.

Organisational buying decisions can be fraught with complexity and have consequences that are far-reaching – and can be career-limiting.

So when it comes to buying storage products, what should you look out for in suppliers’ storage performance metrics?

Buying storage centres on performance – capacity, latency, throughput and so on – and there are numerous ways manufacturers massage spec sheet figures. But, storage products also come with features, which may or may not be desirable in their own right, but can also affect performance.

So, what are the key ways suppliers mislead customers?

To find out, we asked a range of storage industry analysts to list and explain the ways they see storage suppliers put a spin on their marketing.

George Crump, Storage Switzerland, president and founder

Do you get the IOPS they say you will?

Buyers need to look at spec sheets in the context of their own environment and compare apples to apples.

For example, Pure Storage, Violin Memory and IBM all publish IOPS benchmarks, but you need to know parameters such as block size and what features were enabled during testing. Features such as data deduplication, compression and thin provisioning affect flash and its low latency in a way that was not an issue with spinning disks.

Beware of benchmarks

Does a benchmark give you enough granular detail to enable you to translate it to your environment? Two tests are valuable – raw IOPS tests and Loginvsi, which fires up a bunch of VDIs and runs them through day-to-day tasks to replicate real world testing.

Greg Schulz, StorageIO, senior advisory analyst

Dodgy performance claims

Startup and mainstream suppliers of disk, flash and hybrid flash storage systems like to toss out performance claims, and sometimes those claims get backed up with an actual metric.

One of the classics is when a supplier says they are 3x (or pick your own number) faster than their most popular competitor and twice as fast as the previous version of their own solution.

What they don’t usually tell you is that they are comparing their newest, optimally configured, lightly loaded system against one of their competitor's oldest and poorly configured systems.

If a supplier makes a claim that they can do “X” number of IOPS or “Y” throughput without any other information, ask them about context

 Greg Schulz, StorageIO 

Not all IOPS are equal – context matters

Suppliers will say a storage system can do 1 million or 2 million or 8 million IOPS, but you later find out those are 64byte (or smaller) sequential read I/Os that are being cached. This might be real-world for some applications, but out of this world for most others.

So, context matters. If a supplier makes a claim that they can do “X” number of IOPS or “Y” throughput without any other information, ask them about context. Questions should include what size I/Os, whether they are reads, writes or mixed, random or sequential, what was the response time, how much storage space was used and how was it configured?

Are those sales figures for real?

When a supplier says they have shipped 500 systems, is that the total number of instances, or the number of controllers/nodes/processors? For example, if an array maker has a dual controller configuration and says they shipped 500 systems, is that actually 250 arrays with two nodes, or did they ship 1,000 controllers with 500 systems? For cluster, grid and scale-out architectures things get even more interesting, if the supplier has, for example, four, eight, 12 or more nodes in a system.

Thanks for the memory – when it exists

When suppliers quote memory and storage space capacity, they might say a system has 8GB of cache. But, is that 2x4GB if there are two controllers/nodes/processors/systems? And if it’s a dual controller array, is half the cache memory in each node set aside for the partner controller, meaning there is only, for example, 2GB usable per controller?

Some suppliers will also count cache memory for data as well as their own internal memory to show a larger number.

Tony Lock, Freeform Dynamics, programme director/analyst

The important step is to understand what you really need and try to look at the whole field

 Tony Lock, Freeform Dynamics 

Don't pay for performance you don't need

I don’t know many organisations that really need or want peak performance. Most customers need products that provide the performance level and characteristics the services supported require. Not less. Not more. Raw performance specs of systems in preconfigured tests are rarely of any use, because test workloads don’t have much in common with the real world.

Do quoted features match your requirements?

Buyers need to accurately identify the features they absolutely have to have, those that might be useful and the rest for which they have no likely use in the foreseeable future. Then they should line this up with architectural requirements to fit into the existing landscape with minimal disruption and their “comfort zones” in terms of skills.

Most storage suppliers have core features that most customers require. Some have detailed technical differentiators that may be important to some clients, but may not be of interest to many others. This is the way IT has always been. The important step is to understand what you really need and try to look at the whole field, not only those suppliers you know well.

Great feature, but do you need it?

Very, very few IT suppliers do a good job of marketing use cases for their solutions. Nearly all are very close to the technology and love talking about bits and bytes, not real-world use. Most rely on their pre-sales teams and channel partners to work these out, although this is changing slowly.

The biggest area where suppliers could improve is to tell customers (and channel partners) not just what technology and software management features their products possess, but to help explain where and why technical features could be important. Few suppliers in storage or any other part of IT do this well.

Valdis Filks, Gartner research director, 
information technology infrastructures

Suppliers get artificially high performance by avoiding writes to disks and writing to controller cache instead

 Valdis Filks, Gartner 

Beware of benchmarks - again

Suppliers performance claims generally should be taken with a pinch of salt. Open benchmarks – such as those from the Storage Performance Council (SPC) – are better, but they also need to be read carefully. Generally speaking there is no ideal benchmark, but in terms of transparency the SPC ones are quite useful, because they show the configuration and the price per IOPS.

Some suppliers say benchmarks are useless but then quote benchmarks for some products and not others. This is a good indication of which products do not perform well. The ones without open and transparent benchmarks are the ones to be wary of.

Suppliers cache in

Suppliers get artificially high performance by avoiding writes to disks and writing to controller cache instead. Sure, in reality lots of I/O is cached, but this depends on the workload.

Software smarts are key

Suppliers use the same drives and Intel CPUs in their controllers. This means the one that produces the fastest controller will have a short-lived advantage because, in months, most other suppliers will have the same hardware in their controllers. So, the differentiating factor is the storage software's efficiency.

New storage arrays use memory page management techniques, while older arrays are stuck in Raid groups. Raid group designs do not adapt well to new data reduction techniques, which require higher array performance.

Read more on SAN, NAS, solid state, RAID