JumalaSika ltd - Fotolia

Fourth-generation flash to rewrite software development

Flash has outstripped Moore’s Law. If software development caught up, the solid-state datacentre would be a cost-effective reality, bringing a step-change in agility and performance

Are you ready for a fourth generation of enterprise flash storage?

That is, flash used not just to accelerate IT, but to transform it – and in the process to enable whole new ways of building applications and give IT the agility it needs to keep up with today’s business.

On the hardware side, most of what is needed is already here.

The first generation was flash as an adjunct, to accelerate servers and storage systems, including creating hybrid disk/flash storage arrays.

The second generation was relatively simple all-flash arrays as point solutions to accelerate specific applications that were I/O-bound.

The third generation was when all-flash arrays began to acquire the sort of management capabilities that are standard in traditional enterprise storage systems. This was when a company could move many, or all, of its applications onto the same shared all-flash array.

This is the key enabler for generation four, which expects storage to be built for general-purpose use, not as a point solution.

But the real change to fourth-generation flash does not come from hardware.

It comes with the realisation that flash storage systems now allow IT to do things that simply were not possible before. That is because these new storage systems are more capable.

They are faster, they save space in the datacentre, draw less power and require less cooling, and provide far more consistent performance than mechanical storage.

Running ahead of Moore’s Law

Underlying this is the fact that enterprise flash, with its massive density growth, is currently running ahead of Moore’s Law. And the corresponding cost reductions have the potential to turn the programming world on its head.

For decades, programmers have been taught to avoid writing to disk wherever possible, but that is no longer necessary. Tasks in areas such as analytics, which would previously have required too much disk I/O, or simply taken too long, have now become practical.

“When I looked at the future datacentre, it was clear there was a significant disconnect between the modern business world and IT,” says Dani Golan, CEO of flash storage developer Kaminario.

“Business has changed dramatically over the last 15 years – it’s agile, it scales, it’s fast to change and flexible. IT is the opposite, so every change in IT, whether it’s the cloud, big data or 100% virtualisation, every change is an attempt to close that gap.

“Flash is the biggest revolution in IT today, maybe the biggest since virtualisation. We are not just solving a storage problem, we are rebalancing IT. But there is a whole generation we need to re-educate. If you were taught programming in the last 45 years, you were taught ‘Don’t go to disk’. We are turning that on its head – we say ‘Go to disk as much as you like.’”

Some major software developers have already adapted their platforms to take advantage of flash memory’s attributes, and a few companies already deliver flash-specific versions of their software.

Read more on flash storage

  • Computer Weekly surveys an all-flash array market in which the big six in storage have largely settled on strategy, but key new technologies – such as TLC flash and 3D NAND – are emerging.
  • Computer Weekly surveys the startups and specialists in the all-flash array space and finds a market in which advanced storage features are becoming the norm, while suppliers battle down to $1/GB.

However, these are still mostly second- or third-generation implementations, where flash is used to accelerate existing programs or processes. Applications have been rebuilt, for example, to have faster and fewer write cycles, or to remove the intentional I/O delays that were sometimes used in the past to make the processor wait for a hard disk to catch up and deliver data.

It is a hugely disruptive opportunity, says Carlo Wolf, EMEA VP at all-flash pioneer Violin Memory.

“You relieve application developers of a constraint, and now they can develop differently,” he says. “First, you accelerate their work anyway, but then the storage moves closer to the application, so now companies can design their applications differently. The structure of writing applications so far has always been with the expectation of disk on the back end.”

Wolf says that in one extreme case, during testing with a customer, Violin encountered an application that would not work with a flash array. “The storage was too fast for the application, which didn’t expect the data to be ready yet,” he adds.

Flash-enabling new business processes

The real generational change will come when organisations take the next big step and build whole new programmes and processes, or rework existing ones in ways that were simply not possible before flash.

Wolf offers the example of a gas distribution company that changed its logistics completely because it can now wait until 5am to do its routing, when staff are actually loading the trucks, instead of having to start at 2am.

Rohit Kshetrapal, CEO of Tegile, which has developed software-defined flash arrays, offers another example closer to home. His arrays can, with their owner’s permission, report usage data back to Tegile. This allows the company to predict when the customer will run out of capacity or cache hits, and to proactively offer upgrades or maintenance – and it only works because of flash.

“It’s a huge amount of data, and we couldn’t do it without flash because collecting the data would have slowed the systems down,” he says.

But how far has the wider business moved in this direction?

A recent survey carried out on behalf of Kaminario suggested flash adoption is still mostly driven by affordability and the desire for higher performance – which suggests most organisations’ thinking is still at generations two and three. Also, it is implemented in less than a third of UK datacentres, and even when you include the US, very few organisations are 100% flash so far.

Wolf cites other research that suggests perhaps 10% of the global installed base of storage has gone all-flash, for its primary storage at least. That leaves 90% of a $20bn industry still to move, he says.

Implementing flash generation four

A move to the fourth generation of flash is about teaching developers and systems analysts new ways of thinking that make use of flash’s unique properties, and then using those properties to re-engineer business processes.

It is not about the flash array hardware, but it does place some requirements on the hardware, and if the underpinnings are wrong, you will not be able to make everything else work as you want.

General-purpose storage is essential. Because the change is so sweeping, it almost goes without saying that you need an array designed to handle mixed enterprise workloads, not something designed more as a point solution. Some current enterprise flash implementations will therefore work well as a foundation for generation four, but others may not.

Scalability is another essential, ideally both scale-up and scale-out. Part of the fourth-generation flash story is the transition to a solid-state datacentre. There may still be mechanical storage in your backup and archive tiers, but it will be all-silicon on the front line. The last thing you need here is array sprawl, with independently managed arrays added in discrete siloed chunks to meet growing capacity demands.

Cost-effectiveness is another objective. There are several ways to build flash arrays, for example using solid-state drives, specially designed flash cards, or a software-defined storage approach to bind together commodity storage units. Be sure to choose one that not only scales well, but does so economically and can also take advantage of advances in flash density and price performance.

Of course, there are challenges in moving to fourth-generation flash. There is a lot of legacy thinking to unlearn and baggage to abandon, and there are new techniques to take on board and the new flash-enabled datacentre infrastructure to plan. But get it right, and flash could be the final piece of the jigsaw that enables IT to reboot the business it serves.

Read more on Software development tools