JumalaSika ltd - Fotolia

Big data poses storage challenges in Australia and New Zealand

Big data is like digging for gold, except you must keep every single rock. Computer Weekly unearths some of the main challenges facing those mining data

Organisations are getting to grips with the fact that the data generated by and for their business is hugely valuable – and also just plain huge. It contains nuggets of intelligence, seams of rich information and lodes of insight. And, to continue that metaphor, it must be mined carefully to fulfil its potential.

The mining side of things is handled by the new wave of data scientists and analysts, with the software tools of their trade being written almost as fast as the data is generated. But, to stretch the metaphor even further, how do you store the rocks that you haven’t yet fully mined? Where do you keep the ore?

Big data is, by definition, big. There’s a lot of it, and it’s coming into your organisation at high speed. You don’t always know for sure what’s valuable and what isn’t – or what might turn out to be valuable in the future.

The storage of big data can be a major headache for organisations in Australia and New Zealand. A recent study by NetApp and Tech Research Asia investigated the “megatrends” that Australian and New Zealand business and IT leaders believe will affect their organisations over the long term.

According to that report, four in 10 CIOs, and six in 10 CxOs cannot quantify how much data their organisation holds. Of those who can, 70% of CIOs expect the amount of data they hold to increase, on average, by 50% over the next two years. For that same period, 60% of CxOs expect their data to increase, on average, by 61%.

They may be underestimating. Consider a company such as Xero, the cloud accounting firm with offices in both Australia and New Zealand. According to Duncan Ritchie, the company’s chief platform officer, Xero currently manages 760TB of data, growing at a rate of 60% per annum. That’s not something you can simply dump in a server room and hope for the best.

Big data storage has to be managed carefully, because the potential rewards are significant

Big data storage has to be managed carefully, because the potential rewards are significant. “The research tells us that, on average, ANZ organisations perceive an improvement of anywhere between 18% and 32% in a range of financial metrics and operational activities by using data more effectively,” says Steve Manley, managing director of NetApp ANZ.

Much of that data storage management involves the cloud, of course. We know that the majority of ANZ organisations either have a cloud strategy they are pursuing today, or want one,” says Tim Dillon, director of Tech Research Asia.

Xero has one. We use a mixture of traditional SAN (EMC vMax and NetApp) storage and cloud storage (Amazon S3 and Azure),” says Ritchie. With traditional storage we need to plan well ahead to ensure we have sufficient capacity on hand. Prior to moving to the vMax we regularly outgrew SANs and had to undertake complex migrations. With cloud storage, the underlying capacity management is undertaken by the vendor so planning has shifted to a financial focus.”

Sluggish storage systems affect productivity

It’s not only commercial organisations that have to cope with the increasing weight of big data bearing down on them. It affects central and local government too. Tamworth Regional Council in New South Wales recently found that increasing volumes of data were bringing productivity to a crawl. Daniel Murphy, co-ordinator, IT infrastructure and support, says he was spending a lot of his time – “many hours a week” – managing the councils storage and backup solutions.

A lack of storage performance can affect employee productivity, as the council found. Because tape still formed a key part of backups, the IT team needed up to 12 hours to complete a nightly backup, and data restoration for small files could take anywhere between one and eight hours.

The council engaged Dell to revamp its storage systems. That resulted in 80TB of fast storage, along with spare IT capacity that is now leased to other councils in the area.

Read more about enterprise IT in Australia and New Zealand

Doug Williams, director of information management at Dell Software for Asia-Pacific, sees many ANZ organisations struggling with big data. When it comes to storage, the biggest challenge is the fact that they have to manage all of the data. For customers, there might be the perception that the speed or the volume or the complexity, or even the usability, is by itself a challenge. But all of them combined is really what’s new to organisations. They’re not used to having to juggle all four of those things at the same time. It’s all happening at once.”

Not only that, but this must be achieved on top of running the business as normal. Yet the organisations concerned know what they face – and are also aware of the benefits.

It’s the lines of business that are driving this rather than the IT team,” says Williams. So it’s switched around compared to 10 years ago. For example, hospitals have all kinds of patients with huge amounts of patient data being generated. They don’t want to burden the systems and compromise the core business, but they do want to use that data in beneficial ways.”

Andrew Diamond, ANZ storage director for Dell Enterprise Storage, sees a similar story from the hardware perspective. A big challenge is the fact that [customers] have got disparate data sources. They aren’t centralised, nor do they have an easy way of consolidating that data,” he says.

We’re finding that they have very diverse requirements, even within the same organisation. Many have different views of what they want to get out of the data. Some of the use cases can be quite common, such as for security purposes. But then the different verticals have entirely different requirements.”

Among those verticals, the big data leaders are banks and finance companies, telcos and energy, mining and resources organisations. But even some smaller organisations are considering it,” says Diamond. You could perhaps call that big data on a smaller scale.

Tools for data mining

Of course, Dell’s not the only organisation selling the shovels in this new gold rush. Several large hardware and software companies have products to help businesses collate and store their big data so they can make better use of it. And, as you’d expect, new tools arise all the time – such as data as a service (DaaS).

Delphix is one DaaS software provider. With DaaS, users can create virtual copies – think of them as clones – of [data storage] environments using existing storage investments, and they can do so in a fraction of the time otherwise needed, says Delphixs vice-president for Asia-Pacific and Japan, Chris Poulos.

Delphix DaaS software also provides added security through integrated data masking, enabling sensitive data to be scrambled before it ever makes its way into an environment, which drastically reduces risk of potential data breaches,” he adds.

All of this has the feel of a brave new frontier. The vast seams of business intelligence hidden in big data are only just being uncovered. The more data can be stored in sensible and accessible ways, the more intelligence will come out. Organisations in Australia and New Zealand may only be starting to store their big data effectively, but they’re learning fast.

Alex Cruickshank has been a technology journalist since 1994. He grew up in the UK and now lives in New Zealand, where he runs his own writing business, Ministry of Prose.

Read more on Virtualisation and storage