Meeting the demand for data storage

As the information storage needs ...

As the information storage needs of many large enterprises grow and become more complex, IT executives must have better policies to guide their efforts.

For many companies, data storage has become one of the fastest-growing parts of the IT budget, thanks to enterprise-wide transactional systems, massive data warehouses and explosive growth in e-mail traffic. Although the storage market has been growing 7% a year, in some large enterprises, networked data storage -- disk drives, tape systems, specialised network gear -- and the people and software to manage these has grown by 20% or more annually, even where IT budgets have barely budged.

The demand for storage has grown more than 50% annually in recent years, a rate faster than the rapidly decreasing unit cost of storage. If storage costs continue their rapid rise, enterprises must manage storage more efficiently if they are to exploit new forms of information such as more detailed financial data, digital images in life sciences or video in media companies. It is a daunting challenge for IT managers to combat the rising volume, cost, and complexity of storage. They must make decisions on a wide range of issues about storage architecture, design, operations and performance.

Business users -- the internal customers of IT -- commonly ask for more than they need because they are not accustomed to making decisions about trade-offs between storage cost and quality. In many cases, storage decisions result from policies designed by legal and regulatory teams, which focus primarily on risk mitigation. Some IT managers err on the side of caution to guard against personal cost, were data to be lost. As a result, companies store non-critical data on expensive configurations or keep too many copies of information in too many places.

IT managers must implement policies that define storage options and help business users set priorities based on economics and needs. IT managers need to engage with business leaders in a more systematic way on the trade-offs of investments in risk, performance and cost. This includes providing business customers with standard storage configurations that define clear service levels and unit costs. By working closely with the business, IT leaders can help internal customers make better decisions to balance the competing demands of storage, cost, resiliency and performance.

IT managers can then take important steps to streamline the process for making new storage requests, planning future demand and procuring hardware and software. One large enterprise reduced its storage costs by 40%. It spent the capital on new capabilities and applications it needed in the immediate future.

Struggling to manage storage

New applications, more complex business analytics and the need to meet regulatory requirements fuel demand for additional storage. Pharmaceutical companies, for example, are considering digital-storage options for all their imaging data such as MRI scans, a change that could hasten new products to market but double the storage needs of these companies. Similarly, a hedge fund's storage costs could increase exponentially as it captures and stores more granular "tick-level" trading data to meet financial regulations and develop trading strategies.

In many enterprises, the storage environment has grown so rapidly it has outrun the IT department's ability to manage it effectively. As a result, many organisations are experiencing a common set of problems.

Too many copies of data

In many cases, companies find it quicker and easier to back up information by making multiple copies. But across a large organisation, this approach increases storage volumes dramatically as companies keep too many copies of business data. An electronics manufacturer, for example, kept as many as 12 copies of its core production applications, each with its own full copy of data. This tripled its storage needs for these applications.

The problem is further complicated by rampant replication. Network storage derives its resiliency, in part, from the replicas of underlying business data it creates, but multiple replicas increase the demand for storage. An 11 terabyte database can require 170 terabytes of raw storage for its replicas, depending on its configuration. One pharmaceutical company stored ten copies of clinical trial information indefinitely when four would have met its legal obligations. Few companies archive and purge outdated information aggressively, either because they are risk-averse or they believe it is not worth the effort.

Placing data on inappropriate storage configurations

For every storage decision, managers have to consider the type of network, the size of drives and the required mirroring and replication. Each choice has implications for cost, but these decisions are often made haphazardly and can place data on storage alternatives of a higher performance than applications require. One electronics manufacturer stored 70% of its test and development data on expensive storage options, even though much of this data typically did not have stringent requirements for uptime or performance.

Buying excess capacity

Well-managed companies use 80% or more of their available storage, but in others that figure hovers around 40% to 50%. One large IT organisation used only 50% of its storage capabilities. Some of its individual storage systems were at just 10% to 20% of capacity and one of its businesses used only 33% of the entire amount of storage it had requested. One could argue storage always needs to catch up with capacity, so having a little excess is not so bad. But as storage consumes a growing share of IT budgets, the cost of maintaining excess capacity becomes material.

Why effective management is difficult

Poor storage decisions stem from a variety of underlying issues, notably time pressure, information gaps and breakdowns in the relationship between business users and IT departments. Frequently, these challenges are exacerbated by a capability gap. The storage manager is promoted for technology skills rather than for business and financial acumen. This focus on technology limits the manager's ability to understand the total cost of storage and to communicate those costs and the related trade-offs in business terms to the rest of the company.

The article is continued on the McKinsey Quartley website >>

This article was originally published in The McKinsey Quarterly Copyright © 2008 McKinsey & Company. All rights reserved. Reprinted by permission.

By James M Kaplan, Rishi Roy, and Rajesh Srinivasaraghavan, © The McKinsey Quarterly, June 2008

About the Authors:
James Kaplan, a principal in McKinsey's New York office, leads the technology infrastructure practice for McKinsey's global IT group Rishi Roy is a consultant in the New York office and Rajesh Srinivasaraghavan is a consultant in the Mumbai office.

The authors would like to thank Edward Hsu for his contributions to this article.

Read more on Networking hardware