Compliance and competitive advantage are driving an insatiable demand for storage.
The demand for storage is driven by the desire of organisations to improve their business intelligence and their need to comply with data retention legislation.
"Even though storage technology has got cheaper over the past five years, the spend has increased exponentially," says Paul Hammond, managing partner at storage services provider GlassHouse Technologies UK. "Back then, storage might have been 10%-15% of an IT budget, whereas today it is nearer 30%-40%. This is because we hoard data. We do not throw anything away because of the regulatory compliance issue of storage."
Storing information electronically in an easily accessible form can directly improve efficiency. Islington Council, for instance, is embarking on an ambitious digital archiving project aimed at improving efficiency, reducing response times and freeing valuable office space. The project follows the successful introduction of the first phase of the council's Xerox Docushare installation, which is managed by its long-standing document outsourcing partner, Xerox Global Services.
The project will make Islington one of the UK's largest users of Docushare. The council is rolling Xerox's secure web-based content management system out across 10 departments.
The project is the main component of Islington's plan to reduce its reliance on hard-copy documents wherever possible, which is part of a wider strategy to support sustainability, improve services and reduce costs. The council hopes that the vast majority of documents it stores will be converted into searchable, digital documents.
Compliance with the Companies Act is another reason why UK businesses have needed to increase the amount of data they retain.
Hamish MacArthur, co-founder and CEO of MacArthur Stroud International, a storage-focused European analyst, says, "When the Companies Act was revised last year, it meant that a company officer could no longer say, 'I am not sure if we have this data or not.'
"If you do not know about the data, but someone has a record of it elsewhere, the company directors can be held responsible, where there has been either sloppy management or someone trying to hide something by allowing data to be destroyed.
"For many years, responsibility for data has been left to individual departments, but that now needs to be centralised in the CIO to ensure that the data is present, can be accessed, and can be trusted."
In the US, the Sarbanes-Oxley Act of 2002 has affected US multinationals wherever they operate around the globe. Jeff Rodwell, head of IT at law firm Reed Smith in the UK, says, "People are becoming aware it is the retrieval of storage that is really important."
The area that really interests lawyers is where there are different requirements in different jurisdictions. A corporation cannot simply divide its data up if it has regional servers, but needs a common set of data to satisfy different laws.
"I do not know of any multinational that has one server per country, because it is antithetical to any contingency planning," says Rodwell. "Multinational CIOs need to have a global perspective. One of our clients, for example, has to review 43 countries."
Any large company has an obligation to provide all data that is accessible, but it will have enormous amounts of data in archives. Demanding information under the Data Protection Act is one tactic that has been used in litigation, and in some cases is an abuse of the act's purpose.
"If you get a flood of such applications, let us say from a campaign group, you can argue to the data protection commissioner that it is an orchestrated campaign. But it does not relieve you unless the commissioner agrees," explains Rodwell. "Moreover, he will not agree within a short period of time, and unless you can prove it is an orchestrated campaign and not a random event of people wanting personal data, you will have to comply."
Rodwell argues that the obligation is not only to have the data, but also to have a system in place that permits you to produce that data on demand. Many companies have now introduced, or are in the process of introducing, systems for complying with data subject access requests, but very few can deal with volumes of requests.
"A large organisation will have personal data on tens of thousands of people - customers, employees, suppliers and business contacts - and every one of them is entitled to ask for the data," Rodwell says. "This is an EU right that is not actually available under Sarbanes-Oxley. It does not matter where the person is resident, by the way - it is not only for UK or EU citizens. I can be in Africa and ask any corporation for my personal data."
Anyone can write to the company, and if it fails to comply, they can complain to the information commissioner, who will investigate and may demand that the company comply.
"If the company does not comply, it will be in breach, punishable by a fine," says Rodwell. "If the breach continues, this can result in contempt of court. This is also about to change because the Department of Trade and Industry has just announced a proposal to make it punishable by up to two years in jail."
And it is not just compliance data driving storage growth. Merger and acquisition activities are also a big factor. When one company acquires another, you would expect the acquirer to need more storage because it will now have more data, but the same holds true for the divesting company too.
This was the case at brewing company Foster's in the UK when the Australian parent sold its lager brand licence to Scottish & Newcastle, while still holding a substantial wine portfolio.
"The main business of Foster's here is premium Australian wines, like Penfolds, Rosemount and Wolf Blass," says Ken Kaban, technology service manager for Europe at Foster's. "I came over from Foster's in Australia about four years ago to help build the wine business up. There was an Exchange 5.5 server held together with sticky tape connected to a 64Kbyte ISDN phone line.
"As we were trying to grow the business, sales staff wanted to send sales presentations to customers with 8Mbyte e-mail attachments. They wanted to incorporate videos and high-resolution images into Powerpoint presentations."
The company has grown in Europe from six to 150 people in this time, which causes headaches when backing up travelling salespeople's laptops from their homes.
"I have always disliked synchronising folders on a shared virtual private network and had some nasty experiences where things have not synched up," says Kaban.
Foster's keeps images of about 60 laptops in case of hardware failure. Kaban began investigating Equallogic's iSCSI storage area network technology using Ethernet instead of fibre.
"It is pricier than other iSCSI technologies, but you save the money in the long term in admin costs," Kaban says.
Another driver is demand for multimedia applications, especially storage-hungry video. Bishop Burton College in East Yorkshire spent £30,000 on a network whose storage soared from 60Gbytes to 4,000Gbytes, so that the college's 3,500 students and teachers could have space to invent videos, store complex data, and surf.
Alan Anderson, computer services manager at Bishop Burton, claims that the college was one of the first in the region to connect to the internet eight years ago.
"Students use digital video cameras to illustrate their work, and this has driven the need for more storage," he explains.
Case study: How Egmont improved its recovery rate
Publishing company Egmont UK sells about 16 million books and 11.5 million magazines a year for children and teenagers. Nearly 200 employees in premises scattered around London work to produce the ongoing streams of new publications, both on PCs and Apple Macintoshes.
"As the technology has been changing, the size of the files has been getting bigger and bigger," says Severyn Sawicki, IT manager at Egmont UK.
"Unfortunately, we are stuck with the tape medium for archival storage for the next five to 10 years, albeit in conjunction with other storage technologies.
"We also back up from disc to disc because, being a media company, the sheer volume of some of our data means that it would simply take too long if we had to retrieve it from a tape backup."
The time needed for back-up was increasing, as was the time and cost of retrieving and restoring data.
In 2004, full back-ups at Egmont's books division, where there are 10 servers, including two that bear the brunt of the heavyweight design files, were taking two days and 13 tapes. When one of the servers went down, restoration of all data took 42 hours.
The conception-to-production cycle for a new Egmont book averages six to 18 months, so while the impact of a lengthy file restoration might be frustrating for users, it would not endanger the business. But for magazines, two days of file non-availability could have greater repercussions for the business.
When the magazines division decided to produce some of its publications fortnightly rather than monthly, the worst-case scenario of up to two days of file unavailability would have been a disaster.
Egmont therefore bought two REO 4000 disc-based back-up and recovery appliances from Overland Storage. "The back-up, which had been taking something like 12 hours on the old set-up, was now taking two hours," Sawicki says.
"Not only are we backing-up faster, but we also have online access to any file that needs to be restored without having to call out the back-up people."