Nobody has summed up the phenomenon with a Moores Law-type formula yet, but it seems that as the global volume of data increases, the rules obliging you to keep it all for many years multiply exponentially.
Nowhere is that more evident than with e-mail. Market research firm IDC estimates that the volume of business e-mail sent worldwide in 2007 will approach five exabytes – about one billion gigabytes – a volume that has doubled in each of the past two years. At the same time, the Storage Networking Industry Association (SNIA) estimates that there are about 10,000 regulations worldwide that compel organisations to retain data.
The move to paperless working was supposed to herald a new age of simplicity and efficiency, but it has actually made the keeping of records more difficult for business, says Hamish Macarthur, chief executive at analyst firm Macarthur Stroud International.
“To archive all business transactions so they can be retrieved and used was straightforward in paper-based working, but as we virtualise everything it has become more difficult to keep the information we need for legal and regulatory reasons,” says Macarthur.
There are a number of UK regulations that enforce the retention of e-mail. The Freedom of Information Act dictates that public sector bodies must be able to supply copies of “recorded information” on request, even if generated before the act came into force in 2005. Through the Data Protection Act, a member of the public can request information held about them. The US Sarbanes-Oxley regulations apply to e-mails containing company financial information for Nasdaq and NYSE-listed businesses and their UK subsidiaries. And there are also industry-specific controls, such as those enforced by the Financial Services Authority, which require all e-mails be held for six years.
In addition, there are the increasingly onerous evidential requirements of court proceedings, which require a verifiable e-mail audit trail from “reliable” systems to be admissible. Providing this quickly can save a lot of money and mean the difference between legal success or failure. Financial services firm Morgan Stanley found this out last year when US courts fined it £7m for not keeping complete e-mail archives.
Organisations are caught in a web of growing e-mail volumes and increasingly regulatory requirements. “In the UK, the primary drivers are regulatory compliance, storage resource management, e-mail serviceability and recovery,” says Glenn Weavind, senior consultant at CA. “Litigation is also proving expensive to businesses – one firm in the City recently spent £250,000 on contractor time recovering e-mails to support a case.”
So, what are the options? Well, you can, of course, use the information store of your e-mail system to maintain an archive, but this a risky strategy, as leaving tens of thousands of old messages sitting in the database will soon degrade its performance.
Niels Abildgaard, sales specialist with IBM System Storage Solutions, says, “Maintaining e-mails within the core e-mail production system will pose significant challenges from an IT operational point of view, which is reason enough to drive the need for e-mail archiving by reducing the size of the production e-mail systems.”
The next option is to off-load old e-mails to storage media. This fulfils the technical requirement of keeping the e-mail server in good shape performance-wise, but means that searching for specific e-mails is not possible other than by relatively laborious processes, which fall short of the highest standards of compliance and waste storage space by, for example, duplicating numerous identical attachments.
So, businesses are increasingly opting for specialised e-mail archiving technologies, which come as software, hardware, services and combinations of all three. At their most basic, the key task of archiving technologies is to automate the process of storing e-mails from the live server after a given period. This ensures that the production e-mail system is kept trim to optimise performance, and that e-mails are retrievable in future.
As you go beyond basic levels of functionality, archiving systems begin to include features such as the removal of duplicate attachments, search functions (including the ability to search for attachments in some cases), tamper-proofing, audit trails and reporting.
Dedicated e-mail archiving systems generally fall into three types. First, the software-plus-storage approach, in which organisations develop or buy archiving software and any necessary hardware, such as servers and storage devices.
In this type of system, policy-based archiving software schedules the copying of e-mail from the information store to a repository, and software on client PCs enables search and retrieval. It is possible to knit together components from different suppliers or to buy all of them from a single supplier, such as HP. HP uses its own software to schedule and provision archiving using its grid-like Smartcells system (see Coda case study).
A more self-contained approach is the e-mail archiving appliance. Instead of buying software, servers and storage from one or several suppliers, all the hardware and software required to begin archiving is supplied in one box. Suppliers of e-mail archiving appliances include the Forensic and Compliance Systems (FCS) (see Guildford Council case study).
FCS’s history is in compliance tools, which it has merged with a hardware system. FCS claims “forensic” standards of e-mail retention that provide an unbroken record of e-mail and instant messaging communications at arms’ length from existing operational systems.
Finally, there is the outsourced approach, in which a third-party software interface connected to the messaging system captures and transmits messages to a service provider’s datacentre for storage.
This method is often hybridised, as is the case with Messagelabs’ service. Software and hardware is installed at the client’s site and is combined with secure storage managed by a third-party provider with full encryption, the keys to which are held by the customer.
Dennis Szubert, principal analyst with analyst firm Quocirca, says, “The advantage of this hybrid model is that because the archiving equipment is local, mailbox indexing, retrieval and similar functions are faster. The off-site provider holds the data, but not the key required to read it.”
Case study: Coda
Financial accounting software provider Coda has 550 employees and operates in 14 countries with 2,500 medium and large user organisations as customers. Annual e-mail traffic is estimated by group IT manager Richard Hall to be about eight million to nine million e-mails, to which spam “adds a zero”, he says.
“We started off by trying to deal with personal folders in Outlook. We used inbox limits and reminders to users, but we could not apply a corporate policy this way and searching at this level was impossible,” says Hall.
“We were dealing with back-up issues and ever-increasing storage demands, and we were spending hours on manual back-up and information retrieval,” says Hall.
“With millions of e-mails passing through the company every year, it was time and cost intensive to archive and search them, and we were aware that we could be leaving the business open to risk. Also, for e-mails and other documents to be admissible in court, you must be able to prove that items have not been tampered with.”
Coda opted to procure an HP-based e-mail archiving system in 2005, and expects to gain complete return on investment this year. The system is based on HP’s Reference Information Storage System (Riss). Riss is an HP technology that ties together software and hardware in a grid-like configuration of so-called Smartcells.
Data is stored securely, with date and time stamping of objects to mitigate risk and prevent tampering or changing of retained records. As e-mails arrive at Coda they are simultaneously stored in Coda’s Riss architecture, which is currently configured at 2Tbytes.
Attachments are archived out of Outlook after seven days, and after 30 days everything is auto-archived and single instanced, so that multiple copies of identical data are not kept.
“We used to have situations where people were spending hours every day searching for things. We had a case recently where we had to retrieve an e-mail from four years ago and were able to find it in seconds rather than go through old tapes,” says Hall.
Other benefits include cutting down on exchange maintenance, as well as significantly reducing time spent on disaster recovery testing.
Case study: Guildford council
Guildford Borough Council has 1,100 e-mail users, and 90% of their communication with the public involves some form of electronic response. This volume alone is enough to generate a considerable e-mail archiving challenge, but, like all local authorities, Guildford has also had to contend with a barrage of legislation enforcing legal compliance.
The introduction of the Human Rights Act, Data Protection Act, Regulation of Investigatory Powers Act and the Freedom of Information Act placed new restrictions on the council’s right to monitor staff e-mail and strengthened the individual’s right to privacy.
“I could go in and look at anybody’s e-mail. There was no trail. If you wanted to, it could be abused, and with privacy in the work place it was obviously in breach.”
In 2003 the council implemented FCS’s Cryoserver, and by 2005 was fully compliant. FCS likens Cryoserver to a black-box flight recorder for e-mail. It records every single e-mail ever generated by or sent to a Guildford-council account and has a Google-like search facility. It also creates a tamper-evident audit trail so an administrator cannot use Cryoserver to look at employee e-mail information without creating an audit trail that is automatically sent to three “data guardians”.
Besides compliance, a key benefit for the council has been the ability to resolve potential disputes. Because Cryoserver keeps a record of all e-mails, it is able to provide evidence that e-mails do not exist as well, as evidence for those that do.
“We had a situation with one of our contractors where there was a dispute over damages claimed for an over-running contract,” says Poole. “The contractor was alleging that an extension of time had been agreed in an e-mail. I tracked Cryoserver and there was nothing there. It is very useful to be able to prove a negative.”
Guildford’s success in implementing e-mail archiving systems has seen it become one of only 28 councils out of 273 in England that have been awarded “excellent” status by the Audit Commission.
Data retrieval techniques for e-mail and databases >>
Tips for data archiving >>
Information security: the route to compliance >>
This was first published in June 2007