But the industry's move to compliance really accelerated with the fall of Enron Corp. in late 2001 and WorldCom in mid-2002. Early this decade, each world-class company (along with numerous smaller companies) unraveled to reveal an intricate web of fraudulent business practices and questionable accounting methods -- quickly eradicating hundreds of billions of dollars in shareholder equity and shaking investor confidence at every level. The U.S. government responded to this by imposing strict regulations in the Sarbanes-Oxley Act of 2002 (SOX) administered by the Securities and Exchange Commission (SEC). Failure to comply with SOX carries large fines and imprisonment.
There are many other regulations that affect particular states or specific industries. For example, SEC Rule 17a-4 for the financial industry requires data to be stored offsite on nonrewritable media that is indexed and easily retrievable. The National Association of Securities Dealers (NASD) has imposed rules 3010, 3012 and 3013 to address a CEO's supervisory policies and procedures within organisations. Some states impose additional regulations that affect any company that does business in those states (e.g., SB 1386 in California). International financial regulations are also emerging in Europe.
The concern with most regulations today is their sweeping and general wording. For example, SOX only dictates which records must be retained and how long they must be stored. But the regulation does not specify how those goals should be accomplished, so each IT department is left to implement storage, policies and practices that they hope will satisfy compliance needs. In many cases, it is not completely clear just what data needs to be saved or how long, often leading company executives to "save everything." This requires more sophisticated tools to search through burgeoning volumes of data.
Compliance practices and strategies
Compliance rules and regulations can differ between states and industries, so organisations typically tailor their storage practices to achieve compliance in a specific business. Still, successful compliance strategies typically involve three distinct areas: data integrity, data retention and data security.
Data integrity assures that information has not been changed or lost through corruption or media failure. This usually involves read-only media like CD or DVD disc, along with write-once disk platforms like content addressed storage (CAS). A discussion of integrity also involves data restorability schemes like backups, migration, replication and disaster recovery, in addition to the company policies and procedures in place to manage those activities.
Data retention defines how long data must be kept by an organisation. This is usually the main focus of any compliance regulation, but "keeping" the data just isn't enough; data must be retrieved quickly to meet the demands of compliance auditors or legal discovery requests. Much of the problem with today's storage isn't keeping the data, but wading through that data to find specific files within a huge storage environment. Another retention issue to consider is render ability -- the ability to read data after a period of time. For example, email records saved today may not be readable by operating systems and applications 20 years from now, even if the media is completely intact. Part of retention planning should involve periodic conversion and migration to ensure that the data remains readable even as the enterprise and its platforms evolve.
Data security ensures that only authorised individuals can access data and that policies and procedures are implemented to protect data against loss or theft. Most compliance regulations address data security and access, and increased attention to security issues is driving the evolution of encryption tools for tapes and servers.
Compliance costs money. Part of the cost involves the hardware and infrastructure needed to meet retention and integrity requirements -- disk, tape and other media. Another part of the cost is in software to manage the storage process and actually find data as needed. Finally, there is a cost to draft, implement and maintain the internal policies and procedures needed to meet compliance regulations. For some organisations, the cost of compliance can be staggering. Back in 2004, General Electric Co. revealed about 30 million dollars in compliance costs just to meet SOX regulations. Smaller organisations will typically incur significantly less expense, but costs are always an important consideration in a move to comply.
There is no single set of compliance products, but leading storage vendors provide a wide range of hardware and software products that can accommodate compliance efforts. EMC Corp. is typically a leader with recognised disk-based archiving products like Centera and Clariion. Network Appliance Inc.(NetApp) is also a notable player with SnapLock and LockVault software running on NetApp FAS and NearStore storage platforms. The Axion storage system from Avamar Technologies Inc. handles legal discovery and support for regulatory compliance. IBM offers the DR500, while Hewlett-Packard Co. provides the Reference Information Storage System (RISS).
Email archiving software is another popular category of compliance products. EMC provides Email Xtender, along with Enterprise Vault from Symantec, Message Manager tools from CA Inc., and Enterprise Archive Solution from Zantaz Inc. Many of the email archiving products now allow users to manage unstructured data and email within the same product.
Enterprise content management systems with workflow and information lifecycle support are available in products like EMC's Documentum, the P8 software platform from FileNet Corp., Hummingbird Enterprise from Hummingbird Ltd., with additional tools from IBM and Intervoven Inc. Data search, indexing and migration tools are available including auto-stor software from Arkivio Inc., the IS1200 line of appliances from Kazeon Systems Inc. and a family of Active Policy Management tools from Orchestria Corp.