EXCLUSIVE IBRS ANALYSIS: How to avoid archiving project failure

Independent analyst firm IBRS has researched Australian and New Zealand organisations attitude to archiving. This article contains their report and is exclusive to SearchStorage ANZ readers.

Conclusion: As discussed in “Backup is not Archive!1 all IT organisations should evaluate deployment of an archival platform. However, based on numerous client conversations and a recent survey, it is clear there are significant project risks in implementing archiving. One-quarter of archiving projects take more than two years to implement and nearly half of IT managers state that they would not recommend the archiving product they had selected!

Observations: Over the last 18 months IBRS spoke to a number of organisations that have experienced significant problems implementing archiving. To collect more information, IBRS conducted a survey 2 on archiving in Australia and New Zealand in September 2008 through to November 2008. The survey focused on the business drivers and the status of archiving implementation, and attracted 51 responses from a wide range of organisations.

Why Do Organisations Implement Archiving?

Most respondents stated that technical benefits – reducing storage hardware and operational costs, avoiding purchasing more primary disk and reducing the backup window – were more important than the business benefits of enabling compliance and e-discovery. This indicates that archiving is being viewed as a technical tool to deal with the growth of data and its storage requirements. Part of the reason for this is that IT managers are finding it difficult to engage the business managers in conversations about the benefits of archiving.

A common IT complaint was the difficulty in getting business to participate in defining archival policies. If an organisation has a Business Records and Document Management Framework, IT managers should leverage this framework to drive creation of archiving policies, otherwise treat archiving as a technical project to create a storage tiering architecture.

What data are organisations archiving?

The survey found that email is the most important data type to archive; with 93% of respondents saying this was either “important” or “very important”. File data was the second most important with 85% saying this was either “important” or “very important”. On the other hand, SharePoint was the least important data type; with only 35% of respondents saying this was either “important” or “very important”.

Given the massive growth rates of email and the fact that most businesses consider email as an essential infrastructure, it is not surprising to find that email archiving was the most important.

The fact that SharePoint scores low is due to the relative immaturity of SharePoint projects and the fact that it has usually been implemented as small tactical projects3. As the amount of information in SharePoint repositories grows we expect the importance of SharePoint archiving to match that of email archiving. However, archiving SharePoint will be complicated as the product has multiple ways to store and access information (folders, SQL databases, formal document management solutions and so on).

In the survey 72% of respondents had tried to implement an archiving solution. Clearly there is a lot of activity in this space. However, only 60% of these projects had been accepted into production, with an alarming 5% being cancelled and a further 16% experiencing significant delays. A third of all projects have been in progress for at least 12 months and a surprising 23% more than two years! Worse, less that 60% of respondents would recommend the archiving product they had tried to implement!

This meshes with IBRS’s experience that a significant proportion of organisations attempting to implement archiving are experiencing great difficulty in making the products work. Based on this survey, and a large number of client interviews, we found the following common drivers for the archiving projects being delayed or failing.

Email platform: We found that Lotus Notes based organisations struggled to find products that work with their email platform, though in recent months this has changed somewhat. When short-listing products we recommend confirming how mature the archive platform is for the organisation’s existing products, with special attention especially paid to email platforms.

Decentralised deployments: While email is often a centralised deployment, it is still common to find multiple email servers distributed across the organisation. Further, users’ files are usually located in the same office as the users and are highly distributed. Depending on the product, the introduction of archiving can make the management of this distributed environment much more complex.

Archiving data distributed over a wide area network (WAN) creates significant problems because most archiving products are only designed to work over local area networks (LANs). When an application accesses a file that is archived over a WAN, the time taken to recover the file can cause serious problems for the application. For example, office applications, such as Microsoft Word and Excel, can fail when accessing large archived files over a WAN. These problems are compounded by misleading application error messages – Microsoft Word, for example says the requested file is missing or corrupted.

Archiving across a WAN is not a transparent process and can make the file and email environment more expensive to manage and less usable.

Weak Products: The other common reason we found for project failure was poor product sets. In particular, EMC’s products stood out as being troublesome: nearly all IT managers we have spoken to who have attempted to deploy EMC diskXtender and emailXtender have not been able to implement these products satisfactorily.

We strongly recommend IT organisations undertake detailed reference checks and visit organisations with similar environments that have successfully implemented the archiving products being considered prior to final product selection. Even where hands-on reference sites suggest that implementation is possible, conduct a rigorous proof of concept that covers all the complexity of the production environment to confirm that the product will work in your environment.

In short, while archiving is simple in principle, there is significant diversity in email and file environments that cause otherwise successful products to fail. The successful archive projects IBRS have seen in Australian are typified by being relatively simple environments, such as a single, centralised Exchange server with fewer than 3,000 email boxes.

Next Steps: If your IT organisation has not implemented archiving:

1. Evaluate the benefit to your organisation from archiving. For further details see “Backup is not Archive!,” IBRS, February 2009.

2. Treat archiving as a high risk project and ensure that the project is properly scoped and managed to mitigate these risks. In particular:

  • Conduct detailed reference checks at organisations with similar environments: same email and file platforms, similar scale and same degree of decentralisation.
  • Undertake an extensive proof of concept.

3. If your organisation has a Business Records and Document Management Framework, leverage that to drive the archiving policies. Otherwise implement archiving as a technical project for storage tiering.

1 Backup is not Archive, IBRS, February 2009.

2 The survey had 51 respondents, with 50% from Private enterprise, 12% from Federal Government 20% from State Government and 18% other. Approximately 25% were Enterprises with more than 3,000 desktops, 36% were midsized with 500 to 3,000 desktops and 39% had fewer than 500 desktops. Most organisations have deployed Exchange (72%), with Lotus Notes (16%) a distant second and Novell GroupWise (4%) in third place.

3 Avoiding SharePoint Implementations That Bite, IBRS, March 2009

Read more on Data protection, backup and archiving