How can IT directors ensure that upgrades go smoothly, given that organisations are increasingly running their IT on a 24x7 basis, reducing the opportunities to upgrade without serious disruption?
"Fundamentally, it requires a good set of processes," said Simon Mingay, vice-president at analyst firm Gartner. "It needs good change, configuration, release and asset management. You need a very controlled environment."
Before the work begins it is vital to carry out a comprehensive analysis of how the upgrade will affect the rest of IT and the business operations it supports.
"You need to set user expectations about planned downtime in the service level agreement," said Mingay. "You must understand the interdependencies [between systems]. What catches people out in upgrades is not realising what will be touched by it."
So upgrades need the same degree of attention as any other commercially sensitive project. "The biggest mistake is to take upgrades lightly," said Tim Eustace, head of operations at financial sector services company Bottomline. "They need a proper project structure with an executive sponsor, a project manager who sees it through to the end, a project team, a budget, and a clear understanding of the objectives," he said.
This approach will be smoothed by following a standard methodology, such as Prince or Itil, said Ollie Ross, head of research at user organisation The Corporate IT Forum. "You need a planning stage to agree scope and objectives, a preparation stage to issue a statement of work and to specify functionality, an execution stage to install the software, test, train and go live, and a post-implementation review to plan the next upgrade," she said.
Because of their effect on business operations, upgrades should be seen as business projects, not IT projects. It also means they should be subject to risk assessment, identifying what problems could arise and what the risks of not upgrading are. Neil Macehiter, research director at analyst firm Ovum, said, "Before an upgrade project gets underway, you need to have an unambiguous business case for doing it. Upgrades are disruptive to people and operations, so they have to be justified. It is a business risk decision."
Timing upgrades is a vital part of the planning process. "Roll-out timescales can be lengthy. It can be difficult to find a window to do an upgrade, and this pushes us towards the end of an application's life," Ross said. The danger then is that in a phased roll-out, the last to be upgraded is already out of support.
Where a roll-out is phased, the new release also has to be simultaneously backwards compatible with the incumbent version. With critical systems, parallel running can provide a safety net, but that requires not just extra processor capacity, but also dual data.
Documentation is also essential - not just of the upgrade, but also the incumbent system. "The trickiest upgrades are where the system is more than two years old," said Eustace. "By that time there will be add-on functionality and customisation, and the original people who implemented them will probably have moved on. For example, if there is a report specifically written for a user, that will have to be redeveloped for the upgraded software."
Whether the roll-out is big bang or phased, testing is as important as for a new system. "Test, test, test," said Owen Williams, head of IT at property company Knight Frank. "Replicate the live environment and test it, upgrade out of hours and test it. And test again after the upgrade."
Integration testing of applications is the biggest time consumer, according to Ross. "Multiple upgrade steps require multiple tests, and when software changes radically it changes the level of regression testing needed," she said. "Beware of applying back-to-back upgrades for a single installation. You may not be able to pinpoint which step is causing a particular problem."
But perhaps the most crucial aspect of upgrading is to have a fall-back position. "You need a predefined escalation process and more expertise on-site or on-call at implementation, plus back-out plans," said Ross.
Ben Booth, IT director at research organisation Mori, agreed that you have to watch the upgrade very carefully and decide whether it has gone successfully enough to run with it. "That can depend on whether it is just one application, when the problem will be bounded, or if it is a desktop operating system which, if the upgrade goes wrong, can bring the organisation to a standstill," he said.
Do not be surprised if even with good planning and project management the upgrade is bumpy. "There can be shoddy packages which are insufficiently tested by suppliers," Booth said. And when patches arrive thick and fast, though best practice may stipulate testing pre-installation, that cannot always be done.
It is also important to keep end-users as much on side as you can if the upgrade hits problems. "I am not sure how much you can warn them about impending failure, but you need them to help you spot glitches and report them back," said Booth.
How well end-users take a bumpy upgrade is not just dependent on service level agreements, it is also about having a good relationship with users to draw on that established goodwill.
A smooth upgrade also relies on the goodwill of the IT team itself. Booth said, "Teamwork is critical. If you have good staff, and they are not exhausted from constant overwork, they can pull out long hours. I have known upgrade teams work through the night."
And that means you also need a good pizza delivery service
Checklist for upgrade success
- Justify the business case
- Do not underestimate the work and criticality of the upgrade
- Run upgrades like a business project, not an IT project
- Stick to a formal methodology, but not so you lose flexibility
- Do a risk assessment on both doing the upgrade and not doing it
- Time the upgrade carefully to balance organisational convenience, business criticality, cost, functional benefit and going out of support for existing versions
- Decide whether to have a "big bang" or phased roll-out
- Do a comprehensive impact analysis on how end-users and other systems will be affected
- Visit reference sites that have completed the upgrade
- Set user expectations for anticipated downtime and upgrade failure scenarios
- Check all documentation for the existing system, especially any customisation
- Do capacity planning for any parallel running before the switch
- Check resources for supporting both versions in a phased roll-out
- Upgrade disaster recovery systems to the new version
- Have a robust roll-back strategy in case of upgrade failure
- Test repeatedly
- Train end-users and IT staff on the new version.