Stormy weather

Take a leaf from the books of three IT directors who have endured serious management and technology problems and successfully...

Take a leaf from the books of three IT directors who have endured serious management and technology problems and successfully resolved them.

They are common to all sizes of organisation across every sector: IT projects that you have wholeheartedly sponsored, that you sincerely believed would work and bring clearly-defined benefits to the organisation, but they have failed because of problems you could not foresee.

US research firm the Standish Group reports that 46% of IT projects are over budget and overdue, and 28% of projects undertaken fail altogether. Another of its studies cites even more grim statistics: only 24% of IT projects undertaken this year by Fortune 500 companies will be completed successfully. The issue has been around for as long as businesses have been using information technology.

You may not be able to avoid the problems, but at least you can recognise them before they become fully-fledged disasters. To that end, Computer Weekly asked three senior IT executives to share their experiences of failed IT projects. Our purpose is not to focus on the failures, but to look at the lessons learned and help other IT directors avoid the same traps.

Key supplier goes bust with no prospect of it being taken over

The first Jean-Marc Fritsch knew about the collapse of hosting company KPNQwest was when he started receiving sales calls from competing suppliers. As head of IT at Manutan, one of Europe's leading distributors of office supplies, Fritsch was busy with a new e-commerce project and had not noticed what was happening to his hosting company.

"I'm responsible for global projects and one-off implementations, and I really didn't pay too much attention to what was going on in the telecoms industry," he confesses. "The first we knew was when KPNQwest announced the network would be shut down within two weeks."

The immediate task was to ensure that the company could continue to operate without the KPNQwest network. All regional IT managers were instructed to order ADSL lines to support vital applications, such as invoicing and order processing. However, the ADSL was a short-term solution - a leased line was the only way for remote offices to access the centralised enterprise resource planning application, which is located in Manutan's French head office.

The next step was to meet the chief executive and other directors to begin the process of approving a new supplier. Ordinarily this process would take about two months, but now it had to be completed in less than a week.

Major hosting companies including Equant and Deutsche Telekom were brought in to pitch for the business. However, the directors were concerned that, with the unstable telecoms market at the time, another supplier could go out of business, leaving Manutan in the same sticky situation.

"We needed a cheap connection fast, but we also didn't want to take the risk of having a single supplier in the future," he says. "We insisted on knowing exactly what partners and network access each potential supplier proposed using, and insisted that it would be written into any contract we signed."

Fritsch also met Vanco, a UK-based managed hosting company. "I was impressed by its chief information officer and also, because it used a virtual network, it could be up and running very quickly," he says. "We took a certain amount of risk because it was a new company and unknown outside the UK, but we had to balance that against the risk of not having a supplier in place before our deadline."

With the chief executive's approval, Manutan signed a contract with Vanco the same day. "We agreed that it would offer us access from a number of specified suppliers, but we opted to own all our hardware and software internally to mitigate the risk," says Fritsch.

Within two hours of the deal being signed, new hardware and software had been ordered and a team of project managers was put together to oversee the migration to the new Vanco network. "We brought in a consultant to work with Vanco to speed up the process. This was a worthwhile investment - in some offices the migration was completed within 48 hours," he says.

The complete migration eventually took 10 weeks, with the company's UK office the last to go online. "In the UK we were dependent on BT installing the last mile access, and it was not particularly reactive," says Fritsch. "Because we were working with a small supplier, it was difficult to press on BT to do things any faster."

Two years after the crisis, Fritsch is careful to keep an eye on the financial health of all of his suppliers and makes a point of reading the technology press. "You have to be aware of the technology choices, even if you are not planning to purchase anything. It is important because you never know when you will have to make a quick decision," he says.

Vital server crashes and the back-up plan does not work

For most IT executives in the UK, a server failure in Africa would not be a major problem. But for Martin Cooper, head of global operations at engineering group Arup, it was a catastrophe.

"We got a call in the early hours from the local IT manager, saying the server had failed, and they could not get it working again," he says.

With the server down, Arup's consultants across Africa had no access to e-mail, the corporate intranet and the design applications they use to help clients with their major building projects including vital infrastructure such as airports and bridges. "We had 120 people working in the region, and once you start to add up their hourly rate, the cost of downtime quickly becomes appalling," Cooper says.

Normally, local IT staff would completely reboot the server and restore any lost data using a back-up. However, it quickly emerged that the African IT department did not have another copy of the server operating system, and the documentation had been mislaid. "They were not even sure when the data had been last backed up, and their back-up had never been tested," says Cooper. "That is the kind of thing you really don't want to hear when you are thousands of miles away dealing with staff whose first language isn't English."

Cooper created a think-tank comprised of IT staff from the UK-based corporate IT department and told it to find a solution to the problem that would not compromise data integrity. Then he left the team to it. "The IT director has a vital role in this kind of situation, but that role isn't to actually solve the problem," he says.

Instead, Cooper's priority was to explain to his bosses what was going on. Within a few hours of the failure, he met the company's directors to explain what had happened, and what was being done to solve the problem. "The best approach is to get in with a pre-emptive strike, before they notice the problem," he says. "Otherwise, they are running around screaming at the IT guys to do their jobs and get the system fixed. The worst thing you can do is leave them out of the loop."

Following the meeting with the directors, Cooper checked in with the IT taskforce who had come up with a plan to restore the African system. The best advice was to boot up the server to a point that enabled the local IT manager to install a second, new operating system. The machine could then be booted up and the configuration settings changed to help staff to identify whether the problem was hardware- or software-related.

The process helped to prevent well meaning IT staff making the situation worse. "Bitter experience has taught me that if a server fails, someone invariably tries to fix it and makes it worse," he says.

The think-tank approach provides an opportunity for experts to share their expertise and, hopefully, avoid damaging action. "The point is someone might have an idea that works for hardware, but not the software, or someone is stronger on disaster recovery but doesn't know the operating system. That is where you really need to make the most of all your internal skills," says Cooper.

The successful installation of the second operating system revealed that the underlying problem was a faulty disc raid controller, which was removed and replaced. The server was back up and running within 48 hours, although Arup let the server limp along for the first day, then performed a full system maintenance check during the night to make sure there were no other problems.

Servers will inevitably fail, but Arup's IT managers are now clearly instructed on how to prepare and test back-ups in case of failure. The global IT department has also reconfigured the network to remove all single points of failure. "Next time a server fails, it is a case of who cares?" says Cooper.

Enterprise-wide software roll-out goes pear-shaped

When the Scottish Agricultural College first began rolling out SAP software in 1997, it hoped to streamline its supply chain management, make authorisation of payments faster and automate purchase orders and invoicing - and make substantial cost savings. Seven years later the project is almost complete, and the college has learned a great deal about implementing enterprise software. "For a while, we thought we had bought the golden chicken that didn't lay any eggs," says Parvase Majeed, business systems director at the college.

SAC's problems began when the college hired consultants to help to roll out SAP's materials management, HR and payroll modules. However, problems quickly became apparent. Majeed says, "We are a public sector organisation and the consultants were not experienced in working in our sector. Also we had not contracted with SAP for the level of support we needed."

With hindsight, the organisation deployed the software too quickly and should have spent more time planning the changes to underlying business processes. For example, SAC was using a number of finance management systems across six campus sites. It would have made more sense to streamline the accounting processes before deploying the new software. However, the college rolled out SAP to all of the sites first.

The team was also hit by a number of problems ahead of the go-live date in July 1998. It was discovered that the internal procurement system used by SAC was not compatible with SAP, and so needed further development. The servers running the SAP system were beginning to buckle under the pressure. And finally, most of the consultants working on the project left, leaving the college with very few staff with SAP skills. "There literally were not enough managers with the skills to keep the systems going," says Majeed.

At this point, Majeed sat down with the remaining IT staff to work out a rescue plan for the project. The team identified all the areas that needed attention, and drew up a recovery plan consisting of 170 application enhancements. Majeed also worked with the college's chief financial officer to renegotiate its support contract with SAP. The college appointed a new team of consultants, from Absoft, to help clean up the system and train the internal IT team to support SAP.

The key lesson that Majeed has learned is that organisations must take the time to analyse processes and existing systems before deploying anything new. "It's rather like an iceberg when you roll out something like this - 90% of the work takes place under water. It could be months before you even take the software out of the wrapper," he says.

Today, the SAC is using its experiences to help other colleges across the UK with SAP implementations. IT managers from the college have worked with universities at Newcastle, Leeds and Warwick on SAP projects.

Long-term lessons from IT problems   
When a supplier goes bust, your first concern is to maintain the service to your end-users by getting a new supplier in as quickly as possible. Despite the pressure to sign up a supplier, insist on knowing who it will be teaming up with and what they will be using. Put this information in the contract and keep an eye on the financial health of all your suppliers. 

Something as mundane as a server failure can highlight major shortfalls in your recovery plans. It is only when the server has to be rebooted, you discover that the back-up plans have been filed in drawers, system software discs lost and data tapes are blank. Issue clear instructions to all IT teams to prepare and test their back-ups and if possible reconfigure the network to remove all single points of failure. 

The lesson most organisations learn when implementing enterprise-wide application software is that any business process changes must be thoroughly planned before deployment.

Read more on Business applications