Distributed systems offer significant business benefits, but they also bring a whole new set of management headaches. When Anfis (Abbey National Finance and Investment Services), the central services firm that supports the different brands within Abbey National's Life Division, introduced client/ server-based systems for the...
1,800 staff at its head office in the mid-1990s, it found it much harder to deliver the quality of service it had come to expect from its mainframe operations.
"We didn't have the visibility we had in the mainframe environment to manage servers, desktops or the network," explains Scott McElney, IT service delivery manager at Anfis. "We would know that we had a problem, but we wouldn't know exactly which server or router was causing that problem. The only way we could find out was to work methodically through our systems until we found it. That meant we were taking far too long to resolve these issues."
These difficulties were particularly frustrating for the IT service delivery team at Anfis because it had a strong background in using systems management tools in its mainframe operations to provide high quality and high availability services to users. What it wanted to be able to do was to take its mainframe automation methodologies and plug them into the client/server environment but the market for systems management tools for client/server applications was still in its infancy. It wasn't until Fraser Cowan, at that time IT service delivery manager and since promoted to head of IT services, saw a presentation on Computer Associates' Unicenter TNG at a conference that Anfis felt it had found a product that could address its needs.
In fact, Unicenter TNG met its needs so well that Anfis felt it didn't need to develop a detailed business case for the project. As Cowan put it, the decision was a "no-brainer". "We simply had to get it in," McElney confirms. "From our service automation background on the mainframe we were very aware of our requirements and the benefits we could get."
Yet, despite the obvious potential benefits of Unicenter TNG, it was still a brave move to commit to it lock, stock and barrel: like the other client/server management tools available at the time, it was still a relatively immature product.
Of course, having realised that more sophisticated management tools for distributed systems were finally becoming available, the Anfis team did carry out an evaluation of other products, including leading rival Tivoli. However, it chose to stick with Unicenter TNG. "We felt Unicenter offered us everything we needed and, at the time, we believed it would be a lot more cost effective than Tivoli," McElney explains. "We also knew that another company in Scotland had recently bought Tivoli and we felt they would have already got the pick of the best Tivoli people. As the only organisation in Scotland at that time with Unicenter TNG, we felt we would be able to get the skillsets we needed more easily."
Given the Anfis team's mainframe systems management experience, it had clear ideas about what it wanted to do with Unicenter TNG - but it also recognised that it had very little knowledge of the product and exactly how to use it to achieve its goals. Because Unicenter TNG had only recently been launched, skills weren't widely available in the market for Anfis to employ on a permanent basis and, in any case, the team wanted to get the project moving quickly so that it could improve the service to business users as soon as possible. It therefore decided to bring in an external consultancy which could provide an injection of knowledge to kick-start the project.
Anfis chose ICL as its partner because it felt it provided by far the best response to Anfis's invitation to tender. "Most of the other responses were rather thin on the ground and talked a lot about themselves but not about Unicenter and we felt they didn't seem to understand Unicenter," McElney says. "ICL had a lot of information about the product and how to set it up. At the time, it appeared to be the only company that had detailed knowledge about the product."
ICL became a part of the Anfis project for about eight months, joining in January 1998. It supplied two technical consultants and a project leader to work alongside the internal team, although the main project management role was assumed by a member of the Anfis team. The first step was for the ICL team to interview staff from Anfis to determine what systems the company had in place and how it might benefit from the various Unicenter TNG components. This allowed ICL to produce an design outlining which Unicenter TNG components should be installed, and how they should be implemented in Anfis's environment to deliver the systems management services Anfis was looking for.
Once this design had been agreed, ICL set up a proof-of-concept lab at Anfis to test how those components would work with the platforms Anfis was running. Graham Bennett, a senior consultant in ICL's IT consultancy unit, who was the technical design manager for the Anfis project, explains that the proof-of-concept lab allowed ICL and Anfis to test whether the product actually delivered the functions in its specification and to check whether it had an adverse impact on system performance.
Another key benefit of the proof-of-concept lab was the opportunity it provided for skills transfer. The Anfis team worked alongside the consultants from ICL, carrying out some of the installation and configuration work under their supervision. On top of that, as the ICL staff installed the Unicenter software in the proof-of-concept lab, they were able to produce and test the build scripts for each component and system. These provided a written record of the work in the lab and could be used by the Anfis staff once they came to roll Unicenter out to their live systems.
"ICL, produced a lot of thorough documentation which helped achieve the necessary skills transfer," McElney says. "In particular, there were some tasks which the ICL team had mostly carried out and where we'd not really been involved, but at least we knew what had been done because we had the documentation."
Once the solution had been thoroughly checked out in the lab, ICL supported Anfis while it piloted it in one area of the business. Working alone, Anfis has since successfully applied Unicenter to a second area of its operations, as well as working hard to develop and customise the solution further.
Throughout the implementation, the Anfis staff worked on developing appropriate systems management processes and procedures for its distributed systems, drawing on the techniques used in its mainframe operations. "Our mainframe procedures were already best practice," McElney says, "and we have found we can translate that best practice to our distributed systems and work in much the same way."
However, the process is not as straightforward as it sounds. McElney points out that a mainframe produces very well-structured messages, but there are no standards for message structures and content - such as object IDs - across the various distributed technologies and distributed systems often generate very long messages where the crucial information is buried among a mountain of other data. A lot of work is needed to make sense of the messages generated by each platform and merge them into a single interface.
This was not the only headache Anfis faced. The very size of its implementation became an issue and one lesson Anfis has brought away from the project - which, McElney says, has been confirmed by other Unicenter users - is that it should not have tried to implement so many modules at once. "Don't plan to do more than six months work at a time," McElney warns, "otherwise it can become like painting the Forth Bridge."
From ICL's perspective, Bennett admits that the major difficulty was the lack of a formalised method for deploying the product, since the Anfis project was one of the first Unicenter TNG installations ICL had undertaken. ICL has since used its experiences of this project to define a more structured approach to implementation.
Technically speaking, the only major headache occurred with the OS/2 system agent from Computer Associates (CA). McElney points out that OS/2 was at that time already becoming obsolete, but he feels the agent offered by CA simply "wasn't as reliable as it should have been and didn't provide the functionality we needed." Anfis ended up integrating a third-party product into Unicenter to handle its OS/2 needs. However, McElney believes the experience shows that, even with a comprehensive package like Unicenter TNG, you need to be prepared to use third-party solutions in some areas, because it is likely that some of your framework supplier's modules will not be up to scratch. On the plus side, if you have picked one of the leading frameworks, most of the third-party systems tools suppliers will have already incorporated the necessary integration in their products.
This breadth of coverage is one of the key benefits of a systems management tool like Unicenter. Another is that, once the company's policies have been embedded in it, the software ensures they are enforced and that the correct procedures are followed.
McElney admits that it has sometimes been hard to get the strict process discipline which is commonplace in the mainframe environment accepted into the less structured working practices that have grown up in the distributed systems world. "One of the challenges we still have is to leverage a lot of existing system management policies and procedures into actual practice," he says. "Not everyone is on board with implementing what we want to achieve in systems management terms into their policies and practices."
This cultural change extends down to the day-to-day working practices of support staff. "Mainframe operators are used to console operations where all messages come to a single screen," he explains, "but it took some time to get across to support staff who are used to looking after distributed systems and networks, that all messages would be sent to their consoles.
"We had to convince them that they should be looking at their consoles constantly to find out what's happening - not just for notification of problems but also for proactive information they can use to prevent problems arising in the first place - and that they need to keep their pagers with them at all times so that they can receive alerts from the system while they're away from their desks. We also needed to get them to understand that they could use the Unicenter console to diagnose and resolve problems without necessarily having to visit the piece of kit which was at fault."
Anfis was also quick to recognise that the implementation project wasn't over once the software was installed. "The stage that is the easiest to avoid doing but which provides the greatest business benefits is the ongoing exploitation of the product once it's been implemented," ICL's Bennett says. "A lot of people see the implementation as the goal, but you won't get huge benefits unless you continue to tailor the systems management solution, because the out-of-the-box product is quite limited." For instance, you should analyse data about systems issues occurring in the previous month and determine whether there were any indicators to suggest that a systems crash was about to take place. You should then look at whether your systems management solution can be reconfigured to detect those warning signs and alert you that a crash might be about to happen.
McElney agrees, "It's really a continuous customisation process. We are always looking for areas where we can improve our Unicenter implementation and are looking for third-party products to plug gaps where we can't find a Unicenter product that's good enough or works in the way we want."
For instance, Anfis is now working to develop support for layered automation. The system was originally set up, McElney explains, to page the relevant engineer when particular system messages were received. Now, when a message requiring action is received, Unicenter can automatically initiate a predefined action. If this resolves the problem, the message is cleared. Only if the problem persists will the engineer be notified.
"We can't automate everything, but where we can, we will," McElney explains. "For example, we had one system which sent a message to Unicenter that sometimes meant the system was down - but not always. If the system was down, it had to be rebooted and, in the meantime, Unicenter was automatically firing off an e-mail to the business saying the system had failed. Before we automated the process, it could take 40 to 50 minutes to resolve the problem. Now Unicenter pages the engineer, who checks the status of the system. If the server is down, he can press a single button which reboots the server and sends a message to business users letting them know that the server will be down for 10 minutes while it reboots. If the system is OK, he can press another button which clears the message and e-mails users to let them know the system is OK. We've been able to cut downtime to just 10 minutes and keep the business better informed."
Anfis has also recently installed the Web management option and is now looking to bring its mainframe and more elements of its network under the Unicenter umbrella. "We still have limited visibility of our network and we would like to improve that," McElney admits. "Also, CA is coming up with new modules all the time that we might implement."
In fact, McElney believes Anfis will be developing its use of Unicenter TNG for many years to come, as the support team develops new practices for existing systems and learns to deal with new products as they are added to the infrastructure. However, one area where McElney feels Anfis has gone as far as it can with Unicenter TNG is with the anti-virus option. "It's a good product, which gives us confidence in its detection abilities, and we have worked on it constantly until we feel we have achieved closure apart from various upgrades," he explains. "For instance, we have set the server up to collect the latest virus signature files automatically at scheduled times and to fetch new signature files in reaction to e-mail alerts from CA. That meant that, for example, we had the update for the 'I Love You' virus loaded onto our machines by the afternoon of the day it hit."
It is this speed and level of control which lies at the heart of the benefits Unicenter TNG delivers to Anfis, although McElney points out that it's very hard to quantify the benefits from a financial perspective. He suggests it's more about avoiding disruption and downtime to the business, while minimising the number of support staff needed or being able to deploy them on tasks that add value to the business, as it is about making savings to the existing budget.
"Unicenter is our eyes and ears and we can't live without it," he says. "If we took it out, we would rapidly be in a mess, with more downtime and more extended downtime. That would affect the service we could offer to our customers, so it would soon have an impact on the company's bottom line. We have to ask people to remember what it was like before we got Unicenter - and we were running less complex systems then."
At a glance
Anfis is a central service company which provides resourcing and facilities to support the different brands that make up Abbey National's Life Division portfolio, including Scottish Mutual Assurance, Abbey National Life and Pegasus. By sharing services such as actuarial and technical advice, human resources, finance, IT systems and customer services the various brands in the group benefit from increased efficiency and operational costs which are amongst the lowest in the life assurance industry. Anfis now employs about 1,800 staff at its offices in Glasgow and is responsible for managing about £20bn of client funds.
Anfis had installed a number of client/server systems to meet business users' needs, but was finding it difficult to deliver the kind of high quality of service it was achieving with its mainframe operations. It needed a management solution that would give it greater visibility of the elements in its distributed systems, so that it could identify the cause of faults more quickly and, if possible, solve them remotely.
Computer Associates' Unicenter TNG offered a suite of modules which could: present information about the various elements in Anfis's distributed environment through a single console view; allow support staff to diagnose faults and take appropriate actions remotely; and control the actions they could take, to ensure Anfis's processes and procedures were followed.
Secrets of systems management success
The Computer Weekly/Buy IT case studies offer an in-depth analysis of a successful IT project, with expert comment from a panel. BuyIT was launched in 1995 by the DTI and an alliance of top industry bodies. BuyIT has selected best practice examples on a range of projects. Each case study is scrutinised by the BuyIT team of experts who make their recommendations and comments. The BuyIT Computer Weekly Best Practice Series is endorsed by Fit for the Future, a CBI-led, government-backed campaign to get business learning from business.
What the BuyIT experts say...
Anfis is typical of a lot of companies which moved from mainframe to distributed systems and found that maintaining performance standards was more time consuming and expensive than they had expected.
The central services company for Abbey National's Life Division found the software to help with this problem but then realised it lacked the skills and experience to tailor and implement it over such a complicated IT network.
It was fortunate that Anfis found in ICL a company that could combine experience of designing and implementing systems of this kind with a good understanding of the financial sector. Had Anfis not been so lucky, this would be a tale of woe, not a success story.
We liked the approach ICL took in training Anfis' own employees to recognise and deal effectively with problems rather than using this as an opportunity to create further work for itself.
Anfis now has a system which consistently saves them time and money. It is managing its network at a much greater level of detail than before and its staff are sufficiently familiar with the system to be able to handle future roll outs themselves.
I would be interested to hear from other companies which have not had such good experience of the transition from mainframe to distributed systems.
Chairman of the IT Faculty, Institute of Chartered Accountants in England & Wales
One of the key considerations I would like to have seen included in the Anfis story is security. Companies have found they needed to bring about a sea change in end-users' attitudes and approach to security when moving to distributed computing.
Security requirements arise from three sources:
The assets to be protected may take many forms. Principally for information security, you will be concerned with information assets. These may be supported by documents, software, physical assets, people and services. In a distributed processing environment these factors have to be considered at the level of individual user departments and sections within the organisation. Everyone has to understand that risk analysis is more concerned with the cost of the loss of the use of the asset than with the value of the asset itself.
One of the principal assets of any organisation is its image and reputation. This may suffer when its information assets are lost or corrupted. This, in turn, may occur when harm befalls any of the other types of asset.
When processing is distributed to the end-users, the reputation of the company depends on the end-users themselves.
Six steps to systems management heaven
ICL's method for implementing systems management solutions involves the following six steps:
Do you have any comments on this case study or any examples of best practice of your own? Please send comments and input to email@example.com
What did Anfis implement?