Watering systems

Pouring an enterprise integration tool onto Anglian Water's mix of legacy and proprietary systems created the code it needed in less time and with fewer errors than expensive human programmers

Pouring an enterprise integration tool onto Anglian Water's mix of legacy and proprietary systems created the code it needed in less time and with fewer errors than expensive human programmers. 

I have gathered a garland of other men's flowers, and nothing but the thread that binds them is my own," wrote Montaigne. The same might as well be said for the IT infrastructure of any major organisation.

Each individual application might be flowering beautifully in its own right, but if you want corporate IT to be worth a bouquet rather than a brickbat in the business, there must be a thread binding applications together. Islands of information are not an acceptable option. Enterprise application integration (EAI) is inevitable.

But if building causeways between the islands of information is tough - and it is - then building them under severe time pressure is even tougher. That was the problem facing Anglian Water Services in the run up to the millennium. As part of the water company's Y2K-compliance strategy, it was moving away from its existing collection of heavily-customised, mainframe and mini-based legacy packages to new Y2K-compliant client-server packages. Since these new packages would be running critical, heavy-duty production systems (see box p36) it was essential they could pass data along the thread binding them.

The company's existing systems all did so with hand-coded custom interfaces. Even well-integrated modules within a package present problems when transmitting data between packages. Not only does each package usually duplicate core elements such as the name-and-address file, but there will inevitably be differences in business rules and data definitions that complicate data exchange. There is no get-out clause from the necessity of doing EAI.

"A loop of data runs around our big four applications, with data feeds in and out. It's all highly complex," says Ian Montgomery, senior systems analyst at Anglian Water's outsourcing company Computer Sciences.

But although Anglian Water was prepared to start a major hand-coded data conversion and migration exercise to build new interfaces between the mix of new packages and renovated legacy systems, when it did the project sizing, the company realised what it was up against.

"We realised we were looking at 20 data conversions and a dozen data interfaces," recalls Montgomery. "Manual coding was going to be far too long-winded."

With a go-live for compliant systems of October 1998, to give the company a six-month safety margin before the start of the financial year in April 1999, the scarcest resource was not money, but time. The next scarcest was equally critical - people.

The interfaces for the new infrastructure would need to be written in Cobol. With everyone working on Y2K projects, Cobollers were not thick on the ground. Nor were programmers capable of writing code in SAP's proprietary Abap language. This would be needed to write interfaces into Anglian Water's R/3 package being implemented as part of the £30m Y2K programme.

"By November 1997, we had identified the number of interfaces and conversions we would need and began looking at the scale and complexity of the problem," says Montgomery. "We would have needed about 20 Cobol and Abap programmers and neither were cheap nor readily available."

Moreover, the recruitment drive itself would have taken time. "We were facing a major exercise to bring on board that size of team," he says - even if such a team could be found.

So, if hand-coding in Cobol and Abap was out, what could Anglian Water do? Choose other languages for the interfaces where skills were not under such pressure?

"We thought about writing the interfaces in Visual Basic or C," says Montgomery. However, these languages were not suited to the task. Cobol code is much easier to maintain and the interfaces had to run on a variety of platforms - something else Cobol is good at.

So it was back to Cobol and Abap, but not by hand. Instead, Anglian Water started to investigate EAI tools specialising in data migration and conversion which could automate production of interface code. But how to choose between the shortlist of three products?

"Hewlett-Packard's data migration product only dealt with SAP environments," says Montgomery, which would leave non-SAP interfaces to be hand-coded.

Conversely, Constellar, at the time, could not generate Abap, he says.

Anglian Water considered using both packages together to complement each other. But two varieties of glue would mean two products to learn, license and support.

The outcome was opting for ETI's Extract extraction and transformation product. Since it produced Cobol source code Extract could be ditched after the migration project was complete, leaving the generated - and well-documented - Cobol to be maintained by Anglian Water.

But Extract also had SAP registration. "That meant SAP had looked at the product and said it generated good Abap that would not corrupt the R/3 database," says Montgomery.

Data integrity

To ensure data integrity, R/3 does not allow code to be loaded directly into its underlying database. It has to go through the SAP screens to check it is not breaking SAP rules, such as missing an index table or a relation which would stop the system working properly, and be difficult to detect as a cause of malfunction.

"SAP registration was a real bonus for ETI," says Montgomery.

The only limitation ETI had was not being able to generate interfaces directly into Anglian Water's Jobs and Asset Management (Jam) package. However, Jam's reseller, AMT - the package itself being written by an Australian mining company - wrote bespoke code that would either load data from flat files into the Jam database or extract data from the Jam database into flat files where it could be picked and fed to other applications using ETI-generated programs.

The outcome was, "we decided on ETI in January 1998", says Montgomery.

Although it is not cheap - "you get the package with a number of modules and some consultation included", says Montgomery - Anglian Water was offsetting this against the cost of Abap and Cobol programmers.

Additional benefits from using ETI included reducing the testing burden. Having purchased it for SAP, it was then available for use on the Jam project with only maintenance costs to pay.

ETI was delivered in March 1998 and an ETI consultant set the product up and got it going. Montgomery went on a two-week master-user course so he could be the Anglian Water ETI expert.

"It was a steep learning curve," he recalls. "My background is NT and Bull, so I was also learning Unix [the Extract platform Anglian Water used]. It was a challenge!"

However, if becoming Mr ETI at Anglian Water was tough, learning to use the tool to produce interfaces was much easier.

"The two-day course was enough to write conversions," says Montgomery.

Because of that, and since IT people were scarce and expensive, Anglian Water took the bold step of assigning non-IT people to the data conversion project. Six staff from Anglian Water's business units were brought in and trained as conversion specialists.

"It was as fast to get business users involved as it was to explain to a coder what was needed," says Montgomery.

The business users also knew exactly what data they needed to have available in each system, and how it was structured. Because they knew the business rules applications followed, they could see swiftly if they were breached by the conversion code.

"They'd spot the problem earlier," says Montgomery. "If a look-up table has to translate values, they understand what they should be. To a programmer, one value might be as valid as any other."

Knowing the system's business functionality was, he says, a real benefit in the way the user conversion team worked to produce the conversion specification.

One of the business analysts brought to the conversion team says, "The first task was to define the selection criteria: what data would be migrated and which fields in the data structure would be retained? Finally, we had to set the mappings from old values to new. A few were straight mappings but most were filters, setting out conditions of translation."

"In data conversion, the biggest problem is identifying data that needs converting," warns Montgomery. "You can't apply a tool to find the data."

Preparing the conversion specification was the bulk of the effort, therefore, then ETI took over and generated the required code.

Code cutting itself was very swift. "With ETI you can write a simple interface in as little as 15 minutes from scratch," says Montgomery.

That interface can then get to work on the data. "Once you knew it would convert 100 records properly, you knew it would convert 100,000, and very fast," says the analyst.

Extract uses a high-level scripting language to make it easier for non-technical users to write conversions. The generated code quality is, says Montgomery, "excellent and well-documented".

Moreover, he adds, because it is Cobol source, it will compile and run at the speed of the platform.

Testing was automated as much as possible. Although all programs underwent a four-phase testing cycle - unit, integration, end-to-end and user acceptance - the quality of generated code meant it usually worked immediately when tested end-to-end - ie pulling data through a sequence of programs - even without prior unit testing, says Montgomery.

In all, Extract generated 450,000 lines of code in three months, a quarter of the time required with traditional methods.

"As things kept changing, and we had to make changes to conversion programs, it was easy to regenerate and retest programs with Extract. It is the difference between 10 minutes and an entire morning's work."

The most difficult of all the 30 interfaces and conversion programs to tackle was the interface to Anglian's Regional Income Management System, the GCOS-based billing system. It comprised, says Montgomery, "40 programs, about 80,000 lines of code in total, with seven very complex key programs. A good Cobol programmer works at the rate of 1.2 mistakes per thousand lines of code. That means if we hadn't used Extract, we'd be repairing at least 90 coding errors before testing."

Like all projects running against a very tight deadline, Anglian Water's data conversion project was bowled a googly. In the middle of the project, the company decided to get the performance it needed out of R/3, it was going to have to run it on Digital Equipment Alpha Risc servers, not the Intel servers implemented.

Although R/3 stayed on NT and SQL Server - Alpha runs NT as well as VMS - it meant a rapid rewrite of the data conversion interfaces for a Risc platform. It wasn't a big deal however, says Montgomery.

"Because we had Cobol source code, we could drop it onto the Alpha box and use the Risc compiler to recompile it, then run the last two test cycles again," he says.

With no further googlies, the conversion was finished ahead of time.

"Compared with traditional methods, our use of ETI Extract cut the entire development and testing cycle down by a factor of 10:1 - a 90% reduction," says Montgomery. "We were so far ahead, the migration to SAP could have gone live earlier."

Because the conversion code is Cobol, Anglian could, if it wanted, say goodbye to the ETI product. But it is still using it as an integration tool in the post-millennium aftermath.

Like all companies, Anglian Water is under intense competitive pressure to make business changes rapidly in response to both its market and, for a water company, its regulator's requirements. It cannot afford for IT to hold it up, especially because of data integration issues.

"From January to March this year, Anglian made a major change to its business so there were a number of changes to the raw data," says Montgomery. "We've used ETI to effect those changes. We've found we can use it as a rapid application development tool.

"Like many companies, over the years we've purchased best-of-breed packages along with hardware platforms that offered the best performance. As a result, we've ended up with a mix of packages and platforms. The ability to generate source code means you are not necessarily dependent on a platform - Cobol will run on post platforms," reminds Montgomery.

Data conversion and migration might seem a tedious, technical problem. But as Montgomery points out, "data migration is crucial. Without data you haven't got a system."

At a glance

The Organisation

Anglian Water Services is the largest UK provider of water and waste water treatment by geography, serving about 5.8 million customers across an area of 27,500 sq km, and a further 5.8 million worldwide. The company outsourced its IT to Computer Sciences Corporation (CSC)at the end of 1995, and has been moving from a mainframe/terminal-based environment to a client-server, PC/networked environment.

The Challenge

Moving away from mainframe and mini-based legacy systems to new client-server packages presented Anglian Water with a major data conversion headache.

The Solution

The use of an enterprise application integration tool not only enabled rapid data migration but enabled non-IT end-users (who understood the business functionality of the systems) to undertake conversion work.

The Computer Weekly/Buy IT case studies offer an in-depth analysis of a successful IT project, with expert comment from a panel. BuyIT was launched in 1995 by the DTI and an alliance of top industry bodies. BuyIT has selected best practice examples on a range of projects. Each case study is scrutinised by the BuyIT team of experts who make their recommendations and comments. The BuyIT Computer Weekly Best Practice Series is endorsed by Fit for the Future, a CBI-led, government-backed campaign to get business learning from business.

Anglian Water's production systems

The four main production systems are:

  • SAP's R/3, running on Microsoft's NT and SQL/Server, on a Digital Equipment Alpha platform, for finance and procurement. It replaces a highly customised Vax-based package for which support ceased. By October 1998 it had to be populated with previous six months' worth of financial information from non-Y2K compliant Vax systems and additional data migrated from payroll, time and job costing. Links to all financial feeder systems would continue to pass data into R/3, including regional income and billing systems also being built at the same time
  • Unix-based jobs and asset management and job scheduling system
  • New Billing System (NBS) is part of a bespoke system, Regional income management system written in the Pacbase fourth generation language to run on the GCOS 8 mainframe and Codasyl database. NBS runs on Unix and Oracle
  • Operational Common Database (addressing module), a bespoke legacy system running under Digital Equipment VMS and RDB on a Vax platform, it stores geographical information such as asset location.

What the BuyIT experts say

Sam Jacobs
Director of alliances (Constellar), DataMirror, and chairman

Anglian Water's experiences demonstrate that while Y2K may have been the initial trigger to implement an enterprise application integration (EAI) strategy, the benefits to the business are still felt in terms of improving ability to operate in a highly competitive marketplace and ease the introduction of new software packages. On the basis of its specific application, Constellar Hub was not appropriate, but the implementation does demonstrate wide ranging benefits of EAI.

Implementing EAI means a company can bring together all data it holds on customers and analyse this to provide a better, more targeted service that meets customer needs. This information is also vital to analyse operational performance and meet stringent regulatory requirements of the utility market.

With the rise of new front-office applications such as e-business and customer relationship management, it is necessary to quickly integrate new packages into an IT infrastructure - particularly as project timescales are now measured in weeks rather than months. Integration tools aid this process without compromising existing investment in legacy systems and with minimal disruption to the business.

Therefore, as Anglian Water has shown, while it may appear daunting at first, enterprise application delivers benefits not only in the short term, but in the long term too.

Ken Deeks
Managing diretor, Kaizo, and chairman
BuyIT Communications Working Group

It's true that by introducing an enterprise application integration (EAI) solution at the same time as migration to a client-server architecture, Anglian Water made a sound technology decision. This is the company's answer to the islands of information challenge and is more than a technology implementation success story.

EAI is about reducing costs, increasing operational efficiencies, improving return on IT investment and speeding times-to-market to achieve long-term business benefits. By introducing an EAI solution as part of its Y2K-compliance strategy, Anglian Water has not only put in place a framework for keeping pace with future technology, it has also taken a strategic step to achieving and maintaining long-term customer value.

Management consultancy PricewaterhouseCoopers states "Companies today are increasingly judged by knowledge assets and use of them to create value." By linking the disparate systems and platforms, Anglian Water's EAI solution has created a communications blueprint that will deliver customer value today and ramp up its ability to respond and adapt to future change.

By creating an invisible bridge between the company's front- and back-office applications, Anglian Water's EAI solution introduces an effective knowledge management infrastructure that will leverage internal relationships and company know-how fully for the first time and play a crucial role in enabling the company to deliver on its brand promise to customers.

Alistair Fulton
Chairman, BuyIT, and president, CSSA

Integration has always been a key worry for IT management. As more systems are brought in to deal with business needs, the spaghetti of custom-built interfaces becomes increasingly complex. As Anglian Water discovered, by thetime mainframes are replaced by user department-owned client-server systems, the potential integration problem can be huge.

Now that e-commerce is changing the way we do business with customers and suppliers, being able to move data between applications seamlessly (and in real time) is essential to business performance.

We thought it would be useful to illustrate one aspect of the issue with this case study, focusing on conversion of data at Anglian Water.

The case study also highlights another important issue: the use of an automation tool to carry out data conversion and code production.

Increasingly, code-cutting will be left to computers and business-rule engines will be introduced. This not only significantly speeds up the process, it introduces the possibility of involving non-IT people.

For the first time, business end-users can bring their understanding of business processes directly to the programming of systems needed.

Read more on Integration software and middleware