One2One automates billing system

One2One automated its billing systems in just one month, and realised a return on its investment within a week. Julia Vowler...

One2One automated its billing systems in just one month, and realised a return on its investment within a week. Julia Vowler reports.

With the corporate world fast dividing into two camps - the new economy stockmarket darlings in technology, media and telecommunications, and the boring old economy has-beens, that is, everything else - it must be comforting to be firmly entrenched in the former.

But being a high-growth mobile telecoms company like One2One can be as scary as it is exhilarating - like riding a motorbike along a highwire. If the engine stalls it can be bad news, which is why the work of IT in keeping the One2One's engine purring along is so critical. This is especially so when it comes to a telecoms company's most critical operation after keeping the lines open, keeping the money coming in and keeping the customers happy. One2One's billing and customer care system is right at the heart of the IT function.

"It's very, very important," emphasises Ray Murphy, IS infrastructure manager at One2One.

It isn't just an all-or-nothing question of keeping the system up-and-running. The criticality of the system is felt the moment it even hiccups - any knocking or misfires in the engine and the motorbike starts to judder on the highwire. The problem for One2One was that the opportunity for system hiccups was considerable.

One2One uses an AS400-based package, Virtuoso, bought in from US supplier, Alltel Corporation and heavily customised to convert it from a billing system originally designed for a fixed-line telecoms operator to One2One's mobile telecoms business. Virtuoso gets a lot of thrashing. It runs 24/7 online all day to One2One's customer call centres for customer enquiries. Because, unlike traditional utilities, One2One does not do quarterly billing runs, Virtuoso has to use the overnight batch window to pump out a new off-line billing run every 48 hours. This is due to the fact that the company scrolls through its customer base on a rolling schedule to tot up the bills for its customers' 160 million calls a week within the UK and across 133 roaming networks in 75 countries. "There are 14 billing cycles per month," says Murphy.


Virtuoso was also thrashing the AS/400s it was predominantly running on (with some reaching out to mainframe and Unix platforms at One2One).

"We've broken IBM's transaction processing records for AS/400s," points out Murphy. "We even had to send dumps to Rochester because they didn't believe us!"

With the explosive growth in mobile phones, One2One's increased volume of business has translated directly into increased pressure on the billing system. "I've been here eight years and business volumes have increased 100% year-on-year," says Murphy.

"Three years ago we were doing 150 transactions an hour," says Murphy. "Now it's over a million transactions an hour."

Moreover, the billing run is complex and interactive, as well as long. At midnight, says Murphy, the call rater for the accounts being billed that cycle is closed, and checked to ensure that all calls are through. Then further checks are run in case that account has been subject to a change in service conditions since the last bill, or any new discounts or promotions are applicable, and calls itemised. Interfaces to payment agencies like Bacs have to be triggered, and before each bill can be sent to the printer to generate an image for posting, an account audit is carried out to ensure that the sum totalled is what was expected. If it isn't they have to go back and find out why. After that the bills can be printed and posted out the following day, with the customer care records updated to record that the customer has been billed and when payment is expected, in order to be prepared for any customer calls coming into the call centre to query the bill. "We do this every two days, 365 days a year," says Murphy.

Nightly run

Each nightly run consists of a considerable number of individual jobs, each of which has to be scheduled into the run within what is, inevitably, a very tight batch window. This was where trouble could start. All the jobs were scheduled by human operators.

"We used totally manual scheduling," says Murphy.

A shift worker would start the bill schedule at 10.30 pm, and then spend the night sitting still and running job after job - bringing it up, releasing it, running it, checking the returns and then going on to the next one.

"There were hundreds of jobs, one after another, and we expected them to be able to move on to the next job within seconds," says Murphy. He freely admits that the job was "demotivating".

Worse, it was all too prone to error, caused, for example, by operators releasing jobs in the wrong order.

With the batch window closing all the time, "the biggest worry was errors causing a batch overrun," says Murphy, all too aware that, come the end of the batch window, "a thousand people are sitting at their screens, and if the application isn't upÉ".

Disgruntled customers unable to get answers to their queries from call centre staff are not what One2One wants. Nor does it want customers to have incorrect bills to be disgruntled about in the first place.

"Our next biggest worry was the wrong billing information going in, causing errors in the bill," says Murphy.

With a tight batch window, recovering and correcting any errors was a problem. Together with the danger of missed deadlines, this had significant potential for generating a highly undesirable knock-on effect - some with an indirect financial penalty, such as disgruntled customers taking their custom elsewhere, and some more immediately felt. If running late, or having to delay while correcting a bill, then the printing slot could be missed, resulting in the bill being sent out several days late. Expected revenue would then come in late, or call centre operation costs would increase as customer queries about late bills increased.

"All of this can cause service degradation, which means we are not meeting our customers' needs and requirements," says Murphy.

Originally written not just for fixed-line telecoms billing but for IBM's System 36, when the software migrated to the AS/400 it was "not rewritten, just recompiled" says Murphy.

This meant that what operators saw was not what was going on under the covers. The software also had to capture and feed information to other jobs running on mainframe and Unix platforms, as well as the AS/400.

Murphy knew he had to take the potential for operator error out of the billing run.

"I'd say 98% of the problems were human error, and 2% were things like the tape media, hardware or operating system."

Automating the job that human operators did was the answer, but knowing that he needed an automated workload scheduler did not make one it happen.

"I'd been looking for a scheduler for three years," says Murphy. The likes of Computer Associates and Boole and Babbage had been tried out, but none had proved satisfactory.

In the end, Murphy opted for a workload management system, ESP, from Cybermation.

"By the time we got to Cybermation - with the same promises as every other vendor that they could deliver a solution - we weren't at all receptive," says Murphy.

So much so that One2One squarely put the onus on Cybermation to prove its software in the field of battle.

"We challenged them by saying that if they could deliver automation for Virtuoso within a month, the contract was theirs," says Murphy.


He's satisfied the system came up trumps.

"ESP was the only product that could automate our customer care and billing system," says Murphy.

What kind of impact has automation had?

In a word - immediate.

"I got my return in the first week," says Murphy. "Manual operator intervention dropped noticeably, and our batch times immediately became more efficient."

The billing run is carried out on two of One2One's own AS/400 machines, but they are housed and operated by Cap Gemini. As soon as ESP was in action the regular production reviews produced by Cap Gemini for its customer showed the immediate reduction in operator-generated error.

"Because we've been able to get rid of human intervention, operator errors have been almost eliminated," he says.

Instead of at least 30 operator errors a month, there have been only one or two a month since ESP went in. Moreover, the length and complexity of the billing run meant that, manually, it would take up to three hours to set up a schedule. That's gone now - only any ad hoc jobs have to be manually scheduled. Even though billing cycles can still take five hours to run, that window has absorbed a huge increase in actual numbers of customer bills.

"We've reached three million customers and now manage our automation with less staff - a lot less," says Murphy.

Murphy also has the comfort of knowing he is not living hand-to-mouth with the batch runs.

"We now have two to three months' schedules set up," he says.

If those need to be changed, ESP's graphical front end makes it easy for operators and support staff to show up any dependencies which changing the schedule might deleteriously impact.

"If we put the wrong job in, it will flash up," says Murphy.


ESP also provides on-screen monitoring of the progress of the run. The manager can see how the run is going, how long it will be before specific points in the schedule will be reached, and when it will complete.

"With ESP we can easily predict the future. We know what will be happening and when, and this allows other parts of the business to plan better," says Murphy. "We have reduced the effort in managing what people have to do. For example, the 100-page batch schedule which we used to produce and which Cap Gemini would work with, is now five pages, and these only cover additional requirements.

"The number of staff required by Cap Gemini to provide the facilities management requirement has been reduced with the introduction of ESP. We now receive a higher quality service that's manageable and predictable. The reduction in staff is due to ESP's high efficiency, and is self-evident. Cap Gemini is certainly using a lot less people on our account which proves that ESP is a very good investment."

As far as Murphy himself is concerned, automating the billing run scheduling has had a direct and immediate impact on his life.

Not only has Virtuoso stopped being one of the first things to appear in the company's nightly bulletin for senior managers, but it has also improved Murphy's night-times.

"I don't get phone calls at two in the morning," he says. "That's my return on investment - and my wife's!"

For all the new-found peace of mind and uninterrupted sleep that Murphy now enjoys thanks to automated scheduling, he can't expect a totally quiet life. Ironically, One2One is already on the point of migrating off Virtuoso and moving off the AS/400 on to a Hewlett Packard Unix platform, which will run a new billing package from Israeli supplier, Amdocs.

"Amdocs calls it WFOP - wireless family of products - but we're calling it Excaliber," says Murphy. Perhaps WFOP sounds catchier in Hebrew. "The major reason for the migration is time to market," says Murphy.

As the mobile telcoms market grows, mutates and perpetually changes - One2One itself got sold off by its joint owners, Cable and Wireless and US West to Deutsche Telekomm in July l999 - the IT underpinning operations and business strategy has to be ever-more flexible and responsive.

"Virtuoso didn't lend itself so readily to change," says Murphy, "as it was not designed for wireless telecoms, so it was harder to manipulate."

Because Excaliber comes with its own workload manager, One2One will probably start out using it in preference to a third party product, on the grounds that migrating to a new billing system is enough without applying a third party job scheduler as well. But once Excaliber is bedded in - cut-over should happen in the middle of this year, and the package has been in customisation since last October - One2One will see if a third party product is preferable.

"We'll benchmark a variety of products - we don't want rocket science or bleeding edge, just something that is stable, but best of breed," says Murphy.

As for ESP and Virtuoso, they will be deployed by One2One as the core platform for the company's new joint venture with Virgin Mobile for its prepayment mobile phone system. "And any other partnerships we form," says Murphy. "They've been tried and tested."

The deployment of the ESP workload scheduler on One2One's billing operation is a classic example of the use of software from the less glamourous end of IT. However, it has proved a key enabler when it comes to ensuring that IT delivers quantifiable and actual financial benefit to the business using it.

And all without even a sniff of the ubiquitous e-word - just a lot of old-fashioned but good sense day-to-day best practice.

The technology

  • One 2 One's environment features nine large AS/400s - the most leading-edge models that IBM can offer

  • Its customer care and billing systems run on two of these, with the others providing pre-production, development and the financial systems

  • The mainframe is on the tail end and is not greatly significant. Looking ahead, the strategic plan is firmly UNIX-based

  • For its interface into ESP Workload Manager for defining, monitoring and controlling workload it has standardised on the graphical user interface, ESP Workstation

    Best practice tips Peter Duschinsky, secretary, BuyIT, explains what One2One did right. It:

  • Worked closely with its system supplier Cybermation, treating them more as a partner than a supplier. That means the supplier understood their business, there was a proactive exchange of ideas and One2One got the best out of their investment

  • Recognised that new does not necessarily mean best in a mission-critical environment - it has to be robust and scalable

  • Ray is said to be "relentless" in getting the best solution. He followed best practice by benchmarking products to establish best in breed before choosing to go with one that meets the immediate and future business requirements of his organisation

    And what did it achieve?

  • Less boring and tedious jobs for operators by eliminating mundane tasks, making it easier to recruit and retain skilled people and dramatically reduced operator errors

  • Bigger online "window" because completion of the bill run can now be predicted more accurately - and achieved

  • The customer care and billing system can now be managed by one person - which means more staff time freed up for development rather than just making the application run, and more efficient use of outsourcing staff

  • Ability to handle huge growth rates in customers and billing volumes

  • Move to management by exception

  • Assured and timely access to the system online by call centre staff and better, faster and easier to read information for support staff

  • No more customer complaints about errors and late bills

  • IT doesn't get mentioned in management reports for the wrong reasons!

    What the Buy IT experts say BuyIT Best Practice Group

    Alistair Fulton, chairman

    After the excitement of e-billing in our last BuyIT feature, we stay on billing as a topic. But instead of focusing on how a company prepares for potential growth in e-billing, the One2One case study illustrates the importance of adopting best practice in dealing with a high-growth operation.

    Like many organisations, One2One's billing service started with technology which developed for another purpose and was rather inflexible to change and growth. Finding ways to update and cope with the rapid upscaling of the business demanded a rigorous selection approach, an innovative solution and excellent project management.

    The scale of this particular requirement was daunting and outpaced most of the contenders, but Cybermation's ESP product "did the business" and astonishingly, within a month! We are becoming blas‚ about Internet-based projects being brought in within weeks, in contrast to the months and years that projects used to take. But it is still worth a headline (and a case study) when it happens in an operational back office environment.

    Now One2One are moving on to take advantage of more flexible technology, but using the same high standards of management. Things cannot be allowed to go wrong - as customers, our expectations see to that.

    BSI panel on service management

    Jenny Dugmore, Chairwoman

    There is more to customer service than saying "have a nice day". Most suppliers find that customers only stay satisfied if their expectations are exceeded. This requires continual improvements in the service, which requires a sound basis for management.

    There are particularly difficult challenges for "highwire" firms such as One2One. Customer service matters just as much when your customer base is a thousand customers as when you have 1.5 million customers, but One2One's growth rate of 100%, year-on-year, presents a daunting challenge. As One2One is all too aware, getting the bills right in the first place is fundamental, and however good the new system, the change had to happen without the service being disrupted. For One2One, moving this fast and this far must have taken a great deal of careful planning, and courage. The pay-back has been reported as immense, but the change could have gone badly wrong, turning a difficult situation into a wholesale disaster.

    The British Standards Institute has also recognised how important this is to the service industry and has produced a code covering industry best practices on managing support services. The code will soon be joined by a series of workbooks for self-assessment, and a specification for formal audits.

    Details are available

    MD of IT communications consultancy, Kaizo

    Ken Deeks

    Brand creation, brand management, brand building. The mobile telephony sector is outrageously competitive and One-2-One has done a great job so far in shaping a strong, credible brand image.

    Wooing 1.5 million customers is no easy task, particularly in an industry where if customers don't like what they're getting, they have no hesitation in voting with their feet. One2One knows that creating and sustaining a reputation for service excellence is the key to delivering long-term customer loyalty.

    By investing in an automated customer care and billing system, One2One has not only raised the bar on standards of mobile telephony customer service - it has also made a strategic investment in the company's brand equity.

    Peppers & Rogers hit the nail on the head when they said, "Consumers are not on-off switches. They are volume dials. They can, and do, turn the volume up and down on the various products and services they use over a lifetime."

    It's satisfying to see a company like One2One in the throws of phenomenal growth holding true to a very simple sales and marketing premise - that, at the end of the day, meeting and surpassing customer expectations is what really counts.

    Coming next: RAC How did a motoring organisation's used car inspection service cope with the huge growth in demand since its inception, without placing an enormous strain upon the organisation?

    Inspection engineers now capture all the relevant data for the vehicle inspection report easily and quickly at the point of action and transmit that data seamlessly to the report-generating back-end systems.

    Find out how they achieved this in the next BuyIT Best Practice feature.

    At a glance

    The company: One2One is the UK-based Deutsche Telekomm-owned mobile network operations company. It provides 98% coverage of the UK and has 133 international roaming networks across 75 countries worldwide. In the past year One2One has almost doubled its customer base and achieved a 43% growth in revenues.

    The Challenge: How does a customer service and billing operation built to cope with 1.5 million customers in 1999 still meet its customer's expectations when the number of customers has more than doubled year-on-year?

    The solution: By automating the customer care and billing system - but it's not as easy as it sounds. After a three-year search, One2One found a workload management system that works. It was delivered in one month and they achieved their return in a week!

  • Read more on IT risk management