The growing size and complexity of IT systems has pushed the UK's largest telecommunication company into adopting lean, agile software development techniques for its entire new product development process.
This is revolutionary for most new product development teams: candidate projects are no longer subject to the traditional committee-driven, silo approach where information cascades down to developers and filters back up to the client for sign-off.
Instead, BT assembles project teams that consist of every manager and producer relevant to the proposed project, in a space in its Adastral building near Ipswich, Suffolk, the headquarters of BT's network of global development centres. This gets all the decision makers in the room so that ideas can be debated, tested and signed off during the meeting. Teams work in two- or three-week 'sprints' on very short-term goals, and no one project is dated longer than 90 days.
"All I ask is that you give me your gold-plated developers, the best you have, for three weeks while we go through three or four iterations of your system," says Alan Bateman, director of next generation engineering in BT's research and development division, Innovate & Design (I&D), who is responsible for implementing the new approach worldwide.
To make collaboration and documentation across distances and time zones easier, BT has developed a 'collaboration cell' from off-the-shelf components. These consist of two video cameras, one to show what's on the touch-sensitive whiteboard and one to show the people seated at the conference table. The conference table itself has directional microphones to transmit voice as well as jacks for plugging in laptops and other devices.
The entire system works on standard 2mbps telephone lines, which helps to keep costs to about £20,000 per cell. "The cost of dedicated videoconferencing centres with dedicated MPLS networks meant we could probably afford about two," says Bateman.
BT now plans to install hundreds in its development centres in Adastral, Pune (India), Delian (China), Dallas and central Europe, as well as smaller centres such as Cardiff and Edinburgh.
BT is also fixing hygiene factors by revamping offices, adding colours that energise developers, more open spaces where people can congregate to share ideas, and ensuring canteen food is plentiful, attractive and cheap.
BT has some 20,000 development staff around the world, as well as clients in most countries, many of which operate in many countries. Co-ordinating people, skills sets, time zones and timeframes was a nightmare. "It was getting harder to deliver," says Bateman. But the risk was not about getting anything wrong as such, but rather not delivering customer satisfaction, he says.
The £1bn losses in 2008 at Global Services, one of the main users of BT's global development centres, is likely to have cemented the conviction that things needed to change.
The money to be had in the different communications markets has encouraged BT to be a customer in core network technology, and to make its money from services. In BT's latest results, for 2Q09, Openreach earned £513m on revenues of £1.3bn, while Global Services earned £123m on revenues of £2.1bn. The difference in margins, 39% as against 6%, also indicated the need for a step change in productivity.
The first 'total emersion' project to use the new process was the roll-out of BT's 21st Century Network (21CN) next generation network. Russell Strevens, technical director of BT's Openreach division, which runs 21CN, estimates it took less than half the expected time to develop; absorbed a raft of critical changes of decision during its lifecycle; and still gave an 80% return on investment in its first year.
This capacity to deliver quick results, to accommodate late changes and to deliver fast pay-back, persuaded management that agile development is the way to go.
Bateman says using modern software development techniques to develop new communications products was specially apt. "Most telecommunications is now an app," he says, a reference to the converging nature of data, voice and image traffic, aided by platforms such as 21CN.
Very often, the biggest difficulty in an ICT project is knowing what problem you are trying to address. Sometimes, simply tuning the infrastructure can pay huge dividends, says John Wittegreffe, chief ICT researcher at BT Innovate and Design.
Wittegreffe and colleague Jon Clark are developing a universal testbed that models a firm's global IT applications and infrastructures. It also provide a dashboard that reflects their key performance indicators so that they can examine the effects of changes in real time.
Wittegreffe says CIOs can use the system to assure the performance of key business applications to agreed levels. To speed things up they can use data reduction and compression techniques to cut the amount of data passed across networks. This can help to cut bandwidth demands by 30% to 90%, lowering network costs and deliver Lan-like performance across long Wans. They can also use it to schedule just in time access to cloud IT resources so that they pay only for the processing they need.
BT has already used the testbed to help Global Services to fine-tune systems for about 50 customers, he says. Putting the testbed on a mobile phone potentially allows CIOs or operations managers to run their shops remotely, or at least to received alarms when things go out of spec.
Andy Miles, specialist technology consultant at Adastral, is also working on virtual dashboards, using XML-driven widgets to represent real-time status conditions such as weather.
Built in Adobe Air to link to Flash to generate moving images, the prototype provides an easy to use approach to set up virtual displays of data from sources that can produce an XML feed. Global Services managers are playing with the tool to monitor the state of their networks, he says. "We can do this for any system that produces an XML feed, even HR," he says.
It is hard to find out what people are really saying about you or your company, and what you don't know can really hurt you.
Short of paying a clipping agent a fortune to scan the media, blogs and social networks, most companies rely on luck. That's not good enough, says Simon Thompson, a chief researcher and responsible for Debatescape, an early warning tool that scans the electronic media for mentions of BT, and collates and categorises it for managers.
Still in its infancy, Debatescape picks up threads on social networks such as Twitter and Facebook that discuss issues such as poor customer service incidents so that, if appropriate, managers can take direct action with the author.
BT is treating the system as a customer service resolution issue. Managers are already addressing some 3,000 cases a month through the pilot version of Debatescape. "We are now thinking of how to 'operationalise' it as part of the business process," Thompson says.
21st Century Network
BT's 21st Century Network (21CN) is an end-to-end Internet Protocol (IP)-based network. It consolidates 17 separate network platforms into one. It replaces the complex network and systems infrastructure with a physically simpler and more reliable network, to ensure the delivery of the next generation of converged services faster, more efficiently and more cost-effectively.
21CN is a single network infrastructure that supports voice, data, internet and video services. In this world, services are applications, so voice is simply one of many applications running on a common platform, and the systems, services and management processes are shared between the apps.
It required BT to replace the equipment infrastructure in its telephone exchanges across the country.