Extreme programming:Far out software

Extreme programming uses revolutionary working practices to improve the quality of software and the efficiency of its...

Extreme programming uses revolutionary working practices to improve the quality of software and the efficiency of its development, but practicality and traditionalists could delay the popular adoption of XP by decades.

Listening to Ken Auer talk about collaborative working techniques or the importance of individual accountability, you would be forgiven for thinking you had walked in on a human resources management seminar rather than a conversation about software development.

But the principal of RoleModel Software is actually at the forefront of a movement that champions 'extreme programming' - or XP - a methodology that turns many of the traditional assumptions about software development on their heads. Gone is the stereotypical image of the nerdy, lonesome programmer hammering away on a keyboard into the dead of the night. In its place is a movement that focuses on limiting the amount of hours people work, improving team communications and building software specifically to meet customer demand.

As part of a broader movement emphasising 'agile' development methodologies, XP is a small but important contribution to the evolution of software programming. Adopted on a limited scale by a number of well-known organisations -most famously DaimlerChrysler - in many ways it reflects the current business landscape. Management obsession with internal business processes in the early 90s has generally given way to a more outward-looking customer focus, where the overriding aim is to deliver what a customer wants to buy rather than what a vendor wants to sell.

Revolutionary practices
For the software industry, of course, that concept is something of an eyebrow raiser. For years, large parts of the development community have got away with churning out code that's badly designed, riddled with errors, overbudget and overdue. Release 1.0 of a software application has become synonymous with advanced beta testing - it may be billed as a commercial product, but everyone knows it's going to take a couple more versions before the bugs are fixed and the functionality is anything near complete. In many sectors, 'maturity' of a product has become a selling point because it implies stability - an illogical concept for an industry that's supposed to be at the bleeding edge of innovation.

As well as quality control problems, the software development sector has also struggled to cope with the practicalities of meeting customer demand. While service-oriented industries repeat the mantra that the customer is always right, the reality is that customers often have no real idea of what they want and, if they do, their requirements are going to change anyway. Much 21st century management philosophy is geared towards tackling this problem and the concept of business agility is a direct response to the fact that if demand shifts, vendors must shift their ability to deliver.

For the software development community, however, that isn't always easy. On a large-scale project, a significant shift in customer demand can imply major recoding efforts. XP is an attempt to tackle these issues using a combination of best practices, focusing on the way software is developed and the working environment.

So its 12 core practices range from pair programming - a requirement that developers work in twos so that one colleague can challenge the decisions of another -to a stipulation for reasonable working hours, designed to reduce the errors that creep in when programmers are tired. Above all, it aims to tie the customer into the programming effort, so that its real demands are met, and build in quality controls.

The methodology has its roots in a troubled development project undertaken at Chrysler six years ago, commonly referred to as 'C3'. Kent Beck, acknowledged as the founder of XP, was called in to salvage the project and, working with colleagues such as Ron Jeffries, established a series of rules to take the project forward. These guidelines evolved and were subsequently mapped out in Beck's book Extreme Programming Explained: Embrace Change.

Team work
At its heart, XP is underwritten by four core values - communication, simplicity, feedback and courage. Although the last requires some explanation (it can best be understood as having the nerve to rise to the challenge and meet changing requirements), the first three are largely self-explanatory and would make the cut in any management handbook.

In reality, that reflects the nature of XP. Its component parts are often simple and critics point out that many aren't particularly new. What XP is fundamentally about, however, is bringing all of these elements into play as an umbrella discipline for effective software development.

Of the 12 XP practices (see box below), the one that tends to evoke most comment is pair programming, often cited as one of the biggest barriers to adoption of XP from a management and programmer's perspective. In a traditional programming environment, individual developers effectively own their part of the software codeset.

There's very little peer review of the code that's been developed and the levels of quality and creativity depend to a large extent on the individual's skill and attentiveness. Pair programming, by contrast, requires two developers to sit side-by-side at the same machine. As one codes, the other questions what's being done and suggests alternatives.

The advantage of this approach is that it brings discipline and creativity to the programming process. Individuals are less inclined to take shortcuts with someone breathing down their neck. At the same time, by challenging and discussing programming decisions, the combined skills of two developers will often produce a better result.

From an organisational perspective, pairing ensures that program knowledge isn't just held in the head of one developer. Since most XP shops also encourage individuals to switch pairing partners on a regular basis, knowledge and experience is shared around the entire team.

Better by design
But for individual programmers, this is a huge culture shock. In effect, pair programming is a relatively advanced form of collaborative team working. Being constantly questioned about the decisions you're making is a hard discipline, particularly for more experienced developers. It requires an open, blame-free culture, where there's no expectation that people will get things right first time round. In essence, the kind of point-scoring culture that permeates many organisations has to be eradicated for pair programming to work.

RoleModel Software's Auer, who was briefly involved in the C3 project, acknowledges he encountered problems when he first attempted pair programming, working with a fellow developer to rewrite a client's application. "It was very frustrating at the time, but I had to teach the other guy my approach and this was the best way to do it. I felt I was being dragged down and wasn't so productive," he recalls.

At the end of the project, however, he looked back and calculated that his productivity and the quality of the product had improved. "After three weeks, I realised there was no way I could have produced something that good so quickly alone. You're constantly being challenged. Sometimes, the simplest questions are the most effective, like: Why are you doing that? You're in programming mode, not thinking mode - the other guy's thinking," he claims, adding that people taking this approach should "make sure they know it's okay to be uncomfortable at first".

But if pair programming presents a challenge to individual programmers, it's even more of an issue for managers. The idea of having two programmers working on one keyboard implies lower productivity and the most common complaint is that programming resources are being wasted. Advocates of pair programming counter that the productivity drop-off isn't anything like as severe as you might expect and that improvements in quality more than make up for it.

Laurie Williams at the University of Utah, for example, carried out a controlled experiment in 1999 with undergraduates on a software engineering course (see box below). In the first experiment, a team of two collaborators took more than half as long again as an individual developer to carry out an assignment. But when the pair had sufficient time to gel, the extra time taken fell to a mere 15 per cent. The code they produced, meanwhile, had 15 per cent fewer defects, which ultimately represented a huge saving. In addition, there was a wide range of softer benefits, from improved employee satisfaction to better design.

Too much too soon?
In addition to pair programming, several other aspects of XP raise eyebrows, including the concept of continuous testing. As Ron Jeffries writes in an introductory guide to XP: "Extreme programming is obsessed with feedback and in software development, good feedback requires good testing."

Unlike the traditional development environment, XP tests are designed before programming is carried out and because the programming effort is broken into short chunks, the testing process permeates the whole development cycle. It also includes regular client acceptance tests. Compared with pair programming, this practice generally goes down well with managers who appreciate the quality implications - but once again, individual programmers may find it a hard discipline to swallow.

XP advocates also insist that the customer is represented throughout the development process, setting out requirements upfront, getting involved in ongoing planning and ensuring that the delivered product meets their needs. While this all-inclusive approach would delight advocates of customer relationship management, in practice it can prove challenging.

One senior programmer at a major software developer in Silicon Valley, for example, recalls being impressed by the XP approach after he attended a training course, but ran into problems because his customer - an internal department - couldn't clarify its needs.

These kinds of concerns are likely to hold back widespread adoption of the methodology for the foreseeable future. While a number of large organisations have experimented with XP, they have mostly done so in small pilots rather than as a wholesale corporate strategy. That will only change slowly. "I think it will happen, but it will take a long time. The relational database took 20 years to be adopted by the mainstream and Unix took 20 years. I think XP is going to take ten to 20 years," Auer predicts.


The 12 core practices of extreme programming
XP is commonly broken down into a mixture of 12 techniques and rules:
  • Whole team - all project members sit together as one team, including programmers, testers, a coach and representatives of the customer
  • The planning game - XP's two-stage planning process reflects the fact that requirements change as projects develop. In 'release planning', the customer lays out its requirements, costs are estimated and a plan is set out, which is regularly revisited. This is further refined in a more detailed fortnightly 'iteration planning' process, again involving the customer
  • Testing - tests are designed before programming begins and are run each time a developer sends code to the central repository. Customers also define acceptance tests for each major feature
  • Small releases - the emphasis is on supplying code frequently. At the end of every iteration, fully tested software is delivered to the customer
  • Coding standard - all code is written to a common standard to give it the same look
  • Simple design - XP guru Ron Jeffries says: "Design in XP is not a one-time thing or an upfront thing, it is an all the time thing." The focus is on simplicity
  • Pair programming - all coding is carried out with two programmers on one screen, constantly challenging and reviewing each other's work
  • Design improvement - a process of continuous design improvement aimed at removing duplication and increasing the cohesion of the code
  • Continuous integration - XP teams focus on frequent integration (several times a day) as part of the development process
  • Collective code ownership - traditionally, code is 'owned' by the developer, but with XP, where pair programming is the norm and teams are switched, any pair of programmers can improve any code
  • Metaphor - a simple vision of how the program works
  • Sustainable pace - XP emphasises the fact that programmers are in it for the long term and reasonable working hours prevent burn-out

Sources: Ron Jeffries/RoleModel Software

Pair programming
In a 1999 experiment, Laurie Williams at the University of Utah demonstrated that programmers working in pairs took just 15 per cent longer to build a program than developers working on their own. At the same time, their code had 15 per cent fewer defects.

In The Costs and Benefits of Pair Programming, Williams provides an example to show what the experiment might mean financially. Taking a program consisting of 50,000 lines of code and working at a typical rate of 50 lines per hour, individual programmers will take 1,000 hours to complete it. Pair programmers, however, will take 15 per cent longer, or an extra 150 hours.

Quoting other research sources, Williams suggests that programmers typically make 100 defects per 1,000 lines, of which 70 per cent are removed during a good development process. That leaves 1,500 defects in the code developed by individuals. Pairs, however, would have 15 per cent (or 225) fewer defects.

This reduction in defects translates into savings when the code goes to the quality assurance department. Williams argues that a systems test takes between four and 16 hours per defect. If you assume an average of ten hours, the 225 additional defects generated by individual programmers will take 2,250 hours to fix - 15 times more than the 150 extra hours spent by pair programmers.

Read more on Business applications