Agile chaos consumes Universal Credit

It was the core philosophy of the coalition government’s ICT strategy: creative chaos. But instead of fixing government IT it has let it run rampant, according to the National Audit Office.

The coalition came to power claiming government IT projects failed because they were managed too bureaucratically. Yet its £12bn Universal Credit scheme has gone awry through negligence.

After hounding the last government out of power with a barrage of often overblown stories about IT failures, the Condem government must now take its turn in the stocks for running a big government IT project late, probably over-budget and – here’s a new one: with little idea of what it’s doing or where it’s going.

At least the Department for Work and Pensions, which has spent £300m developing Universal Credit since 2010, knew roughly how it was going to get there: by using the radical “agile” approach to software development.

The Cabinet Office and Institute for Government foisted agile on DWP as way to prevent Universal Credit becoming another IT bodge.

But agile was a poor fit, and had never been applied to an immense infrastructure project like Universal Credit. The agile community has since disowned it, saying it was never suited for to projects held by large contracts to fixed terms with large suppliers.

The National Audit Office report into Universal Credit today tells how DWP’s management of the programme – under Cabinet Office guidance – subsequently took on all the trappings of a classic IT bodge.

Yet it also demonstrates how parliament and politics refuse to accept that software projects naturally comprise change; and that when big government allows change in a big project, it can have big consequences. Whether conceit or misunderstanding, this has been one of the main plot lines in the story of IT bodge. It has been misused for political purposes and has brought down both senior administrators and major projects, like tired bulls bayoneted with the spikes of a matador.

Condemned

The NAO nevertheless published its report today under a simple remit: to ensure when a government department says it wants £2.4bn for a large project it doesn’t end up paying double because the programme managers were up the pub and the suppliers were on the take.

On the face of the NAO report, Universal Credit has not gone well at all. DWP suppliers Accenture, BT, HP and IBM managed to produce a pilot of the system. But, said the NAO, it was deficient. The programme achieved a fraction of what it planned by now. Its October deadline to begin the national roll-out of a full system will now mark the extension of its pilot to a few more sites.

NAO said part of the problem was the DWP never knew quite how Universal Credit should work. It never produced a detailed technical blueprint. Poor planning meant it couldn’t measure progress reliably. It couldn’t be completely sure of what the system should do until draft regulations were approved in June 2012. It is even still not entirely sure.

“The department does not yet have an agreed plan for national roll-out and has been unclear about how far it will build on [pilot] systems or replace them,” said the NAO report, Universal Credit: early progress.

The project was poorly managed by a procession of executives and poorly overseen by a sequence of management boards, said the NAO. The DWP wrote off £34m of software development. The NAO expects it to write off more soon. It went back to the drawing board and, the NAO said, may have to cut back its ambitions for the social security programme the Universal Credit system was meant to deliver: help for people who need it the most.

On top of all that it had trouble managing the major suppliers it employed to build the system. It didn’t really know what they should be doing, so it couldn’t really tell if they were doing it properly. There was confusion about who was doing what and how it all fitted together. DWP had “inadequate financial control” over its suppliers. Their payments were poorly accounted.

Agile reassessment

Curiously, the DWP said Universal Credit was still on track to be completed on time and on budget in 2017.

Its remarkable confidence apparently belied the findings of its auditor. But from another perspective, Universal Credit may have been an extraordinary success for a project that had so many odds stacked against it. These have not been clearly set out by the NAO.

The NAO report has also fallen into the same old trap of describing the natural flux of a software project as unwanted flux in a major infrastructure project.

The problem with government IT is not so much that it’s always running over time and budget because it won’t stay still. It is that parliament and politics insist on treating it as though it should stay still.

Or that when a government says it will build a system that does x,y,z in so many years, that it will crucified if only does half the work in its allotted time.

Read roughly, NAO’s account of Universal Credit reads as you might expect from a large-scale agile project. Agile was designed to respect the reality of software development: that it is not like other infrastructure – it does in fact change as you go along because it is an attempt to create a living mould from the intricacies of human activity and the terms and conditions of social intercourse. It’s not quite like building a high-speed train line, even when you do account for nimbies forcing route alterations.

Some aspects of Universal Credit may have been commendably agile in the circumstances. DWP started work on it before all the parameters were set because there were things it could be getting on with. It did not fix a final design because it was still learning how the system should work. Its October pilot extension includes as much as it was able to do in the given time and circumstances.

Big mistake

Its circumstances were complicated by taking on a handful of major Cabinet Office ICT reforms when it was already a unusually ambitious. This piled on extra strain.

The risk with a big project run under big contracts – the sort of risk that is the professional concern of the NAO – is that change costs big money. NAO was greatly concerned therefore that the DWP contracts were neither tight enough nor enforced strictly enough. It was a question the Cabinet Office never addressed publicly when it pushed for agile government IT: how do you ensure the creative chaos doesn’t turn into an eat-as-much-as-you-like for premium-priced IT suppliers?

Universal Credit was going to provide the answers. It was going to be the test-bed for Cabinet Office’s agile policy. Its lessons were going to be used elsewhere in government. But Universal Credit has dispensed with agile and neither department will say anything about it.

Francis Maude, Cabinet Office minister who introduced the agile reforms, hinted at what the coalition thought the answer was when he trailed his ICT policy before the Public Administration Select Committee in 2011.

The problem with government IT, he said, was “the projects have tended to be too big”.

“That is partly a function of us b eing a very centralised country, so a lot of the programmes for which Government IT projects are needed are big national programmes,” he said.

The solution was to break big projects up, he said, and distribute their functions. Yet his own Government Digital Service – the special forces for the coalition ICT strategy – has been using agile methods to centralise the government’s distributed web estate and put it onto one big platform.

Maude’s policy vision was of distributed teams of software developers whose achievements combined to make more than the sum of their parts. What he has done instead is centralise to cut costs – and people. Many government functions are simultaneously being replaced by computers in the coalition’s Digital by Default initiative. The result will be small government, and big society.

Maude’s policy has meanwhile done nothing to eradicate the need for the big departments of state like the DWP to operate big transactional systems like the one that administers social security.

Neither has the state of Universal Credit proven he was right about big government. If DWP had broken the project up and handed it out to many teams of small, agile suppliers it might have made less progress and cost more money than it has with a few large suppliers on large contracts.

There may have been other improvements, though. It might for a start have made the mistakes and alterations in Universal Credit as indiscernible as the goals.