Why do we never learn? The pre-conditions for public sector systems success

This afternoon I was at a meeting discussing why it appears almost impossible to follow good practice when it comes to flagship public sector IT systems. I agreed to “blog” my article for “Transformation”, the magazine of the National School of Government, Spring 2008. By the end of the discussion we found several more reasons why it is so much easier to follow bad practice. However I will stick to what I what I wrote and blog on those later.  

When thinking about large-scale projects in the public sector, people typically leap to the example of IT. So why is it that large IT projects are so beset by problems? Is it unavoidable? Philip Virgo sets out the lessons to be learned from both good and bad projects in the past.


Success came before failure


It is 50 years since the first public sector computer system in the UK went live. The team that  automated the Army payroll to time and budget had honed their “organisation and methods” and “operational research” skills during the Second World War, rushing new weapons systems from research lab to battlefield: when failure meant death in action at best and lost battles or sunk convoys at worst. 25 years ago there was similar continuity in the team responsible for the computerisation of Pay As You Earn (PAYE). There was no change of minister; the tax system was frozen until implementation was complete; and staff training was budgeted to cost more (both time and money) than hardware or software.


But success bred complacency and folie de grandeur.


There have been many studies into the causes of failed computer systems over the past 35 years. Much excellent guidance material has been produced from the days of the Ministry of Technology to the latest guidance from the Office of Government Commerce. But, as pointed out in a letter to the Comptroller and Auditor General accompanying the guidance produced by the IDPM (now the Institute for the Management of Information Systems) in 1994, most failed systems had been doomed before implementation started because those responsible for initial planning and procurement had failed to follow well-established good practice.


Why Big Systems Fail?


The nominal causes of failure are listed in many reports.


The usual top six, in order, are:


Analysis of “business” needs, missing or wrong: failure to undertake a proper needs analysis. Failure to consult those in the front line of delivery or in receipt of services is endemic among those planning new policy initiatives or changes to existing systems.


Needs change before implementation: the churn of Ministers and political priorities is significantly faster than that of technology. It has been suggested that suppliers commonly bid low on the original specification to get the opportunity to make profits on the subsequent changes.


Over-ambition: with regard to what is achievable in practice, given the people, time and budgets available, as opposed to in theory, with the people, time and budgets that are not.


Delay: particularly in agreeing priorities between conflicting objectives, leading to delay in planning and procurement (compounded by staff rotation) before the project gets under way.


Lack of top customer management involvement: and lack of high-level skills, training or experience with regard to planning, procurement or implementation. This lack of training and experience on the part of the customer causes and compounds the previous four problems  and should therefore come first – but rarely does do so in most reports.


Supplier project or team management: usually because the “B” team is trying to salvage an already doomed system after the “A” team has moved on to the next bid.


“Technology problems” rarely appear in the top six. Complex and untested mixes of semi-incompatible software may result in slow, unreliable or unusable systems but, if so, the fault is not the technology but the failure of the customer to require the use of tried and tested products and services and to check claims of inter-operability or performance as part of the procurement process. Blaming consultants and suppliers for competing to reinvent new wheels at taxpayers’ expense is like blaming fox hounds for tearing apart a tame cat.


 Why big public sector systems fail?


Confusion and conflict over objectives and priorities and split responsibility for policy and implementation commonly mean that no one knows what success looks like or is responsible for achieving it from conception to completion. In the worst-case scenario, proposals originate from policy professionals rather than operational staff; they are worked up by officials with equally little experience of implementation or operational delivery; and turned into procurement specifications by consultants and lawyers who have never seen a project all the way through from concept to success.


There will almost certainly be at least two changes of Minister and one of officials between enabling legislation and statutory instrument, let alone procurement.  The supplier’s  “A” teams will then compete to negotiate lowest cost, blame avoidance contracts, to meet a re-negotiated compromise specification that has lost sight of the original objective. Finally the winner’s “B” team arrives to do its best. Meanwhile those who are to use the system have become increasingly frustrated with delays followed by broken promises as to what the system is supposed to achieve and when.


The main reason why such problems persist long after they were first identified is that those who plan clever policies using fashionable technology are promoted to repeat their mistakes elsewhere before they have time to learn. Rudyard Kipling might have had the relationship between Policy Advisors, Technology Gurus and Ministers in mind when he wrote how:


“They sit at the feet – they hear the word – they see how truly the promise runs

They have cast their burden upon the Lord and – the Lord lays it on Martha’s sons”


For Kipling the “Sons of Martha” were the engineers who made the systems and infrastructures of the Empire work. Today they are long gone and we no longer train their successors to do the same for the systems and infrastructure of today.


Once the planning phase has over-run, leaving the more difficult problems to be sorted out during the procurement phase, which over-runs even more in consequence, there is a rush to catch up, using the newcomers in the performance monitoring team, after those who understood the original requirement moved on during the delay. 


The obsession with studies of “Best Practice”, assuming skilled and experienced staff, is itself a prime cause of the continuity of bad practice. The need is to publicise and enforce adequate practice on the part of the people you actually have available. They must then be trained to do so by those who have done it for real, with opportunities for mentored and supervised experience before they are left in sole charge of a project, let alone programme, of their own. The tendency to ‘muddle along’ delays corrective action while problems grow.


Private sector projects are commonly announced only after they have been shown to work.  Moreover the commercial sector sees a reputation for reporting problems in advance and organising rapid remedial action, however brutal, as career-enhancing, not career-limiting.


By contrast, public sector projects are usually announced in advance. There may be increasingly desperate attempts to fix any problems in private, without calling for additional resources, with notification up the chain of command only if the problems cannot be solved with the resources to hand. This is almost certain to limit the careers of those who called for help.


The culture of private cover-up followed by public witch-hunt used not to be peculiar to the public sector. It used to be found in many large businesses. But those facing global competition can no longer afford to try to conceal problems, as opposed to earning reputations for acting fast to resolve them.


Why is the public sector different?


Government systems do not fail because they are larger and more complex than those of the private sector nor is their size and complexity necessitated by most underlying applications. Government has nothing as large and complex as the market-making and payment-clearing services of the financial services and communications industries. Even its largest databases are dwarfed by those in the transaction and information services of the private sector.


The complexity lies in the accountability structures and in the need to respond rapidly to shifting political priorities, with neither consultation with the users nor consideration of the practicality of the consequent “ministerial” demands for change. 


That said, there are some inherent differences between the public and private sectors. For example:


·         The public sector spends 80% of its time and money dealing with that 20% of the population who the private sector does not want as customers; and 


·         Services that meet the needs of the socially excluded and dependent generate more demand, not revenue let alone profit, for those responsible for paying for delivery.


But these factors do not account for the disproportionate failure rate with regard to large central government systems:  moreover devolution to local government and “the third sector” might well be a more rational response to such differences than outsourcing to the commercial sector. The “national standard” approach to public service delivery dates back to Lloyd George’s response to the “something must be done” campaigns of the early press barons. Their successors (including their peers in the BBC) are, today, perhaps the biggest single obstacle to any real change towards community-based, citizen-centric delivery.


In consequence, the inherent differences between the public and private sectors are compounded by the effect of shortening political timescales, publicity-driven imperatives and rising visibility: particularly what can perhaps be best called “The Doctrine of Ministerial Infallibility”. Once a proposal has been said to have Ministerial support it acquires a mystical status, to be justified and defended at almost any cost, until such time as a new Minister can announce that “technologies have changed” and thus justify a new approach. 


A practical difference is that in motivation and skills. Those working in the public sector commonly joined for a life of service and security. If they had wanted to be risk-taking, bonus-motivated entrepreneurs, they would have gone into the private sector. There is also the  poor provision of opportunities for project implementation experience (let alone training) in the public sector. This is particularly serious when one adds the common expectation that officials can perform roles outside their experience or supervise contractors to do so.


Accountability for public funds is another difference but even more profound is the separation between the programme or project manager responsible for change and the accounting officer. This introduces delays in response to problems and opportunities that are peculiar to the public sector. In the private sector the project accountant may report privately to the Finance Director, but it is the project manager who decides and authorises work. 


Why do we never learn?


There have been many reports into why systems fail, especially in the public sector. There have been many fewer on why systems succeed.


The most authoritative and readable recent report, in terms of “professional status”, is probably that of the Royal Academy of Engineering and the British Computer Society on “The Challenges of Complex IT Projects” (April 2004). This quotes Cobb’s Paradox: “We know why projects fail, we know how to prevent their failure – so why do they still fail?” (Martin Cobb: Treasury Board of Canada) before giving some very useful checklists on how to ensure that yours succeed.


The most authoritative, in political terms, is that of the House of Commons Public Accounts Committee (PAC) on “Delivering successful IT-enabled business change ” (May 2007). It is also very readable. 


That PAC report also contains the answer to Cobb’s Paradox. The common factor to the 24 case studies of success on which it is based is that the departmental champion who started the project saw it though to implementation.


Differential management “churn”, not scale or complexity, is the prime reason for the failure to learn as well as for the differential rate of failure between large central government projects (around 70%) and those in local government or the private sector (around 30%). Where the private sector organisation has management rotation routines akin to those of the Civil Service, these are commonly suspended for major change programmes. Moreover the successful implementation of a change programme is not only well-rewarded but is one of the common routes to the top. In consequence those at the top have personal experience of what is entailed – unlike most of those at the top of central government.


What are the lessons?  


The most important is to manage political expectations, beginning with what is realistic, given the time and resources available. This requires ensuring that the minister’s policy team includes advisors with relevant practical experience of delivery.


The next is to try to confine risk to one dimension at a time. For example, if there is a high risk that the objectives or organisational structures will change you should avoid changing supplier or systems at the same time.


Break the programme into modules. In the private sector, projects which take more than three months are more likely to be cancelled than to go live. If the implementation team has not worked together before on programmes of this type it is even more essential to begin with a series of small projects and quick wins to build experience and confidence.


By all means think big, but “start small, test hard, scale fast” is the route to systems success in the private sector. It is not a new technique and has had many names over the years: from structured evolution to rapid application development and dynamic systems methodology.


Don’t be an early adopter. But novelty is not always what you think. Linux is not new. It is forty-year-old Unix re-written for scalability and reliability and most high performance, high volume, online private sector services have open source software at their core.


Review progress regularly against clear, open and ordered priorities and objectives. A project with more than six has none and is probably doomed. And only the top three count. This is not easy, given the temptation to resolve conflict by adding more, but it is essential. 


Maintain team continuity. bringing component projects in a complex programme in to time and budget also entails ensuring that the key players know their next project and are eager to start, but also know they cannot do so until their current one has passed its acceptance test.




Unless and until we rebuild the public sector skills base so as to successfully plan, procure, implement and monitor big change programmes, no tinkering with best practice methodologies or new outsource suppliers or technologies will change the odds on failure.


And until we have rebuilt that skills base we will not even have the skills to stop repeating the same old boring, but highly profitable to consultants and lawyers, mistakes of the past.


We will not learn.