Lessons learned from failure of online passport application system EPA2

In a rare if unprecedented step, the Identity and Passport Service has published the lessons learned from its key IT projects implemented in 2006 – including the lessons from the failure of its EPA2 electronic passport application system.

Computer Weekly’s articles on EPA2 are here, here here and here.

The main lessons are summarised here.

MP3 files of Computer Weekly’s interview in January 2007 with Bernard Herdan, Executive Director, Service Delivery, Identity and Passport Service, are here. In these audio files Herdan says why some online passport applications got stuck in the system. He also explains the difficult decision to take the system out of service, and issues a challenge to other government departments to publish the lessons learned from their key IT projects

A key lesson not specifically included in the passport service’s “lessons learned” report but mentioned by Herdan in his interview with Computer Weekly:

“The user acceptance testing proved functionality and proved that all the various functions of the system worked OK but it did not prove the rate of throughput in volume. So once we started putting volumes of cases through, it [the system] worked fine to begin with but once the volume of cases began to build up then the performance began to degrade.”

This is a lesson that may apply particularly to those planning and implementing systems under the NHS’s National Programme for IT [NPfIT].

Summarised lessons from the report of the Identity and Passport Service on the main IT projects implemented last year follow.

On Complexity:

– “The EPA2 [ Electronic Passport Application] system, as delivered, was considerably more complex than initially intended.

– “In isolation each change was assessed for impact and accepted where necessary, but together rendered the system too complex and much more difficult to test.

– “The Change Control process should be more robust in its examination of proposed changes. Only business critical changes should be accepted.

– “When business change is accepted, the impact on the project and testing schedule must be challenged by the independent test team as well as the project team.

– “The Change Control process should record a cumulative assessment of changes.

On Managing risk:

– “The project team acknowledged that poor performance of the system under live volume conditions was a risk that needed to be managed, but gave it too low prominence.

– “All test plans should be reviewed by an independent Test Assurance team.

– “All risks, not only those assessed as ‘high’, should be reviewed by the Project Boards at regular, quarterly, intervals.

– “Senior Responsible Owners have been directed to sponsor at least two detailed risk workshops per annum on their projects.

On Contracts:

– “The sections of the contract where volume and performance requirements are detailed should have been more clearly understood within the Identity and Passport Service. This problem was exacerbated by changes of management during the project.

– “At the time of a change of Senior Responsible Owner there must be a formal handover. This must include contractual information.

On testing and going live:

– “ EPA2 testing suffered from having to react to changes to Go Live date. There should have been better planning so that test cycles were not conducted under overly tight and challenging timescales.

– “Target Go Live dates should be confirmed by improved plans at the detailed level that are agreed by the independent Test Assurance team.

– “Volume testing should be given greater importance in the test strategy with a better appreciation of the risk of poor performance.

– “The Release Authority was provided with the standard reports, a “Ready for Service” checklist, and an Impact and Mitigation Plan – and the appropriate staff were represented. On the information provided, and on the basis that it was a live trial in Newport only, no questions on the absence of a pilot or adequacy of testing were raised. Ready for Service” Checklists and Impact and Mitigation Plans should seek assurance that where new business processes are being introduced, pilots have been satisfactorily undertaken and adequate volume and performance testing has been successfully completed.

– “There was sufficient resource and good management control of the implementation. However, in view of the difficulties encountered at Go Live, insufficient ‘floor walkers’ and specialist users were on hand to support the EPA2 examiner teams in the event of unexpected difficulties/glitches.

In General:

– “In hindsight, the Identity and Passport Service placed too much reliance on Siemens Business Systems’ own quality control and testing arrangements and assumed that a good quality product, providing the appropriate performance, would be delivered in line with SBS’s excellent past track record on this contract.

– “The Identity and Passport Service should develop stronger technical capability to challenge supplier assertions. This capability is now being developed with recruitment of permanent staff into a strengthened CIO function. For the short term, to provide more depth of client-side technical capability, suitably skilled and experienced interims will be used.

Key lessons from other projects in 2006 of the Identity and Passport Service:

– “Ensure that client and supplier contractual responsibilities are understood. There was some confusion over roles in the subcontract management and product acceptance process which led to disagreement over IPS [Identity and Passport Service] involvement in the acceptance of the main passport production equipment. This did not impact on the outcomes but responsibilities on product acceptance need to be clear in contracts.

– “Develop and assign ownership to a contingency plan. The use of realistic effective contingency planning proved critical to the project’s success, particularly during a period in July/August 2006 when blank book supplies became critically low at a “pinch point” in the roll out. The acceptance of the contingency plan by business owners meant that its implementation when required was efficient and minimised impacts on operations.

– “Test and quality assurance criteria should be explicitly included in commercial contracts. The inclusion of such criteria in commercial documents would give projects more control over suppliers during product delivery and acceptance …”

The full report of the Identity and Passport Service is here.