Olha Rohulya - Fotolia
Two-thirds of the farmers affected by last year’s problems with rural payments IT did not need to use the planned digital system to submit claims for subsidies, but were still forced down a “digital by default” route, Computer Weekly has learned.
About one-third of all potential users of the rural payments online system – which was withdrawn in March last year after persistent technical problems made it unusable – did not need to update data previously held, and another third required only minor changes.
But because of a government push for the project to be a “digital exemplar”, every farmer was initially forced to use the full digital service even if it was not necessary. For most, a simple screen to confirm or amend existing data would have sufficed.
The over-emphasis on a digital-by-default approach is revealed in an internal review produced by the Department for Environment, Food and Rural Affairs (Defra) just weeks after the problems occurred, which has been seen by Computer Weekly.
“There was insufficient up-front profiling of the different user needs at different times of the year,” the report said. “More data could have been gathered earlier about users’ digital capabilities, enabling the service to be rolled out progressively with appropriate audience segmentation.
“Only later did it become apparent that for approximately one-third of users, all the data was already known; another approximately one-third were mostly known with a small delta of updated information required; and the remaining third were new or significantly different and needed the full process.”
The RPA CAPD Lessons Learned Review report was used as source material for highly critical reports published by the National Audit Office (NAO) and Public Accounts Committee, which included scathing critiques of the governance of the project, and detailed extremely poor relations between the Rural Payments Agency (RPA) and the Government Digital Service (GDS).
But the Defra document reveals for the first time full details of the catalogue of errors that led to the biggest government IT failure of recent years – with a 40% budget over-run from £154m to £215m and further costs expected from European Union fines for late submission of farm subsidy claims.
The report said: “There was an insufficient focus early on in the programme on understanding the specific user needs for CAPD [Common Agricultural Policy delivery] and their segmentation.” This was despite the importance placed by GDS on understanding user needs in digital development.
Had the breakdown of farmers’ differing requirements for updating details been identified earlier, then “alternative service models could have been implemented for the majority of users, significantly improving the user experience and perceptions of the programme” the report said.
Defra claimed there were new requirements in 2015 that meant farmers needed to input new, and change existing, land information as part of submitting their claim.
“The rural payments system is working,” said a Defra spokesperson. ‘Over 86,000 farmers used it to successfully register for the 2015 Basic Payment Scheme and it has been used to pay farmers since the start of December’s payment window as well as take applications for 2016.
“We have listened to farmers and agents and the service has been simplified for 2016. As a result, those farmers applying online this year can view their maps as well as transfer land parcels and entitlements online. Feedback from those using the 2016 service has been good.”
Serious problems were apparent in the rural payments project from its earliest days, according to the Defra report.
RPA management had purchased software called Abaco at an early stage to manage the complex rules around CAP payments, a decision that subsequently “constrained the programme” because “purchasing the product imposed its architecture”, said the review. The report also suggests the Abaco software had to be heavily customised.
“We’ve been criticised for our approach – parts of the approach were great, parts were not. We’ve learnt from it”
Liam Maxwell, chief technology officer, GDS
Abaco is used in other EU countries but, according to the Defra report, it was not designed for use by farmers but by their agents and other intermediaries – a different scenario to the UK farming sector. Also, the report said the Abaco product “uses outdated technology and an architecture that is not designed for large-scale concurrent online usage”.
However, Defra has previously stated: “Abaco was chosen because it has a proven track record in being able to process the complex rules of the new CAP. Abaco is used by 25 ministries and paying agencies in Europe, supporting more than two million customers to manage 250 million land parcels. Without Abaco, farmers would not be receiving payments now.”
Pressure from the start
The NAO report, published in December 2015, highlighted cultural differences between the RPA and GDS. The Defra review described an RPA approach that “appears to view technology as a second-order function” and with a “bias towards the preservation of the type of paper-based and manual processes that have increasingly become redundant elsewhere”.
The early “alpha” stage of the project put the whole programme under further pressure from the start, said the Defra document. It took nine months and ran over time and over budget – with costs of that phase increasing from £250,000 to about £800,000 – despite focusing only on one aspect of the desired functionality, the land management system, instead of testing “an end-to-end slice through the CAP user journey and experience”.
The delays to the alpha phase had “a major impact on the delivery viability of the end-to-end programme”, said the report. Computer Weekly reported last year that an assessment of the alpha phase failed the official GDS test to allow it to continue, but the system progressed to public use nonetheless.
The RPA had historically outsourced all its IT requirements and, as a result, “lacks in-house experience in the use of modern technology practices and methods, and appears to view IT activities and related data processing activities as a risk”, said the report.
Despite this attitude towards IT, the RPA’s business case failed to include a section on technology and IT was also “entirely absent” from the associated risk summary.
“This highlights a significant underlying but problematic schism between the commitment to a ‘digital by default’ solution championed by GDS, Defra and central government, and one based on paper processes, agents, and traditional processes preferred by RPA,” said the review.
The document highlighted a specific example of problems caused by RPA’s “bias away from technology solutions”, concerning known data corruption in the system that went live.
“Rather than fixing the incorrect data in the system directly, the preferred solution was to send the incorrect data to customers in pre-populated [paper] forms then get them to make the correction and return them, and only then would the RPA manually fix the wrong data in the system case by case,” it said.
The report further questions the technology choices made for the project, in addition to the issues with Abaco. The move from outsourcing to a “mixed portfolio” of customising packaged applications, developing new software, and using different hosting environments for different aspects, proved extremely problematic.
“The programme lacked sufficient experience and resource to manage the end-to-end integration of this mixed portfolio, with the few senior technical resources being thinly stretched and inadequately supported, and the earlier systems integration function largely absent,” said the report.
Read more abour the rural payments IT problems
- The Rural Payments Agency (RPA) looks to procure several systems – including an identity management system, infrastructure and HR, for the next stage of its IT.
- The rural payments programme was a “Whitehall fiasco” with too much focus on the digital front end, says the Public Accounts Committee.
- Inappropriate behaviour and too much focus on IT procurement led to rural payments programme costs spiralling to 40% over budget, says National Audit Office report.
Also, there was no “end-to-end testing environment that mirrored production” and at times suppliers were testing their own components in isolation from the rest of the overall system. There was “no non-functional testing” and “inadequate or absent performance testing,” said the review.
“Updates were released that weren’t ready, solely to hit a deadline and claim it had been delivered on time. However, weeks were then absorbed afterwards on fixing what had been released and bringing it up to an acceptable standard,” it said.
Security requirements for the system were also “over-specified”, causing “end-to-end latency and degradation of performance”.
Lack of architecture
The report implies there was little or no over-arching architecture to guide the technical aspects of selecting and integrating systems, and also ended up with “the worst of two worlds” in its poor use of both agile and waterfall styles of software development. “Too much effort was focused on creating a website versus creating an end-to-end system,” said the report.
The December NAO report highlighted a lack of suitable in-house skills as a “central cause of failure” for the project. The Defra review cited a particular reason for this as the imposition of an “arbitrary” day rate for paying contractors brought in to work on the programme.
Defra 'lessons learned' review
“Rather than paying market rates for the right level of experience, less experienced resources were deployed with a detrimental impact,” said the review.
“A programme of this size is unlikely to have exceeded around 120-150 people if the right talent had been utilised. However, it instead exceeded 300.”
According to sources, the core of the technical problems that scuppered the system lay in the Abaco system locking database records at a very low level, so that any simultaneous attempts to access records led the software to lock up.
However, the front-end web portal that was used by farmers to access the Abaco back-end presented multiple connections that the database could, therefore, not cope with. The technical problems ultimately proved terminal, but the real difficulties in the programme had, by that late stage, already sealed its fate.
Criticism of GDS
While the Defra report is critical of the RPA, it also offers more details of the criticism of GDS that led to it being labelled as a hindrance to the project by MPs on the Public Accounts Committee (PAC).
The review acknowledged that “everyone was supportive of GDS’s mission to help transform and reform public services by radically improving the use of technology,” but it also listed a series of failings – in particular a lack of accountability.
The report cited GDS behaviour described as “drive-by comments” and “naïve interventions” from “a range of GDS visitors, none of whom had accountability or responsibility for the programme but whose interventions consumed time, money and attention”.
The report added: “There remains an important issue of governance concerning GDS’s relationship with departments and agencies when working on programmes such as this, particularly with regard to accountability”, although it acknowledges that such behaviour at RPA was “subsequently fixed”.
Other criticisms of GDS in the report included:
- The need for GDS to have a more “diverse bench of senior experienced resources” to deliver major programmes with complex requirements;
- The need to provide a “consistent architectural vision, model or blueprint for how a large-scale programme should be designed”;
- Needing a better model for working with departments, to avoid “a tendency to ‘drift in and out’ of programmes or ‘parachute people in’ with no ‘skin in the game’”;
- A “bias towards open-source software” when more emphasis should be given to open standards;
- GDS “applies agile as the only tool even where it doesn’t fit, and then only uses one agile method”;
- GDS “tends towards a mindset of ‘let’s build everything’ from scratch even where commodity products or services would be more appropriate”;
- GDS “appears biased towards resources with knowledge only of building websites and the front-end user experience” – which the report labels “portalitis” – rather than “a broader capability to provide holistic end-to-end engineering, including integration with complex existing estates and legislative systems.”
And in a further echo of NAO and PAC criticisms of “inappropriate behaviour” from executives at both RPA and GDS, the Defra review said GDS staff “exhibited some immature behaviours” such as “a sense of entitlement”, “rubbing people up the wrong way” and “just plain rude”. The report suggested that GDS discouraged critical feedback “as part of a ‘good news’ culture”, and was “over-promising and under-delivering rather than tackling difficult business and systems engineering issues”.
The report said people interviewed about the project felt GDS should have been more transparent, should have admitted publicly when problems occurred and learned from them, “rather than being in hostile denial”.
Important lessons for GDS
In an interview with Computer Weekly conducted without reference to the Defra review document, government CTO Liam Maxwell – who was brought in as senior responsible owner in the later stages of the project – said that GDS had learned important lessons from its experience at RPA.
“We’ve fully taken on board the criticism that we as an organisation should have put in more capability earlier and perhaps put in some more guidelines,” he said.
“We needed to have a more of an approach to a minimum viable product which would deliver what was there, and in fact that’s what they’re delivering this year. We’ve been criticised for our approach – parts of the approach were great, parts were not. We’ve learnt from it.”