The Post Office’s Horizon IT system should “never have seen the light of day” and bosses at supplier Fujitsu allowed it to be rolled out into the Post Office network despite being told it was not fit for purpose, according to a senior developer who worked on the project before it went live.
Horizon is the system at the heart of the scandal that saw hundreds of subpostmasters wrongly accused of fraud and false accounting until a 2019 High Court case proved the Post Office IT was at fault. Horizon is a counter-top sales and accounting system, commissioned by the government to automate Post Office services. It went live in Post Office branches in 1999 and an updated version is still in use today.
The developer, who has not previously talked publicly about his experiences on the project, told Computer Weekly that in the months leading up to its launch, Horizon’s problems were well known inside Fujitsu.
“Everybody in the building by the time I got there knew it was a bag of s**t”, he said. “It had gone through the test labs God knows how many times, and the testers were raising bugs by the thousand.”
The senior developer said he was contracted to work on the Horizon project between 1998 and 2000, at one point holding the job title Horizon Epos [electronic point of sale] development manager. He has asked to remain anonymous, but is prepared to give sworn witness statements to solicitors acting for subpostmasters in their ongoing appeals against past convictions.
The developer has also asked Computer Weekly to pass his contact details to the government’s Post Office Horizon IT inquiry, chaired by former High Court judge Wyn Williams.
Central to his allegation is that Horizon’s Epos system was initially built with “no design documents, no test documents, no peer reviews, no code reviews, no coding standards”.
He told Computer Weekly: “To my knowledge, no one on the team had a computer science degree or any degree-level qualifications in the right field. They might have had lower-level qualifications or certifications, but none of them had any experience in big development projects, or knew how to do any of this stuff properly. They didn’t know how to do it.”
In 2019, a High Court judge ruled that Horizon was “not remotely reliable” for the first 10 years of its existence. The judge found the IT system was prone to throwing up errors that could and did affect individual subpostmasters’ branch accounts. As Computer Weekly has been reporting since 2009, the Post Office held postmasters liable for these discrepancies. Some were prosecuted and imprisoned. Others lost their jobs and life savings.
Largest commercial IT system in Europe
When it was rolled out, Horizon was described by Fujitsu as the “largest non-military IT system in Europe”. About 40,000 Horizon terminals were installed in all Post Offices across the UK. The user interface was a touchscreen and keyboard linked to a PC under the counter which ran on the Windows NT operating system. Branch PCs were connected via ISDN to a back-end mainframe. The Fujitsu-designed Epos software on the PCs was written onto an off-the-shelf system called Riposte.
Our source said the big flaw in Horizon was the way data was being written to Riposte.“Riposte wasn’t really a database, it was a messaging system based on an XML structure where you write messages down into the message store, and then Riposte took care of replicating them,” he said.
“The first thing that you should always do with a system like that is design and agree a data dictionary and a message library repository, basically to say: these are the messages that are allowed to be written to the message store and they all provide the following function.
“It’s almost like an API [application programming interface] so that you have a list of allowed messages that can all be written to the correct format with the correct content.
“You should also have a layer of software that lies on top of the message store that checks that any application above it which is trying to write a message, conforms to the agreed data dictionary. Otherwise, you can just write freestyle to the message store, which is what they were doing. There was no application interface in there, no agreed data catalogue or anything.”
Computer Weekly also spoke to a former Fujitsu employee who worked in the Horizon Service Support Centre from 2001. The support engineer – who also wants to remain anonymous – recognised this new description of Horizon’s badly built message store, adding: “Our job was to fix these problems as they arose. We all knew the code wasn’t fit for purpose and needed rewriting. The data dictionary was still being added to when I got there.”
Senior managers were aware
The most serious allegation raised by the developer is that senior managers at Fujitsu were aware that an important element of the Horizon system did not function correctly and could not be fixed.
For the first 10 years of Horizon’s existence, transaction and account data was stored on terminals in each branch before being uploaded to a central database via ISDN. Our source says this part of the system simply did not work.
“The cash account was a piece of software that sat on the counter NT box, asleep all day,” he said. “At the end of the day, or a particular point in the day, it came to life, and it ran through the message store from the point it last finished. It started at a watermark from yesterday and combed through every transaction in the message store, up until the next watermark.
“A lot of the messages in there were nonsense, because there was no data dictionary, there was no API that enforced message integrity. The contents of the message were freehand, you could write whatever you wanted in the code, and everybody did it differently. And then, when you came back three weeks later, you could write it differently again.”
He gave an example of a message stored previously when a customer bought a stamp. It was feasible that a new message for buying a stamp weeks later could be slightly different.
“When the cash count came along, it found a message it was not expecting and either ignored it, tripped up, or added something it shouldn’t be adding,” he said.
In 2015, Computer Weekly reported another anonymous source who identified the cash writing program as a possible cause of serious problems. He told us the Post Office was warned about the risk of data corruption on the bespoke asynchronous communication system which sent messages between branches and the central Horizon set-up.
Speaking to Computer Weekly in 2015, the anonymous source told us: “The asynchronous system did not communicate in real time, but does so using a series of messages that are stored and forwarded, when the network connection is available. This means that messages to and from the centre may trip over each other. It is perfectly possible that, if not treated properly, messages from the centre may overwrite data held locally.”
Four years later, former Fujitsu engineer Richard Roll wrote in a witness statement to the High Court: “The issues with coding in the Horizon system were extensive. Furthermore, the coding issues impacted on transaction data and caused financial discrepancies on the Horizon system at branch level.”
Roll’s evidence, which was accepted by the judge, suggests that the problems with Horizon identified by our source had not been dealt with by the time the system went live.
Other experts familiar with Horizon that Computer Weekly approached have also supported the developer’s claims.
The developer said he made his superiors at Fujitsu aware of the extent of the Epos system problems, telling them explicitly that the cash account needed to be scrapped.
“I broke it down and said: you can keep these bits at a push if you have to,” he said. “But that bit in the middle, these bits of the engine, the gearbox, you need to throw them away and rebuild them. Starting with the cash account. You’ve got to throw the cash account away and you’ve got to rewrite it.”
Bone of contention
Our source said Fujitsu’s unwillingness to deal with this central problem became a bone of contention with him. He said he was offered more responsibility on the project but refused unless the cash account was rebuilt from scratch. This, he claimed, resulted in him being moved off the Epos side of the project to become Horizon’s LFS [Logistics Feeder Service] team development manager.
He concluded: “It was a prototype that had been bloated and hacked together afterwards for several years, and then pushed screaming and kicking out of the door. It should never have seen the light of day. Never.”
Computer Weekly contacted Fujitsu, asking for a response to the developer’s allegations. A spokesperson for the company said: “As a long-term partner to UK public and private sector organisations, we are dedicated to supporting our customers, our employees and the people they serve in the UK. We provided detailed responses to all questions raised by the House of Commons BEIS Select Committee and are continuing to cooperate with the ongoing Post Office Horizon IT inquiry.”
The Post Office said it would not be appropriate to comment on individual allegations outside of the current independent inquiry and the courts.
The Department for Business, Energy and Industrial Strategy (BEIS), which holds government responsibility for the Post Office, said it had “established an independent, judge-led inquiry to ensure that lessons are learned, and that concrete changes take place at Post Office Ltd. If anyone has relevant information, we would encourage them to provide it to the Post Office Horizon IT inquiry.”
The Williams inquiry into the Post Office Horizon IT scandal is accepting evidence until 25 February and is due to report in the summer.
A Computer Weekly investigation in 2009 first revealed that subpostmasters, who run Post Office branches, were being blamed for unexplained financial losses, which they claimed were caused by errors made by the Horizon system. The Post Office denied this, and many subpostmasters were subsequently prosecuted for theft and false accounting, with prison sentences, community service, criminal records and heavy fines among the injustices they suffered as a result.
It has become one of the biggest miscarriages of justice in UK history.
Timeline of the Post Office Horizon articles since Computer Weekly first reported on it in 2009
- May 2009: Bankruptcy, prosecution and disrupted livelihoods – postmasters tell their story.
- September 2009: Postmasters form action group after accounts shortfall.
- November 2009: Post Office theft case deferred over IT questions.
- February 2011: Post Office faces legal action over alleged accounting system failures.
- October 2011: 85 subpostmasters seek legal support in claims against Post Office computer system.
- June 2012: Post Office launches external review of system at centre of legal disputes.
- January 2013: Post Office admits Horizon system needs more investigation.
- January 2013: Post Office announces amnesty for Horizon evidence.
- January 2013: Post Office wants to get to bottom of IT system allegations.
- June 2013: Investigation into Post Office accounting system to drill down on strongest cases.
- July 2013: Post Office Horizon system investigation reveals concerns.
- October 2013: End in sight for subpostmaster claims against Post Office’s Horizon accounting system.
- October 2013: Former Lord Justice of Appeal Hooper joins Post Office Horizon investigation.
- November 2013: 150 subpostmasters file claims over “faulty” Horizon accounting system.
- September 2014: Fresh questions raised over Post Office IT system’s role in fraud cases.
- December 2014: MPs blast Post Office over IT system investigation and remove backing.
- December 2014: Why MPs lost faith in the Post Office’s IT investigation, but vowed to fight on.
- December 2014: MPs to debate subpostmaster IT injustice claims.
- December 2014: MP accuses Post Office of acting “duplicitously” in IT investigation.
- January 2015: MPs force inquiry into Post Office subpostmaster mediation scheme.
- January 2015: Post Office faces grilling by MPs over Horizon accounting system.
- February 2015: Post Office CIO will talk to any subpostmaster about IT problems, promises CEO.
- March 2015: Post Office ends working group for IT system investigation day before potentially damaging report.
- March 2015: MPs seek reassurance over Post Office mediation scheme.
- March 2015: Retiring MP aims to uncover truth of alleged Post Office computer system problems.
- April 2015: Post Office failed to investigate account shortfalls before legal action, report claims.
- April 2015: Criminal Courts Review Commission set to review subpostmasters’ claims of wrongful prosecution.
- May 2015: IT system related to subpostmaster prosecutions under review by CCRC.
- June 2015: Post Office looking to replace controversial Horizon system with IBM, says MP.
- July 2015: Campaigners call for independent inquiry into Post Office Horizon IT system dispute.
- October 2015: James Arbuthnot takes Post Office IT fight to House of Lords.
- November 2015: The union that represents Post Office subpostmasters has warned of a problem with the Horizon accounting system.
- November 2015: An email from Post Office IT support reveals a problem with the Horizon system and supporting processes that could lead to accounting errors.
- November 2015: Group litigation against Post Office being prepared in Horizon dispute.
- February 2016: Post Office faces group litigation over Horizon IT as subpostmasters fund class action.
- June 2016: Post Office chairman Tim Parker says there would be “considerable risk” associated with changing its Horizon computer system.
- November 2016: The legal team hired by a group of subpostmasters will take their case to the next stage.
- January 2017: The group action against the Post Office that alleges subpostmasters have been wrongly punished for accounting errors gets a green light from the High Court of Justice.
- March 2017: 1,000 subpostmasters apply to join IT-related group litigation against Post Office.
- April 2017: Investigation into claims of miscarriages of justice in relation to a Post Office accounting system has appointed a forensic accountant firm.
- May 2017: Hundreds of subpostmasters have applied to join IT-related legal action since March.
- July 2017: Post Office defence in computer system legal case due this week.
- August 2017: Campaigners submit initial evidence in group litigation against Post Office over controversial Horizon IT system.
- October 2017: Subpostmasters’ group action against the Post Office reaches an important milestone.
- November 2017: An end is in sight for subpostmasters’ campaign against alleged wrongful prosecution, which they blame on a faulty computer system.
- November 2017: The High Court judge managing the subpostmasters versus Post Office legal case over an allegedly faulty computer system tells legal teams to cooperate.
- January 2018: Forensic investigation into Post Office IT system at centre of legal case nears completion.
- April 2018: Criminal Cases Review Commission forensic examination of the IT system at the centre of a legal case against the Post Office has raised further questions.
- May 2018: Post Office branches unable to connect to Horizon computer system for several hours after morning opening time.
- October 2018: After over a decade of controversy, next week marks the beginning of a court battle between subpostmasters and the Post Office.
- November 2018: Case against Post Office in relation to allegedly faulty computer system begins in High Court.
- November 2018: High Court case in which subpostmasters are suing the Post Office has revealed a known problem with a computer system at the core of the dispute.
- November 2018: A High Court trial, where subpostmasters are suing the Post Office for damages caused by an allegedly faulty IT system, ends second week.
- November 2018: Post Office director admits to Horizon errors and not sharing details with subpostmaster network.
- November 2018: The High Court trial in which subpostmasters are suing the Post Office has reached an important stage.
- December 2018: CCRC may hold off subpostmaster decision until after Post Office Horizon trial.
- December 2018: Court case where subpostmasters are suing the Post Office set to span at least four trials and extend into 2020.
- January 2019: Subpostmasters’ campaign group attacks Post Office CEO Paula Vennells’ New Year honour amid ongoing court case.
- January 2019: Thousands of known errors on controversial Post Office computer system to be revealed.
- March 2019: Tech under spotlight at High Court in second subpostmasters versus Post Office trial.
- March 2019: Post Office considered Horizon IT system “high-risk”, court told.
- March 2019: CCRC watching Post Office Horizon trial closely.
- March 2019: Judge rules that Post Office showed “oppressive behaviour” in response to claimants accused of accounting errors they blamed on Horizon IT system.
- March 2019: Post Office “lacked humanity” in the treatment of subpostmasters, says peer.
- March 2019: A High Court judge heard that the Post Office did not investigate a computer system error that could cause losses, despite being offered evidence.
- March 2019: The Post Office legal team in the case brought by more than 500 subpostmasters has called for the judge to be recused after questioning his impartiality.
- March 2019: A senior civil servant asked the Post Office to repay public money it had wrongly allocated to paying legal costs.
- April 2019: Subpostmaster claimants’ legal team makes application for the Post Office to pay millions of pounds of costs associated with trial.
- April 2019: Post Office to appeal judgment from first Horizon trial.
- April 2019: The Post Office’s claim that the judge overseeing the case concerning its controversial Horizon IT system was biased has been dismissed.
- April 2019: MP questions government over Post Office Horizon case.
- April 2019: Government says no conflict of interest in trial despite Post Office chairman’s dual role.
- May 2019: The Court of Appeal has refused the Post Office’s application to appeal a major decision in the Horizon IT trial.
- May 2019: The Post Office has applied for permission to appeal judgments from the first trial in its IT-related legal battle with subpostmasters.
- May 2019: The judge in the Post Office Horizon trial has ordered the organisation to pay the legal costs of its courtroom adversaries, and refused to give permission to appeal a major judgment.
- June 2019: Post Office asks Court of Appeal for permission to appeal judgment in first Horizon trial.
- July 2019: The Post Office has admitted that some subpostmasters are at risk of accounts not balancing due to an error it does not understand.
- July 2019: Problem revealed during High Court trial left subpostmaster with £18,000 surplus after IT system failed to register full amount of cash scanned in.
- August 2019: Subpostmasters suffering slow running and frozen terminals while Post Office searches for a fix to issues apparently caused by a software update.
- August 2019: The Post Office has fixed the latest problems with its Horizon system, affecting hundreds of branches.
- October 2019: A High Court judgment for a trial that focused on the Post Office’s IT system at the centre of a multimillion-pound litigation will be announced early next month.
- November 2019: The Court of Appeal has rejected a Post Office application to appeal judgments made in its multimillion-pound battle with subpostmasters over IT system failures.
- November 2019: Peer calls for clear-out of Post Office board after Court of Appeal confirms major court defeat.
- December 2019: The Post Office has settled its long-running legal dispute with subpostmasters, and will pay £57.75m in damages.
- December 2019: Subpostmasters ended their legal battle with the Post Office at the optimal time, according to the lawyer that managed the High Court action.
- December 2019: Subpostmasters proved right on IT system failures as calls for full public inquiry mount.
- December 2019: Criminal Courts Review Commission to review Horizon judgment “swiftly”.
- December 2019: National Federation of Subpostmasters cries foul after court ruling on controversial computer system.
- December 2019: Former Post Office CEO apologises to subpostmasters over Horizon scandal.
- December 2019: Call for former Post Office CEO to step down from public roles after IT court battle lost.
- January 2020: Fujitsu must face scrutiny following Post Office Horizon trial judgment.
- January 2020: Subpostmaster group calls for government to pay legal costs for Horizon trial.
- January 2020: Why subpostmasters are calling on the government to pay Horizon trial costs.
- January 2020: Department for Business, Energy & Industrial Strategy says it did not make decisions in the Post Office’s recent court battle.
- January 2020: Government should not be allowed to dismiss subpostmasters’ claims over Horizon IT scandal.
- January 2020: Police sent information about potential Fujitsu staff perjury in subpostmaster prosecutions.
- January 2020: Prosecutions are a significant step closer to being sent to the Court of Appeal as Criminal Courts Review Commission forms a group of commissioners to review them.
- January 2020: Alan Bates: The “details man” the Post Office paid the price for ignoring.
- February 2020: The government has refused to pay the huge legal costs subpostmasters incurred in their battle with the government-owned Post Office, which they won.
- February 2020: Members of Parliament seeking a public inquiry into the Post Office Horizon scandal face huge challenges, but pressure and time could force justice.
- February 2020: Calls for inquiry into Post Office IT scandal increase in Parliament, with cross-party support.
- February 2020: Care Quality Commission to review concerns over Paula Vennells’ appointment after they were raised by a former NHS consultant psychiatrist.
- February 2020: Government admits it was too passive managing Post Office as parliamentary pressure builds.
- February 2020: Minister says Post Office IT experts misled the government when it asked questions about subpostmasters’ concerns over Horizon IT system.
- March 2020: Boris Johnson commits to “getting to the bottom of” Post Office Horizon IT scandal.
- March 2020: Boris Johnson’s commitment to inquiry into Post Office scandal in doubt.
- March 2020: MPs call on PM to commit to full public inquiry into Post Office Horizon IT scandal.
- March 2020: Those who did not play by the rules in Post Office Horizon scandal “should face prosecution”.
- March 2020: MPs told to hold to account those responsible for Post Office Horizon IT scandal.
- March 2020: The Post Office has sparked anger with secret settlements with subpostmasters outside the recent legal action against it.
- March 2020: Labour MP Karl Turner tells Computer Weekly that the Post Office Horizon scandal is the most grotesque version of predatory capitalism he has ever seen.
- March 2020: MP Kevan Jones has warned a government minister not to repeat the mistakes of predecessors in relation to the Post Office Horizon IT scandal.
- March 2020: Criminal Cases Review Commission to use Microsoft Teams to ensure review of subpostmaster prosecutions is held on time.
- March 2020: Post Office postpones subpostmaster compensation scheme amid Covid-19 crisis.
- March 2020: Meeting reviewing subpostmaster applications to appeal criminal prosecutions moves into second day.
- March 2020: Subpostmaster prosecutions to be considered by Court of Appeal for miscarriages of justice.
- March 2020: How subpostmasters made legal history with biggest referral of potential miscarriages of justice.
- April 2020: Met Police examines information about evidence given in court by Fujitsu staff on the Horizon IT system.
- May 2020: Subpostmasters who had their lives ruined by the Post Office’s faulty IT system have received their damages after a High Court victory.
- May 2020: A senior Post Office executive at the centre of an IT scandal, who tried to mislead a High Court judge in relation to it, has left the organisation without fanfare despite many years of service.
- May 2020: Post Office re-examines hundreds of prosecutions that could have resulted from faults in Horizon IT system.
- June 2020: A campaign group representing subpostmasters wrongly prosecuted for theft and false accounting by the Post Office is raising money to help clear the names of victims of the scandal.
- June 2020: Subpostmasters to force scrutiny of government’s role in Post Office IT scandal.
- June 2020: The Criminal Cases Review Commission sends 47 more subpostmaster cases to Court of Appeal and asks government to review private prosecution powers.
- June 2020: Select committee chair writes to former Post Office CEO demanding answers over her role in IT scandal.
- June 2020: The government has been accused of launching a review that fails in getting to the bottom of one of the biggest miscarriages of justice in UK history.
- June 2020: Subpostmasters will not cooperate with government review into IT scandal.
- June 2020: The government’s proposed review of the Post Office IT scandal has received a further setback as forensic accountants join subpostmasters in refusing to back it.
- June 2020: Call for government review of Post Office Horizon scandal to have the power to force individuals to give evidence under oath.
- June 2020: Subpostmasters seeking justice in the Post Office Horizon IT scandal are regaining momentum in Parliament.
- June 2020: Healthcare regulator will be discussing concerns about former NHS boss chairing an NHS trust at an upcoming meeting.
- June 2020: Second Sight is working with law firm in appeals by subpostmasters against criminal convictions in Horizon IT scandal.
- June 2020: Post Office and Fujitsu blame each other for many of the failings in the Horizon IT scandal that wrecked lives.
- June 2020: Parliamentary Justice Committee to hold short inquiry into the rules and regulations surrounding private organisations’ ability to initiate criminal proceedings.
- July 2020: Victims of the Post Office Horizon IT scandal need to raise thousands of pounds in a week or those responsible for their suffering will avoid scrutiny.
- July 2020: The government is set to face scrutiny over its involvement in the Post Office Horizon IT scandal, described as one of the biggest miscarriages of justice in modern UK history.
- September 2020: The government repeats that it won’t pay victims’ legal costs and confirms review into the scandal will not have the power to call witnesses.
- September 2020: Subpostmasters still not being told about all the known errors in the controversial Post Office branch accounting and retail system that they use.
- October 2020: The Post Office has chosen not to contest 44 out of 47 appeals, meaning most are likely to have their names cleared, but others still face a Court of Appeal battle for justice.
- October 2020: MPs are demanding the government holds a full statutory public inquiry into the Post Office IT scandal.
- October 2020: NHS regulator continues enquiries about the appointment of former Post Office CEO at Imperial College Healthcare NHS Trust as more damning details emerge.
- October 2020: Government minister met with former subpostmaster online in an attempt to get victims of the Post Office Horizon scandal involved in government review.
- October 2020: The Post Office is focusing urgently on fixing an IT error suffered by a subpostmaster amid the ongoing IT scandal.
- October 2020: Labour politicians are calling for the government to give the Post Office Horizon scandal inquiry the power to force witnesses to give evidence if they don’t cooperate.
- October 2020: Imperial College Healthcare NHS Trust has asked for external review of its process when appointing controversial executive.
- November 2020: Government faces scrutiny of its handling of the Post Office IT scandal that destroyed subpostmasters’ lives and livelihoods.
- November 2020: Post Office branches offline during busy business hours after suffering an IT error that the Post Office said related to IT from supplier Fujitsu.
- November 2020: Fujitsu is refusing to explain what caused a national system outage in Post Office branches last week, despite the Post Office confirming the issue was the fault of the supplier.
- November 2020: The Metropolitan Police opens criminal investigation into Fujitsu staff who gave evidence in trials of subpostmasters wrongly prosecuted and even imprisoned for financial crimes.
- November 2020: Post Office criticised over vagueness of its explanation of the cause of a UK-wide IT failure that saw subpostmasters unable to do business.
- November 2020: Post Office says planned firmware update caused the problem that left branches unable to do business for 90 minutes.
- November 2020: Court documents reveal the names of the Fujitsu employees under investigation for potentially providing misleading information in criminal trials.
- November 2020: The government allowed the Post Office to “run amok” and destroy lives, says complaint to Parliamentary Ombudsman.
- November 2020: Campaigning politician demands access to documents that could prove that the Post Office lied.
- December 2020: Government denies responsibility for the abuse inflicted on subpostmasters by the Post Office over faulty IT system.
- December 2020: CEO at the centre of the scandal that saw innocent people bankrupted and some sent to prison steps down from NHS role as pressure for her resignation grows
- December 2020: History made as subpostmasters wrongly prosecuted in Horizon IT scandal have convictions quashed.
- December 2020: The appointment of a former Post Office executive, who tried to mislead a judge, in the Football Association of Wales has been questioned by an MP.
- December 2020: Court of Appeal indicates subpostmasters can pursue appeal route that could do more damage to Post Office’s reputation.
- January 2021: NHS trust defends its director appointment process following an external review of its recruitment of former Post Office CEO Paula Vennells.
- January 2021: Lawyers call for changes to digital evidence rule that made it easier for the Post Office to ‘bamboozle courts’ and make subpostmasters pay a heavy price for its IT failings.
- January 2021: The Criminal Cases Review Commission (CCRC) has referred four more subpostmasters’ criminal convictions to appeal, as part of the biggest miscarriage of justice in modern UK history.
Read more on IT outsourcing
Post Office gives controversial Fujitsu contract another year
Ex-Post Office CEO Paula Vennells walked away from IT scandal with over £400,000 in pay and bonuses
Prime minister says people should be held to account for Post Office IT scandal
Post Office was told in 2013 that Fujitsu witness was unreliable in subpostmaster prosecutions