Botched software update to blame for Post Office Horizon crash

A fault during a scheduled software update caused a UK-wide crash of the Post Office’s controversial Horizon IT system earlier this month.

On 5 November, Post Office branches were unable to do business when the Horizon retail and accounting system crashed. The Post Office blamed Fujitsu for the outage, which it said lasted 90 minutes. Some subpostmasters reported being offline for much longer.

When asked by Computer Weekly for an explanation in the immediate aftermath, Fujitsu hid behind customer confidentiality and a spokesman said: “We are not in a position to comment on anything.”

The Post Office was then criticised for giving subpostmasters a vague explanation for the problem, which experts described as “pointless”. The Post Office told subpostmasters: “It was a memory issue in devices that support and deliver the overall network across the country, which caused the network to fail for around 90 minutes.”

Computer Weekly contacted the Post Office for a more detailed explanation and was told a planned firmware update had caused the problem.

The Post Office told Computer Weekly: “Following a scheduled change to install a firmware update on a network appliance, a fault was detected at 10:03 on Thursday 5 November, which resulted in a network outage. The firmware was backed out and the service was fully restored at 11:29.

“We apologise to all postmasters and customers affected and we are identifying additional steps to ensure this problem doesn’t happen again.”

Post Office has not explained whether the update was performed by Fujitsu or its own staff.

One IT expert commented: “I don’t think the memory comment makes sense other than the firmware update is software that would have been loaded into a device and stored in some kind of memory. Backing out the firmware suggests the memory itself is OK and the new firmware was the problem.”

The Horizon system, used in more than 11,000 Post Office branches, is at the centre of huge scandal.

A Computer Weekly investigation in 2009 revealed that subpostmasters, who run Post Office branches, were being blamed for unexplained financial losses, which they claimed were caused by errors made by the Horizon system. The Post Office denied this, and many subpostmasters were subsequently prosecuted for theft and false accounting, with prison sentences, community service, criminal records and heavy fines among the injustices they suffered as a result. It has become one of the biggest miscarriages of justice in UK history (see timeline below).

Part of the problem was the Post Office’s insistence that Horizon was robust and its denial that Horizon errors were causing unexplained accounting shortfalls. In the multimillion-pound Horizon group litigation, in which subpostmasters were proved right that Horizon was causing shortfalls, High Court judge Peter Fraser described the Post Office’s stance as “amounting to the 21st century equivalent of maintaining that the Earth is flat”.

Referring to the latest Horizon problem, one industry source said: “The good news, for subpostmasters, is that these sorts of network-wide interruptions are so obvious that any resultant shortfalls are likely to be relatively easy to notice and also more likely to elicit a sympathetic response from the Post Office. Contrast that with the scepticism and even hostility with which branch reports of one-off mysterious shortfalls have been met by the Post Office in the past.”