Post Office scandal: What is a systemic error?

“That’s just semantics!” How often have you heard that complaint? It can be justified if someone is guilty of tedious pedantry intended to distract from the real issue. But semantics are about the meaning of words, and meaning is important. When we are dealing with complex technical matters then semantics is vital.

If different people assign different meanings to key words then the result is confusion, chaos and perhaps even human tragedy. We’ve seen that with the Post Office scandal. There was endless confusion over the years about the meaning of an eight-letter word – systemic.

Forensic accountants Second Sight were commissioned in 2012 to investigate Horizon for “systemic defects”. Second Sight clarified that this meant the standard dictionary definition, i.e. defects relating to all aspects of the wider system, including operational processes and training.

This is also consistent with safety engineering, which recognises that causes of accidents fall into three broad categories: direct, contributory and systemic. An example of a direct factor would be the crew of the Herald of Free Enterprise failing to shut the doors before the ferry set sail. Contributory factors to that disaster were the weather, tide conditions and the ferry design. Systemic factors included the corporate culture and management pressure on the crew to stick to timetables.

In computing, a direct cause of a problem might be a coding error, or a user entering erroneous or unexpected data. A contributory factor would be a misunderstanding of a business requirement, or a missing control in the design. Systemic factors might be inadequate user training, an incompetent helpdesk, or an insistence that systems must be developed and launched to meet aggressive deadlines. I don’t think that coding errors are systemic issues in themselves, but they might well be evidence of underlying systemic issues.

You can’t provide a meaningful investigation, or audit, of a complex software system unless you understand what might go wrong at different levels – technical system, procedural, management and cultural. That’s what Second Sight was commissioned to do – a systemic investigation.

However, confusion quickly took over among Post Office management. The correspondence that has emerged at the Post Office Horizon IT Inquiry shows that executives often assumed that a systemic issue meant a technical system issue. When they saw Second Sight’s interim report in 2013, Post Office executives seized the chance to confuse and mislead the rest of the world.

Second Sight reported that it had “so far found no evidence of system-wide (systemic) problems with the Horizon software” but that it was aware of two incidents where “bugs in the Horizon software gave rise to 76 branches being affected by incorrect balances or transactions”. The awkward 76 branches were quickly forgotten about.

From then on, Post Office executives insisted that Second Sight had concluded there were no problems caused by the system that could have led to wrongful convictions. If there was a problem at a particular branch, the corporation treated this as evidence of user error or dishonesty. The system was robust. Second Sight said so. The Post Office spouted this dishonest nonsense to the subpostmasters, Parliament and the press.

Scotland’s public prosecutors, the Crown Office, wanted to know the basis for the Post Office’s confidence in Horizon. The corporation told them that Second Sight’s report provided assurance that the system was robust and its evidence could be relied upon for criminal prosecutions. Misleading prosecutors in order to persuade them to launch prosecutions of entirely innocent people is a big deal. We will surely hear more about that.

When former CEO Paula Vennells appeared before the Post Office Horizon Inquiry last week, she couldn’t give any clear and consistent explanation of what “systemic” meant. Her witness statements and contemporary emails variously used the definition that misrepresented Second Sight’s report, the correct meaning of the word, and a disingenuous attempt to portray “systemic” as a technical IT term about which she could not have been expected to have an opinion. It wasn’t impressive. She was the CEO. It was her job to know.

On 23 May, Vennells endured a humiliating series of questions from Jason Beer KC which established that she didn’t understand “systemic”, that she never took the trouble to be clear what it meant, and that there were never any discussions between herself, senior executives and the board about what the word meant.

Nevertheless, for year after year, the Post Office insisted that Second Sight’s interim report that they had not yet identified any systemic issues meant the corporation had proof that there were no specific system problems that could affect individual branches under certain conditions, like a communications failure.

One of the most valuable contributions we can make when we work in large corporations is to be clear about words we use. “Systemic” is one of those words that confident and assertive managers like to spout to show they know what they’re talking about. Asking the simple question, “Sorry, but what exactly do you mean by that?” requires some courage, but it can save a lot of expense and grief in the long run.

Systemic is a useful word, but it’s one I always shy away from because I know that too many people assume it means what they want it to mean. It is often far more trouble than it is worth. I prefer to say “wider system” or “wider organisational” to make it clear I’m not talking only about the technical system.

Oh, and please don’t confuse systemic with systematic. A systematic error would be one that is consistent and reproducible rather than random.

The Post Office scandal might have been cleared up sooner, at far less expense, and with a lot less unnecessary pain and anguish, if the Post Office had been forced to be clear, consistent and honest about the meaning of crucial words. It’s not “just semantics”. Be willing to ask awkward questions! Be better than the Post Office – it’s not that hard!

James Christie is an independent consultant, an IT expert with extensive experience of software development, IT audit, testing and security management in the IT outsourcing industry going back to the 1980s.