BT fined £17.5m for ‘catastrophic’ emergency network outage

After undertaking an investigation following a UK-wide disruption to emergency call services on 25 June 2023, UK communications regulator Ofcom has handed BT a fine of £17.5m, and slammed the UK’s leading telco for being ill-prepared to respond to what it described as a catastrophic failure of its Emergency Call Handling Services (ECHS) infrastructure. 

Ofcom condemned BT for falling “woefully short” of its responsibilities, adding that it had inadequate preparedness to deal with such a large-scale outage, putting its customers at unacceptable risk.

On 25 June 2023, BT experienced a network fault that affected its ability to connect calls to emergency services between 06:24 and 16:56. During the incident, nearly 14,000 call attempts – from 12,392 different callers – were unsuccessful. Those trying to use the 999 emergency telephone number were unable to connect with the services they required due to the technical issue.

Among those first responders affected were Cheshire Fire and Rescue Service, which warned of a 30-second delay to connect to 999 while the incident was ongoing, and Suffolk Police, which said its system was not working to full capacity. The Metropolitan Police and Bedfordshire Police were also said to have reported difficulties.

Issues have persisted even after BT switched to a backup system, and it reportedly took BT nearly three hours to alert UK government ministers to the problems it was experiencing.

UK law requires the country’s network operators to take appropriate steps to prepare for potential outages with BT connecting 999 and 112 emergency voice calls and providing relay services for deaf and speech-impaired people.

The incident caused disruption to text relay calls, which meant people with hearing and speech difficulties were unable to make any calls, including to friends, family, businesses and services. This left deaf and speech-impaired users at increased risk of harm.

Ofcom noted that its rules require BT and other providers to take all necessary measures to ensure uninterrupted access to emergency organisations as part of any call services offered. As required by law, BT notified Ofcom of this issue, and on 28 June 2023, Ofcom opened an investigation to establish whether the company had failed to comply with its legal duties.

Ofcom’s investigation sought to establish the facts surrounding the incident and examine whether there are reasonable grounds to believe BT has failed to comply with its regulatory obligations.

Specifically, Ofcom opened its investigation into BT’s compliance with General Condition A3.2 (GC A3.2) and sections 105A and 105C of the Communications Act 2003 following BT’s notification of the technical fault. GC A3.2 in particular requires certain communications providers to take all necessary measures to ensure the fullest possible availability of voice and internet services provided over public electronic communications networks in the event of catastrophic network breakdown or in cases of force majeure. It also calls for uninterrupted access to emergency organisations as part of any voice services offered.

Ofcom’s investigation has pinpointed three key stages to the incident. In Phase 1, from 06:24 to 07:33, BT’s emergency call handling system was disrupted by what was later found to be a configuration error in a file on its server. This resulted in call handling agents’ systems restarting as soon as a call was received; agents being logged out of the system; calls being disconnected or dropped upon transfer to the emergency authorities; and calls being put back in the queue. Ofcom noted BT was initially unable to determine the cause of the issue and attempted to switch to its disaster recovery platform.

In Phase 2, from 07:33 to 08:50, the first attempt to switch to the disaster recovery platform was unsuccessful due to human error. This, said Ofcom, was a result of “instructions being poorly documented”, and the team being unfamiliar with the process. The incident grew from affecting some calls to a total outage of the system.

In Phase 3, from 08:50 to 16:56, the rate of unsuccessful calls decreased once traffic was migrated successfully to the disaster recovery platform. Usual service was not fully restored initially as the disaster recovery platform was found to have struggled with demand.