‘Our worst day’: The untold story of the Electoral Commission cyber attack
As head of digital at The Electoral Commission, Andrew Simpson’s mettle was tested when threat actors gained access to the regulator’s email systems and accessed sensitive voter data. Three years on, he tells his story to Computer Weekly
Your worst day can begin so innocuously – you leave home, you stop to pick up your coffee order, you catch your train, or maybe you run for it and just miss it. Perhaps it’s raining. Such minor details make up the patchwork of our lives.
In Andrew Simpson’s case, he should have been celebrating a small win, a milestone in an ongoing – and by-and-large successful – roll-out of a cloud upgrade project. Then things fell apart.
Simpson joined The Electoral Commission – the UK’s election oversight and political finance regulator – in June 2022 as head of digital, information, technology and facilities, to lead a wide-ranging digital transformation project which, alongside transitioning from on-prem to cloud, brought a plethora of cyber upgrades.
But unknown to Simpson or anybody else, threat actors – possibly Chinese state cyber spooks, or a ransomware gang, or both – were already lurking within the Electoral Commission’s systems. Ultimately, it emerged that they exploited the ProxyShell vulnerability chain on an unpatched server to gain access.
The investigation later found the series of breaches started in August 2021, but it wasn’t until one of Simpson’s cloud transition projects was in progress that it came to light.
“Part of that was to introduce MFA [multifactor authentication], and that happened in October 2022, which is exactly when we found the compromise,” says Simpson. “One of the lead engineers on the project spotted that they had 10 attempts on their MFA account within less than a minute. It was glaringly obvious that something wasn’t quite right at that point.”
It turned out that in introducing MFA, Simpson’s team had “unintentionally” locked their attacker(s) out of the system and the threat actor was now trying to get back in.
As an IT leader, what does it feel like to be doing the right thing and to suddenly find yourself embroiled in a major cyber security panic?
“It’s possibly the worst feeling you can ever have in this industry,” says Simpson, who remarks that bringing new tech functions to an organisation’s workforce and helping them do their job better with up-to-date tools is ordinarily a great feeling.
“When you suddenly get hit with a cyber incident, you realise everything we were doing is no longer the priority, so the benefits of what we were doing get destroyed by the compromise, and your mindset changes – we now have to batten down the hatches again.”
Fortunately, the fact that the team had stood up MFA successfully was a small mercy and The Electoral Commission leaned into this, increasing the frequency of challenges – once an hour in the case of its lead IT engineers.
But Simpson still recalls the initial shock, and the dawning realisation that the scale of the compromise was much greater than it appeared. “It’s a horrible thing, it’s gut-wrenching – I think that’s the best way of putting it. I would never wish it on anyone,” he says.
First responders
In an ideal world, Simpson says he would have stood up an incident response team right away, but that wasn’t really an option at the time because the capacity wasn’t there.
He recalls frantic phone calls to contacts at suppliers and the National Cyber Security Centre (NCSC), which helped link The Electoral Commission up with incident responders at Secureworks (now part of Sophos) via its cyber security framework.
Meanwhile, the IT team moved swiftly to lock things down, taking the affected servers offline entirely and sandboxing them. This was highly disruptive, but because the Electoral Commission had one foot in the cloud already, there were still some systems that could be used relatively safely, subject to extra precautions to avoid cross-contamination.
One of the key things as well is that none of this was via email. It was all verbal, phone calls, because obviously they had access to our email system
Andrew Simpson, The Electoral Commission
Overall, says Simpson, The Electoral Commission was lucky. “We caught them working on tooling up and potentially at some point injecting ransomware. We were never at the point where a lot of organisations have ransomware rip through them and destroy them,” he says. “We didn’t get to that stage because we reacted so quickly. We didn’t give them an opportunity. They lost access with immediate effect.”
With Secureworks’ help, Simpson and his team started tracking down the initial compromise. “Very quickly they identified patient zero, which was an on-premise email server, and they did spot some traces of ransomware on that server as well,” he says.
At this point – almost 12 months before news of the hack broke in the media, everything was being done with the utmost secrecy, with the IT team on lockdown.
“No one else in the Electoral Commission knew what we were doing. We did not communicate that out. One of the key things as well is that none of this was via email. It was all verbal, phone calls, because obviously they had access to our email system,” says Simpson. “From the IT perspective, we knew nobody was to discuss this other than my boss, the CEO and executive team members. They were all who knew about what was going on.
“Obviously staff had issues where they were MFA-challenged every day, but I think a lot of people thought that was part of the process of going through the migration. That’s why I say it’s so important we didn’t get hit by ransomware, because staff did not see the disruption – but internally we were dealing with some real issues that we couldn’t talk about at the time,” he adds.
The lockdown process was very effective at keeping the incident from blowing up on a national scale until things were under control and news of the incident did not break until the following August. By then, the Electoral Commission was able to manage the narrative and explain the incident on its own terms, rather then having to engage crisis PR.
Unlike in many other similar incidents when systems are pulled offline in a hurry and outsiders notice an impact, such as the Marks & Spencer attack, it could be speculated that the Electoral Commission benefited from being an organisation that spends a lot of time out of the public eye.
Data crisis
But PR or no PR, there was undoubtedly a crisis. The Electoral Commission has multiple responsibilities in overseeing the UK political process that require it to collect and hold sensitive data on members of the public. It became apparent early on in the investigation that this data was at risk.
“In terms of the dataset we held, it was on what was known as the X Server at the time, and that was the electoral register, with a set of a copy of all the data that comes in from local authorities, so it wasn’t live data, it was a copy,” says Simpson. “[But] that was the key concern, and they did have access to that server. They also had access to our emails.”
Unfortunately, because the system was undergoing upgrades and its firewalls did not have the capacity at the time to hold old logs, it was never possible to prove or disprove if the data was exfiltrated. In the interests of doing the right thing, and regulatory compliance, the Electoral Commission was as upfront as it could be when it came to disclosing this to the public.
“That’s why when you speak to the NCSC and the ICO [Information Commissioner’s Office], you have to say it’s that way in terms of there’s a compromise and they had access to this data. That’s why we took the line we did. We can’t individually contact everyone on that list, [so] you have to have a public announcement,” says Simpson.
Changing the narrative
Thanks to an unnamed whistleblower, it also emerged in September 2023 that the organisation had failed an NCSC Cyber Essentials audit, as Computer Weekly and many other national news outlets reported at the time. We now know this is not the whole truth – the audit never took place because it was obvious to all concerned that the Electoral Commission would fail – a fact the record should now reflect.
“We had things like out-of-date software on laptops and the mobile phones weren’t quite up to date. We weren’t ready to be Cyber Essentials accredited at the time,” says Simpson, who had been scoping out potential improvements to fix these issues and attain certification when the intrusion was discovered.
When that story came out, he recalls taking his kids to Alton Towers and can even remember the ride he was getting on when his phone rang: “These are the things I think people don’t think about. Your life changes in terms of these impacts. They’ll never go away from me – I know where I was when I learnt certain things, every bit of it. It’s scar tissue, but it’s great because you take the learnings, you can’t look at the negatives.”
A pathway to resilience
Three years on, and with the cyber attack in the rearview mirror, The Electoral Commission has made great strides towards improving its cyber security posture.
“My business model is first-line support is internal, second-line support is expert vendors – particularly in this industry, you can’t have enough staff to deal with this,” says Simpson.
In terms of internal support, the first step was to train up the Electoral Commission’s IT teams on the product set that they needed to support – which would have been a core goal even without the cyber attack but was ramped up in the wake of the incident.
I’m speaking across the board to people wherever I can because the only way to help with this is to share information
Andrew Simpson, The Electoral Commission
Simpson then backed up this first line of defence with the introduction of a managed security operations centre (SOC) run through Secureworks, which he says made sense to do because, thanks to its work on the incident response process, it was well-embedded in the organisation’s tech stack.
Through the SOC, Secureworks is now running 24/7 monitoring, extended detection and response (XDR), vulnerability management, and high- and critical-level incident reporting in, with leadership on call day and night if needed.
But Simpson also believes that it’s important for an organisation not to have all its eggs in one basket with one supplier accounting for all its security needs, so on that basis another company is supporting the organisation on Microsoft Defender.
The Electoral Commission has taken steps to address email security, improving its DMARC compliance across the organisation from 40% at the time of the incident to 100% today.
There is also now certificate monitoring in place. “That’s a key thing I think people forget about,” says Simpson, “it’s easy for a certificate to expire, and that creates a vulnerability.”
The other key change has been the introduction of new firewalls to replace those that had let down the investigators. Working closely with Fortinet, the Electoral Commission has introduced a total of eight managed firewalls across its physical sites and its Azure tenancy, with data from them ingested back into the SOC.
“We have a Venn diagram of overlap that means every aspect of our security is protected by more than one vendor, we’re not depending on any one of them, so if any of those cannot deliver, someone else will be able to pick it up,” says Simpson. “That has been a massive change.”
For the organisation’s rank and file workforce, there is now additional security training in place, as well as enhanced password policies. Looking back, Simpson says it’s important not to scrimp on developing and training staff.
From breach to resilience
IT leaders can’t afford to wait for a breach to take place, says Andrew Simpson, instead, they should be investing proactively and engaging with expert partners. Read more here.
“You can spend a fortune on vendors, and some people do, but think about those key staff, not just the IT staff but the actual staff, making sure that they’re aware of anything that can happen, and making sure people are trained up on the technology they have in front of them as well. That’s one of the key learnings,” he says.
The Electoral Commission has since aced its audit and is now Cyber Essentials Plus certified, a demonstrable vote of confidence in its abilities.
“When you look at where we are now compared to where we were, people should be more confident in the way that we handle things. I know that we are much more professional in this way,” says Simpson.
But Simpson isn’t putting his feet up at this point. For example, when former prime minister, Rishi Sunak, announced a General Election on 22 May 2024, the Electoral Commission saw around 64,000 attempts on its systems – most of them crude phishing or password-spraying attacks – and blocked every single one.
Learning process
Overall, one thing is clear, cyber security is a process of continuous improvement. “We will never see a time when this drops off, it’s just part of the game,” says Simpson. “I’ve been in IT for 25 years, there was almost no internet when I started. Now everything is internet-ready [and] could be compromised, so I don’t turn a blind eye to anything.
“I don’t feel overwhelmed. I feel like we have put in place everything we can, but what you cannot be is blasé about it. In every project you do, there needs to be a security aspect, even if that involves an internet-ready fridge…. It can feel overwhelming, but just make sure its ingrained in everything you do.”
In the spring of 2024, the British Library, which fell victim to a cyber attack of its own in 2023, published an extensive rundown of what had happened to it and what it was doing to recover, in the service of helping others to understand, prepare for and hopefully withstand cyber attacks.
Simpson’s goals in speaking out now are of a similar nature and reflect a growing understanding in the cyber security community that transparency benefits everybody. He is becoming an advocate for doing security openly and, crucially, without blame or shame.
“I’m speaking across the board to people wherever I can,” he says, “because the only way to help with this is to share information. For those people who have been through it – [after all,] some people lose their jobs for this – I was lucky.”
Computer Weekly coverage of the Electoral Commission cyber attack
An unknown threat actor who attacked the UK’s Electoral Commission had access to data on millions of UK voters for over a year, the watchdog has revealed.
The Electoral Commission failed an NCSC Cyber Essentials audit on multiple counts at about the same time as cyber criminals breached its systems in 2021, it has emerged.
UK government identifies Chinese state-linked hackers as likely to have been behind attack on the Electoral Commission.
Read more on Data breach incident management and recovery