The Health and Social Care Information Centre (HSCIC) recently announced it had successfully upgraded the infrastructure behind the NHS Spine platform.
Thousands of care organisations, health professionals and patients in England rely on the NHS Spine, which is fondly known as the "electronic backbone of the NHS" and is considered part of the critical national infrastructure.
The system transfers 400 million electronic messages per month and has to be available 24/7, so the final steps in the two-year process to upgrade to Spine 2 had to be completed during an off-peak period to a very precise script.
What makes the upgrade even more significant is that it was the first major project to be conducted internally with a development partner, rather than outsourced to a large supplier.
HSCIC's director of operations and assurance services, Rob Shaw, told Computer Weekly the decision to step away from its existing provider – BT – was partly to keep in line with the government's move away from large, locked-down IT contracts in favour of smaller suppliers.
Internal expertise used to rebuild NHS backbone
But to internally deliver an update to a critical national programme of such complexity was not an easy undertaking.
“Apart from BT, which over the years was a good provider, no one knew how Spine works better than we did,” said Shaw.
NHS Spine 2 capabilities
• Spine 2 supports more than 900,000 registered users, with more than 220,000 users each day.
• At peak times, Spine 2 supports around 1,500 messages per second.
• Spine 2 delivers an 80% reduction in average response times compared with the previous system.
• Spine 2 safely stores and provides secure access to around 500 million records and documents.
• The system supports more than 100 different functional messages.
Because the Spine system joins up a number of different types of care with multiple different systems, the HSCIC became the systems integrator by default, said Shaw. The organisation decided to create a proof of concept to build Spine 2 in-house.
The HSCIC ran a competitive tender to find its development partner – the SME BJSS – on G-Cloud. It then got approval from the NHS and the Cabinet Office to go ahead with an internal development using open-source tools from multiple suppliers.
Built in-house by the HSCIC and BJSS, Spine 2 has been supported by a number of smaller specialist firms, including open-source expert Basho, to create a more flexible system.
“The decision to procure commodity hardware and open-source products was something we wanted to do,” said Shaw. “The products available today are as compelling as a traditional product used previously, and they have the same outcomes and significant reduction in cost.”
While there is always a desire to drive down costs in the public sector, Shaw emphasised that the most important thing was for the HSCIC to provide a system which was “secure, resilient and available”.
“Open source was something we utilised, but it wasn’t a main driver,” he added.
Thorough testing for smooth deployment
Shaw said the team designed the new system to cut out as much complexity as possible.
During the upgrade, it also had to ensure there was the absolute minimum – if any – impact to Spine users. To do this, the teams spent 12 months on integration testing and 18 months on transition testing.
Apart from BT, no one knew how Spine works better than we did
Rob Shaw, HSCIC
During the final upgrade weekend, the team was able to avoid major downtimes for Spine users, which during that off-peak time still included e-prescriptions, hospitals, out-of-hours surgeries, some ambulance services and NHS 111.
It phased the transition in a way that was best for each user group and, at worst, there were only brief downtimes of an hour for e-prescriptions and at most times a read-only version of Spine was available.
To do this, the HSCIC and BJSS developed a detailed plan organised into 10-minute intervals. The upgrade began at 7pm on the Friday evening of the August Bank Holiday weekend and ended around 10pm on Monday night.
“Every 10 minutes we knew exactly what was going to happen,” said Shaw. “We were only five minutes out by the end.”
But it was immensely difficult to make sure everyone knew what was happening at what time, according to Shaw. “It was a brand new approach for the HSCIC and it meant we were able to meet the needs of the users,” he said.
Hard work pays off
The most difficult part of the project, Shaw said, was making sure that the half a billion records from a diverse range of sources were moved safely and successfully.
Read more on the HSCIC
- NHS Spine infrastructure transition successful, says HSCIC
- HSCIC tightens controls over patient data following data lapse
- HSCIC and MOJ first to procure on government managed email framework
- HSCIC announces four steps to transparency
- HSCIC appoints Andy Williams as chief executive
- The NHS is dependent on shared data, says HSCIC
The data transition was carried out over a long period, working one batch of data at a time and working closely with previous provider BT to make sure there was no data loss.
“We had to keep the existing spine and new spine in tandem – that’s why we did so much transition rehearsal,” he said.
The HSCIC and BJSS put together a "red team" of 12 technical experts who were on standby if there were any issues. And although Spine 2 is now up and running, the teams will continue to monitor it 24/7 for 45 days post-transition to ensure it performs as expected.
Throughout the process, Shaw said the staff went above and beyond to deliver the system on time and securely.
“It’s not uncommon for people to work through the night to get new functionality in,” he said, which has brought a culture change to the organisation. “Over bank holiday weekend, people were putting in 20-30 hour shifts – not because they had to, but because they were proud to do it.”
The Spine platform is now seeing a higher throughput than ever before. In one week alone, Spine 2 handled 1,500 transactions per second.
HSCIC has big plans for the future
The update of Spine is just the first step in a move by the HSCIC to become a provider of services.
The organisation already looks after NHS Choices and delivers services such as cancer screenings but, from a technical perspective, Spine was proof that the provider could take on more complex projects.
In the future, it will be working on identity and access management, and the replacement for Choose and Book.
To support this undertaking it needs to build out the capability of its workforce and recruit highly skilled developers and operations staff. But this is easier said than done in an age where the UK is facing a substantial skills gap.
“We’ve had challenges hiring skilled people. Being based in Leeds, competition is high because it’s a vibrant city and within commuting distance of Manchester and London,” said Shaw.
“Demand outstrips availability at the moment, and we need to make sure people see we’re doing far more through technical delivery.”