Sainsbury’s Bank moves data to AWS cloud with Teradata UDA

Bank decided on a fundamental shift to AWS cloud, using Teradata’s Unified Data Architecture, after it was divested from Lloyds in 2013. Here’s how it built from a new ground zero.

Lindsay Clark

Published: 03 Jun 2019

Trying to change the engine of an airborne jumbo jet full of passengers while halfway across the Atlantic may not be advisable, but, metaphorically at least, Sainsbury’s Bank faced what could be said to be a similar challenge, says Greg Watson, its head of architecture and analysis.

When the company decided to lift its banking and data warehouse systems to the cloud, it had to do so without any of its customers or internal users noticing.

“There would be no detriment to them, we didn’t want them to notice we were fundamentally changing the banking platform under their feet,” Watson told delegates at the Teradata Universe conference in Madrid earlier this year.

The mission was completed on 14 March 2019, when the bank transferred the final part of its data capability from a legacy infrastructure to Amazon Web Services (AWS) – but it had started in 2014.

Sainsbury’s Bank was launched in 1997 as a joint venture with the Bank of Scotland, which became Halifax Bank of Scotland in 2001, which became part of the Lloyds Banking Group in 2009. In 2013, Lloyds decided to divest the joint venture and Sainsbury’s bought it out.

“This was starting a new bank from the ground up – the people and the process, everything,” says Watson. “We had to safely transfer all our products and services from the Lloyds legacy estate onto a range of new technology and banking platforms, including mortgages, credit cards, loans, savings and insurance.”

In creating a new data platform, the bank wanted to take advantage of group-wide data, which includes retailers Habitat and Argos, as well as the Nectar loyalty card scheme, which includes Eurostar, Pizza Hut, eBay, Expedia and Caffè Nero.

“As a principle, we decided very early on that anything we could treat as a commodity, that someone could already do better than us, we would seek to outsource as a managed service,” says Watson. “Anything where we could differentiate, we would seek to own ourselves. That does not just apply to our applications, but to our infrastructure estate.

“We don’t have our own datacentre – we use the parent company for some of the commodity datacentre activity, such as email, collaboration, finance and HR, but for those core banking systems, it was in someone else’s datacentre.”

But Sainsbury’s Bank did not want to move onto a new infrastructure with the same capabilities. Watson says the team wanted to take away some of the technical constraints the data community had been working with for the lifetime of the bank.

“We didn’t want customers to notice we were fundamentally changing the banking platform under their feet”

Greg Watson, Sainsbury’s Bank

“They had been battling against the technology to do their jobs,” he says. “We had people with multiple laptops, accessing data on different estates and with dedicated network cables under their desks connecting into the Lloyds environment.

“We wanted to build a data and analytics capability that really leveraged group data and, in particular, gave us that data advantage through Nectar.”

At the same time, the new platform had to comply with established regulatory processes and controls. The team also had to train its technical staff in the new tooling and safely dispose of customer data from the old system. Meanwhile, the clock was ticking – the transfer had to be complete before its service agreement with Lloyds expired.

When Sainsbury’s Bank de-merged, the plan was to take its data platform from on-premise to a platform hosted by a service partner. But the arrival of a chief data officer, Andrew Day, three years ago brought forward the decision to create more expansive data capabilities and plan for the future.

Its approach is based on the Teradata Unified Data Architecture, which encompasses both a distributed Hadoop cluster as well as the supplier’s relational database. It can manage both real-time and batch ingestion of data.

For the first port of call, the architecture feeds all data into Hadoop, but the highest-value “smart data” is stored, modelled and looked after within the Teradata Enterprise Data Warehouse to support strong governance and efficiency in data science, says Watson.

Other financial services organisations, such as Swedbank, are taking a similarly hybrid approach to their enterprise data architecture and are developing business teams to support new data initiatives (see box below).

Pods and squads support Swedbank’s hybrid data shift

Swedbank, one of the primary banks in Sweden and the Baltic region, knew it had to respond to new customer behaviour and competition in the market. As well as a new data and analytics platform, the response required a different approach to hiring and managing people, says Sead Pašalić, head of business intelligence (BI) solution architecture and data discovery.

“When I went to HR and said we needed to recruit for the role of the data scientist, we spent time explaining what that was and why we needed that category of person,” he says. “We are doing the same thing again with the data engineer role. Organisations need to adopt a new set-up, one ready for agility.”

As a result, Swedbank changed its organisational approach to exploiting data. This includes “pods” of hybrid cross-functional teams that can deliver a new product and maintain it. There are also “squads”, which include people from across the business who work towards specific projects, such as anti-money-laundering, in an agile way.

The mission to create a new approach to using data was driven by business challenges, including new customer needs and new competitors in the market, says Pašalić. “Fintech is not there to kill us, but it will make us bleed.”

To respond to these challenges, Swedbank is looking to improve its data discovery, drawing on a wider variety of sources and performing analytics in less time, he says. It decided to extend its current Teradata enterprise data warehouse environment with Teradata’s Aster analytics, and Hadoop distributed data environment for advanced analytics. Like Sainsbury’s Bank, it is based on the Teradata Unified Data Architecture.

Sainsbury’s Bank’s new system uses a range of Amazon cloud services: Elastic Compute Cloud for compute, S3 for object storage, EBS for block storage, as well as AWS and partnered marketplace services.

“When we made the decision to go to the cloud, it was an all-in decision,” says Watson. “We wanted to be able to leverage the benefits it brings us in facilitating new ways of working.”

While much of the UK banking industry remains uncertain about the suitability for cloud platforms for banking and its data analytics needs, Watson says Sainsbury’s Bank’s philosophy, as much as its technical approach, contributed to its success.

“You hear stories of chief risk officers horrified that their data might be in someone else’s datacentre and chief security officers not wanting to relinquish control, so why did we think the cloud was OK for us?” he says.

“Our IT department was formed five years ago to be focused on management and oversight of outsourced IT. If you apply the same policies and the same controls, if you treat vendors in the same way, and you do the same assurance, the cloud is just another form of outsourcing. That has gone down well with regulators in the UK and has allowed us to be quite permissive in our use of cloud services.”

But the transition was not without its difficulties. Watson shares his top dos and don’ts (see box below) for any banks or other highly regulated businesses wanting to embark on the same journey to the cloud.

Sainsbury’s Bank’s head of architecture and analysis Greg Watson’s dos and don’ts when moving to the cloud

Do: Identify core architectural patterns. We chose not to solve the immediate problems under our noses, but looked at reusable capabilities we were going to have to put in place. We used The Open Group Architecture Framework (TOGAF).

Don’t: Run too fast. Take your time, with auto-scaling and auto-healing part of the overall approach. We probably ended up building the wrong things too quickly and having to cycle back.

Do: Automate everything you can to death. In combination with architecture patterns, automation has been a very good story for us.

Don’t: Rely on platform-specific building tools. We are an AWS shop, but we very deliberately don’t use its tools. Instead, we used tools like Ansible and Terraform, which helped demonstrate an exit strategy from our cloud provider that the regulator required.

Do: Use partners to bolster capability. Cloud people are still hard to find. Many in financial services are starting their own cloud journey, so there is a huge demand for those skills.

Don’t: Think about security too late. We did in the first iteration of our data programme. Now we have security in squads and teams from day one.

Do: Reap benefit from cloud. With subscription-based licences, it is a great way to try things and experiment. It’s not like the old days when you bought hardware and installed applications. The cloud fundamentally changed the way we do architecture. We design by doing.

Matt Aslett, vice-president for data, artificial intelligence (AI) and analytics at 451 Research , says firms planning to move their data to the cloud do not plan to replace like with like. “It is not just lift and shift,” he says. “More often, the next version of the application is being developed in the cloud. They want to repurpose and redesign their data and analytics architecture.”

Such a complex project may require outside business events to provide enough impetus, says Aslett. “Generally speaking, there needs to be a larger business imperative to change – company-wide digital transformation or M&As [mergers and acquisitions]. Cost saving might be part of it, but often it is a larger business driver that is the trigger.”

Aslett says researchers are seeing greater adoption of cloud in financial services, with firm plans in place to move data management and analytics within two years. If 451’s predictions are correct, many in financial services will be attempting complex in-flight re-engineering and hoping they do not plunge into the ocean below.