1xpert - Fotolia
During a presentation at Chef’s first major ChefCon event outside the US, Lloyds Banking Group’s DevOps chief discussed the cultural challenges in supporting fast, secure software development.
Mark Howell, head of DevOps at Lloyds Banking Group, describes how, for years, the retail bank’s internet platform had separate developer and operations teams.
“We had developers who wrote and tested code, that was then thrown over to the operations team,” he says. “Operations don’t like change and they will criticise code.”
A few years back, when Lloyds began looking into DevOps, the IT operations team for the internet and mobile banking platforms had a five-day service level agreement on accepting code changes into the production environments.
Given banking is a regulated industry and that any change could impact millions of users of the mobile or internet banking platforms, the IT operations team needed the time to ensure that any code pushed into the production environments passed the required security, stability and compliance requirements of the business
Separation of the two teams meant developers did not understand why the delays were necessary.
Software-powered business strategy
There are few businesses that are not on some kind of digitisation strategy. Businesses of all sizes recognise they can gain a competitive edge and evolve their business by being able to develop new software-enabled products and services quickly, and have these available to customers with the least delay.
In the architecture promoted by Chef in its DevOps toolset, there is a so-called pipeline that ensures every change in code or IT infrastructure software is checked against the whole software stack.
Such an approach uses automation for testing and deploying the patch or the new software functionality into production environments. Security and compliance are considered first-class citizens, right at the front of the DevOps process, rather than at the final stage in IT operations, to ensure software can be moved into production as quickly as possible. Success relies on the developer and operations teams working closely together.
Howell found that having a DevOps team act as a middleman between the software engineering team and the IT operations team created an unnecessary bottleneck in the route between software development and live deployment.
At Lloyds, Howell says the technical implementation of DevOps is the easy part. “The difficult part is seeding change when I have 19,000 colleagues [across Lloyds Banking Group]. It is about changing mindsets, challenging audits and how you recognise and empower people to make decisions.”
Changing the mindset
Among the approaches he recommended delegates at the Chef event consider was to challenge the myths that perpetuate through an organisation. These result in a culture of people automatically saying “no” without checking whether a request is allowed under existing compliance rules or regulations.
“Have a sensible conversation and it will generally get accepted,” he says. “We have to be able to trust [software] engineers to do the right thing.”
From a compliance and security perspective, while humans do make mistakes: “That’s where automation helps. Fundamentally, it’s about empowering the team,” says Howell.
Lloyds established a centre of excellence for DevOps in 2017, which Howell leads. Its role is to evangelise the use of DevOps throughout the bank. “On a day-by-day basis, I can’t influence 19,000 people,” he says.
Rather than the centre of excellence being the place to execute DevOps on behalf of the rest of the business, the focus is strategic, looking at branding and a vision. “What does DevOps mean for Lloyds? How do we move safely with velocity in a banking context where there are 50 million customers?” says Howell.
Read more about DevOps
- There are many reasons to adopt DevOps, and there are many challenges involved in getting it right. But don’t buy into the culture myth that many DevOps evangelists are selling.
- Automated testing remains the key to continuous development and integration. How can an enterprise embrace DevOps culture without embracing DevOps automation?
There is a small amount of consulting and training, such as showing people across the business how to use enterprise source control in DevOps or how DevOps can be used to manage IT infrastructure, but the primary role of Lloyds Bank’s DevOps centre of excellence is to give guidance and offer an informed point of view.
“We bring communities together to share the great stories [that show others] what they are doing, as well as help each other out,” he says.
One way this is achieved is by organising externally hosted events. “We get 100+ colleagues, run an event and have guest speakers,” says Howell. “People then have a platform to share their stories from. For instance, our public cloud team will tell us what’s on the roadmap. This all helps to glue the community together.”
Howell says the team has also spent time with business stakeholders to ensure decision makers across the business appreciate what the centre of excellence wants to achieve as part of the bank’s broader business strategy.
“I genuinely had a half-hour conversation with the head of finance about what a container is all about,” he says. “We are building those relationships with the stakeholders – risk, audit, finance – these are the people who can really help you.”