luisrsphoto - Fotolia

Changing channels: How DevOps, shared infrastructure and smart hiring help ITV innovate

Tom Clark, head of Common Platform at ITV, explains how embracing agile thinking and shared infrastructure is helping the broadcaster stay one step ahead of its commercial competitors

Commercial television broadcasters are heavily reliant on advertising revenue to fund their programming, and their ability to wrest viewers away from streaming services and other digital distractions is largely determined by the quality of their output and how accessible it is.

The latter has seen many broadcasters invest in online catch-up services and platforms, so viewers have the option to watch their favourite shows wherever and whenever they want, rather than have their viewing habits dictated by the broadcast schedule.

That is certainly the case for integrated producer-television broadcaster ITV, which is in the midst of a multi-million-pound, multi-year modernisation programme, that began in early 2015.

The initiative is geared towards improving the efficiency of ITV operations, while ensuring it has the agility and flexibility in-house to meet the ever-changing digital demands of its viewers.

The motivation for launching the modernisation programme was the realisation that, having “pressed pause” on the in-house development of its digital products in 2010 in the name of cost-cutting, ITV was starting to fall behind its competitors.

In response to this realisation, the organisation embarked on a wide-reaching overhaul of how it meets the infrastructure needs of its business units, having outsourced this part of the business – in the wake of the 2010 efficiency drive – to a managed service provider (MSP).  

Fast-forward to 2017, and ITV now has a standardised, open source infrastructure that the whole company can use to host applications. It is called Common Platform.

The company comprises five divisions – ITV Studios, Commercial, Broadcast, Shared Services and Online – and each business unit now hosts at least one application or service in this setup, Tom Clark, head of Common Platform at ITV, tells Computer Weekly.

Ensure consistency

The platform is designed to ensure consistency across the five divisions, with regard to how infrastructure processes are carried out from a logging, monitoring, configuration management and deployment perspective, which, in turn, creates efficiencies.

“They [the business divisions] were doing all of this themselves before, but they never had time to do it properly,” says Clark. “They did it good enough, but it is the job of the core Common Platform team to do it brilliantly now because it underpins everything.”

Each division also has access to platform engineers who embed themselves in that department’s day-to-day processes and ensure they are ready to make the most of what the Common Platform has to offer them.

These “embedded engineers” do their initial training with the Core Team when they join the organisation, so they can learn how to use the Common Platform tooling and all its functionality in advance.

This means that when these individuals start work in the business division they have been assigned to, they are familiar with how ITV’s systems work and can get straight down to work.

“By putting them into the Core Team , I take the hit of a starter slowing my team, and then, when they join the product team, they hit the ground walking, rather than running – but at least they know what they’re doing,” says Clark.

“Also, when people leave, I am less stressed because I can move someone from the Core Team into the product team until we hire someone back. Nothing needs to slow down because someone leaves.”

Down with DevOps

DevOps principles have informed the design of the Common Platform since the start to ensure it can always be improved upon and its functionality tweaked in line with the changing needs of the business.

“The old platform was a big design and big build with no system for improvement, and one of our chartered goals for the one was that it had to have longevity,” says Clark.

To achieve this, part of the Core Team’s remit involves carrying out research and development into what users want from the platform, what products and applications need to be added to it, and how the offering, as a whole, stacks up against what its competitors are doing.

“We’ll then go away and add that into the platform to ensure it is always moving forward and that we’re adding technology to it, so that we’ll never be stuck in the past again,” he adds.

The adoption of the Common Platform across the business has not been an overnight development, says Clark, who took on his role overseeing its development two years ago, with just three engineers at his disposal.

“We’ll soon have 17 or 18 engineers, so the team has grown in that time, but it wasn’t like we had sudden investment and [remit] to ‘do it all now’,” he says.

“I had to prove it could work and slowly build up this competency, and now it’s become the default choice if you’re building an in-house developed system.”

Hire slow, fire fast

When it comes to hiring engineers for the Core Team, Clark says his employment criteria are fairly straightforward – he wants to work with people who are “smart and kind”, and adopts a “hire slow and fire fast” approach to finding them.

“Smart is hard to find, kind is hard to find and smart and kind is really hard to find, but I refuse just to put bums on seats,” he says.

“We make sure we pay them the industry average or above for the work they do, but it’s also about giving people interesting work and lots of autonomy.”

ITV also has a short reporting structure, which means there is not layer upon layer of middle management to win over when it comes to securing buy-in for DevOps, cloud and other technology initiatives, says Clark.

“We are one of the biggest companies in the UK, but the fact that my board director knows who I am and we can have a chat over coffee means the team has lots of influence over the direction of the company,” he says.

“That is very empowering because if they don’t like something, there isn’t tonnes of management standing in the way of fixing it.”

Common ground

The origins of ITV’s Common Platform began in 2014 within its Online division, when a colleague of Clark’s, Rob Taylor, was tasked with creating a version of the broadcaster’s catch-up service to run on Samsung internet-connected TVs.

At the time, the company – and the MSP to which it outsources its infrastructure requirements – was tightly wedded to the waterfall method of project delivery. Taylor, however, decided to adopt a DevOps-style approach to creating a minimum viable version of the app.

“He seconded one of the developers and a platform engineer, put them together and told them they had control to make their own decisions, and he set about requesting some virtual machines [VMs] from the MSP for the project,” says Clark, which is where the firm’s agile ambitions ran into trouble.

After requesting the VMs, Taylor was told to expect a wait of about four weeks for them, which prompted him to provision his own from cloud giant Amazon Web Services (AWS) instead.

“When you are trying to develop in an agile way, a four-week lead time for a VM is not compatible really,” says Clark.

The success of this relatively small project gave Online the confidence that DevOps, as a software delivery approach, could thrive elsewhere at ITV, along with the use of cloud. 

“From there, I was asked to take the minimum viable product that Rob built as inspiration to build the hosting platform for the whole modernisation programme for ITV,” says Clark.

“There are 16 or so key lines of business systems that take billions of revenue through them, that run on this DevOps, open source Common Platform, and it basically underpins the whole of ITV.

“It started with the simple, low-risk stuff and building that up until we get to where we are today.”

That includes having services considered to be “broadcast-critical” running on the Common Platform, including its scheduling system, which dictates what programmes are shown when and on which channel.

“If the broadcast scheduling system breaks, ITV goes off air,” he says. “Not immediately, but within a few hours – but that runs on the Common Platform in AWS.”

Although the organisation has reached a point where mission-critical workloads can run on the Common Platform, that is not to say the modernisation programme is complete, says Clark.

“We have systems that still run Cobol, for example, like the one we use to pay Ant and Dec. We are replacing those with a brand- Java-based microservices stack running in the cloud. We need to get Cobol out of those systems because the people who write it are dying out – literally.”

A shift in process

The move away from waterfall to using agile and DevOps methodologies is a big change for any organisation, particularly for a FTSE 100 company like ITV, says Clark.

Adopting a “start small, think big” mentality was an essential part of this process, particularly when it came to getting buy-in from the powers that be at ITV that DevOps was the right way to go.

So, too, was building protective measures into the Common Platform that ensured that if one service using it went offline for any reason, none of the other applications running on it would be affected, reducing what Clark calls the blast radius.

“That allows us to move as quickly as we want, because if you’re a product owner of an [application or service] that has been ring-fenced away from all the others, and you want to roll out a release on Friday that you know is a bit risky, you can go ahead and do it,” he says.

“If you’re the product owner, you have responsibility for your availability and performance targets, and if you want to press the release button on Friday, you should be empowered to do so.”

Stepping out of the shadows

The decision to sidestep ITV’s MSP partner during the of the Samsung app should not be interpreted as a move to embrace shadow IT, stresses Clark.

In fact, the powers that be at ITV are happy for the Online division to “go rogue” every so often because of the reputation the team has for blazing a trail for the deployment of technologies across the company.

“It wasn’t shadow IT because it was known about and blessed,” says Clark. “Online are a bit special in that they are known to be the trailblazers, so they will often try out stuff, and do the R&D there and eventually the rest of the business will catch up.”

The Online division has a “fast-follower” attitude towards adopting emerging technologies, says Clark, and closely observes how other organisations have overcome the challenges of early adoption before trialling them in-house.

For example, Clark cites the division’s impending plans to start using Docker containers in its production environments.

“We weren’t running Docker in production two years ago, but we are looking to in the next quarter or so, now other people have suffered the early adoption pain,” he says.

The MSP’s intervention has saved ITV huge amounts of money over the years, and its support and services will continue to underpin the delivery of the company’s digital plans for many years to come, says Clark.

“The MSP did save the company a lot of money at a time when we really needed to save a lot of money, but we did find – as you sometimes do – that these contracts can be complex and difficult to deviate from as soon as you want something exceptional,” he says.

Read more about DevOps

Dealing with this has required both parties to tweak the way they work together to ensure the support and services the MSP offers fully complement what the broadcaster is trying to do.

“We had a couple of sys admins or platform engineers who were getting virtual machines from the MSP, and instead of having the MSP fully managing them, we started to take on more of that responsibility,” says Clark, by way of an example.

“That is why we started running things like Puppet. We wanted to be able to say, ‘this isn’t quite how we would have built it, but let’s use Puppet to make it a little more inhabitable’.”

Another example is the work the MSP will be doing to help Clark make Windows a “-class citizen” of the Common Platform, as part of a wider push to boost its compatibility.

“[The MSP] is going to be providing the Windows platform engineers into my team, and it’s a different model for them, because usually they own an entire function,” he says.

“This time they are sending people into my team. I will manage them and set the direction, but they will be managing the infrastructure.”

Natural next step

The project is a natural next step for Clark and his team, and will essentially see them apply their Common Platform thinking and tooling around Linux to the Windows Server world, he says.

“We are following exactly the same principles, so we’ll have embedded engineers onshore, and we’ll have configuration management everywhere, because [at present] all of our Windows estate is manually built,” he says.

“Because of that, the biggest task over the next four or five years is application archaeology. We’re going to have to go through all of those applications, find out how they run, how they’re installed and turn that into configuration, bit by bit.”

ITV currently operates two datacentres, and the need to refresh the hardware inside them is another reason for extending the reach of the Common Platform to Windows Server applications, says Clark.

“So we said, ‘why don’t we take the opportunity to Common Platform principles to the Windows world?’ It’s a massive task and, at the end of it, there will be a single Common Platform for Windows and Linux, for on-premise and cloud.”

Read more on DevOps