This article is part of our Essential Guide: Next generation Agile: Guide to continuous development

Finding success in the public cloud: Financial Times CTO John O’Donovan

As the Financial Times migrates data workloads to the cloud, CTO John O'Donovan explains FT's cloud strategy

For publishers, innovation in advertising strategies can boost revenues. After finding success in using cloud data warehousing products to make its subscriptions and financial data more efficient, the Financial Times (FT) is now looking to migrate its advertising data to the cloud and build new revenue models.

The FT’s CTO John O’Donovan (pictured) said that, following its use of public cloud services for mission-critical data warehousing tasks, the IT team is more confident about using cloud services for other critical data workloads. 

The FT’s IT team built its data warehousing on AWS’s RedShift offering. In doing so, it can now run 450,000 online queries 98% faster than with its previous traditional datacentre, and reduced infrastructure costs by 80%, said Donovan.

Data warehousing is important for FT, as data insights shapes its business decisions. data warehouse is a repository of data that an enterprise's various business systems collect. It is designed to facilitate information retrieval and analysis. The data contained in a warehouse is often consolidated from multiple systems, making analysis across those systems quicker and easier.

Amazon announced its data warehouse product - Redshift - in November 2012 at its first re:Invent conference, making it available from 2013 onwards.

“Traditional data warehouse products are too expensive and have licensing complications,” said AWS senior vice-president Andy Jassy at its launch. “Many large enterprises told us they are unhappy with the existing data warehousing services in the market.”

AWS Redshift is an automated, petabyte-scale data warehouse service in the cloud that helps enterprise IT automate labour-intensive tasks, such as setting up, operating and scaling a data warehouse cluster. It aims to help them provision capacity, monitor and back up the cluster, as well as apply patches and upgrades.

'More pay than go'

“We had a data warehousing system in the FT. It was not rubbish but it was inflexible, it had limited features, it was slow and expensive,” O’Donovan said at an earlier AWS summit in 2014. “Besides, it was not in a single place.” This made data analytics and warehousing tasks difficult for the IT team.

“Our chief executive once said, ‘We quickly came to realise that actually the real power of subscription relationship comes from data,' and this thought really shaped our strategy,” he told delegates at the summit.

“Previously, we had used managed services data warehousing kit on a pay-as-you-go basis. But it was not flexible, it was slow and it was more pay than go.”

Moving the mission-critical task to the cloud has not just reduced processing time by 98%, but also enables the FT’s IT team to deliver projects on time, and to budget. “It’s no more a case of a black-box data warehouse for the FT,” O’Donovan said. “Moving from a report-based scenario to a real-time data analytics situation is hugely beneficial.”

The project gave the FT’s IT team the confidence to adopt more cloud services. “We’ve proved we can put pretty much anything in the cloud,” he said.

After reaping the benefits of AWS RedShift for subscriptions data, the FT is now looking to migrate its advertising data to the public cloud platform.

“Advertising data is equally critical but it is more specialised and not as straightforward as subscriptions data,” O’Donovan told Computer Weekly. “Advertisers can be very demanding but the cloud has enabled us to be innovative there.”

The FT’s IT team is now looking to set ad strategies based on the time readers spend looking at an ad, rather than just based on the number of clicks or views. “Platforms like public cloud give you a chance to innovate. Now we have the technology power to invest our time in strategic tasks that add value to the business. Previously these kinds of ideas would have required huge investments,” he said.

New era of computing

“I remember the days when it took months to get a server. It could be very disappointing.”

O’Donovan urged other CIOs to look at new ways of doing IT. “People spend a lot of time in building resilient infrastructures and their strategy is, ‘Build something so it does not fail’ – but cloud dynamics are challenging this process.”

According to him, now IT must automate the process to make it easy to kill an application instance and start all over again quickly. “We can spin up virtual machines in 20 minutes and quickly move forward,” he said.

“IT cannot afford anymore to spend weeks investigating why things failed in an infrastructure they build so resiliently and on a long time scale. Today, it is more about preparing for failure and being able to stand up a new instance quickly to keep the business moving.” 

“Today no-one should be proud of a hero server that has been up and running for 2,000 hours. Those days are gone. Now, it is about the ease of destroying and creating new virtual environments easily.”  

For those who worry about using cloud, O’Donovan has a tip. “Everyone worries about going to the cloud. We addressed this by tiering our applications,” he said. The FT has tiered its apps and workloads into three categories – bronze, gold and platinum services. “We chuck all bronze workloads on to the cloud. This helps build confidence and helps the transition to the cloud.”

IT automation is key to this, he insisted. The FT’s IT team runs automated scripts for application deployment using Puppet software. “We don't buy a product unless it has an API,” he said.

The cloud-friendly strategy has helped the FT’s IT team improve its speed to market. “Our speed to market has improved from 99 days in 2011 to less than an hour today – that is adding real value to the business,” O’Donovan told delegates. It has also made the team agile and allowed them to focus on more strategic ideas, such as DevOps.

Tagbot – the FT’s in-house “hero” tool

But that’s not all. The IT team has developed tools such as “Tagbot”, an FT-specific tool that switches off applications and instances if it does not find tags. “It can be troublesome when the IT team forgets to tag but it really enforces good discipline in tagging.”

This process means that the FT does not pay for cloud instances that the IT team does not know about. “When you don’t have tags, you don’t know what it is and then you fear to kill the instance because you don’t know what it will do and you keep paying for it forever,” he explained. 

The FT’s Tagbot concept proved so popular among the delegates at the summit many demanded to know if the FT could open-source it. “It really is a very simple, script-based tool that helps us immensely in getting things right. Our development team loves to use it to switch things off! We are building more such in-house tools and cloud has enabled us to do that,” he told Computer Weekly, adding that the team will see if it can release it for others to use too. 

Another cost saving strategy the FT’s IT team uses is switching off test and development instances at night. “You can save up to a third of Amazon's costs by simply switching off unused instances at night.”

“In general, our whole journey to the cloud has been around finding the right balance.” But O’Donovan is not getting rid of the FT's datacentre infrastructure. He’s keeping it lean but won't get rid of it, so there something to fall back to, in case there’s a “cloud Armageddon”, he said.

Read more on Datacentre systems management