vege - Fotolia
Software engineers should adopt a more sales-like approach to problem solving or risk slamming the brakes on the pace of digital transformation in their organisation, it is claimed.
That is according to Simon Tarry, director of engineering strategy at Ticketmaster, who outlined some of the challenges the ticket-selling site has encountered during its cloud migration efforts during a session at the Cloud Expo conference in London.
The organisation regularly sees huge spikes in the numbers of people that visit its website, as concert and theatre goers clamour to buy tickets for events as soon as they become available, putting huge amounts of pressure on the firm’s IT infrastructure.
As an example, Tarry explained how the imminent release of Beyonce tickets a few years ago led to a record number of visitors to its website, pushing the resource capacity of its London-based datacentre to the limit.
“Our server response time was around 0.7 seconds, and we could handle a concurrent load of around 300 users per minute. The rest of the traffic was queuing, and it’s not a great experience if you’re sat in a queue, trying to buy tickets,” said Tarry.
“We ran out of space in the datacentre in London, so we started to look at the options we had in terms of doing a migration, and cloud was an obvious one.”
Making a move
This saw the firm take steps to migrate the workloads and functionality housed in its London datacentre to a private cloud instance in Amsterdam, prompting an internal rejig of its IT teams to accommodate the move.
“We basically pulled together individuals from different teams to make sure we had the right capability to do this migration,” he said.
The company initially anticipated the migration would take two to three months, but some misplaced assumptions about the performance of its underlying infrastructure led to some unexpected delays, said Tarry.
“We made assumptions about the network infrastructure, but that had changed. We assumed it would perform as well as, if not better, than what we had before.
“We did a lot of testing, and basically played a game of hunt the bottlenecks in the infrastructure or in the software application.”
After a while, this process uncovered shortcomings in the input/output (IO) performance of the database server, which was resolved through adding solid-state disks (SSD) to the infrastructure mix, as well as a shortfall in the bandwidth being supplied to its communications card.
The power of no
However, these problems were relatively easy to work through when compared with the bottlenecks caused by the firm’s software engineers and their response to being told “no”, said Tarry.
A software engineer, when tasked with investigating a problem in a certain part of the infrastructure stack, will need to work with other teams to establish what’s going wrong, he explained.
If the infrastructure team does not have time to entertain them, there is a tendency on the software engineering side to turn their attention elsewhere and downgrade the job.
“When someone says no, it’s very easy for a software engineer to think, ‘Right, I don’t need to bother with that area. That team is busy, we’re not going to explore it’,” he said.
This can lead to delays in completing a cloud migration, because the source of the problem takes longer to track down just because engineers are happy to take “no” for an answer, he added.
“We found there was a team holding us up, because– even when we started to question them – they said, ‘no, we’re too busy to help’,” he said.
Simon Tarry, Ticketmaster
That was until Tarry’s team got hold of some metrics that half-proved this team, and its slice of the infrastructure stack, might be the source of the problem.
“We had the metrics to prove it to them, and asked them to look into it, and they said, ‘yes – it is a problem with us’. We got delayed for many months before we got to that point,” he said.
Evidence was essential in dealing with this situation, but another way to prevent similar delays from occurring could be to send engineers on some form of sales training, so they become less inclined to take no for an answer, offered Tarry.
“Software engineers make terrible sales people because we keep accepting no from someone else, and going, ‘oh, I’ll put that problem at the bottom of my list of things to solve’,” he said
“If your sales people accept a no all the time, you’d make no sales and have no business, and this is very much an engineering mindset we apply to problems.”
The move to Amsterdam paid off, said Tarry, as the site managed to hold up in the face of another huge ticket sales at the end of 2014, involving Fleetwood Mac.
“We got what we wanted and the performance we wanted, but it took a long time to get there,” he added.