Technology fail during US 2012 election day averted by practising failure

A “catastrophic” IT failure was avoided on the US presidential election day due to rigorous testing of the IT infrastructure

A potentially catastrophic IT failure was avoided on the day of the US presidential election due to rigorous testing of the IT infrastructure beforehand, according to former Obama campaign CTO, Harper Reed (pictured).

Reed, who was the CTO on the Obama For America campaign in 2012, said the campaign technology experienced no downtime because the possibility of failure had been so well prepared for.

Speaking at the Direct Marketing Association’s (DMA) Technology Summit, Reed said that during the campaign he held a "game day", where engineers and technicians destroyed their work to help them learn from every aspect of failure by aggressively fixing it.

“We learnt a lot about the technology and the organisation, and we had no downtime because of this,” he said.

On election day, the campaign witnessed a “catastrophic” IT failure, according to Reed, but it caused no downtime because the engineers had learnt so much about their technology that they knew when it would fail and how to ensure things appeared to run smoothly regardless of what was going on behind the scenes.

Reed said that companies should implement simple paths in their technology to navigate users away from a failure when systems break down. This could be as simple as providing an answer to default error pages, such as asking the user whether they would like to debug their connection. 

“Never let the users see failure. Bounce them some place where they can do the work,” he said.

Breaking down technology barriers

Reed, a US-based technology engineer, was responsible for delivering Obama’s online campaign and the get-out-the-vote programme. He also created a technology platform to drive campaign fundraising, while also improving voter communication and volunteer management. Simply, he was responsible for engineering the most sophisticated political campaign of all time.

However, Reed admitted that he did not look like a typical political campaigner. He described himself as bearded and techy-looking, like most engineers.

Reed said he found diversity in the workplace a challenge when working as CTO on the campaign because the typical IT engineer lived up to the stereotype of “lots of white guys with beards”. He said he and his team mostly failed at hiring people who looked different because of this.

“We have to try, and we have to talk about it,” he said. “We need diversity, because the internet is a democratic organisation – anyone can do anything."

Reed pointed out that if every member of his team looked like him, they would not reflect the diversity of internet users. 

When he was first hired, Reed also experienced difficulty in seeing eye-to-eye with other campaigners. He said he not only looked different from other people, but due to his engineering background, he tended to speak a different language.

“The language around tech is very ego-based,” he said. “It’s ‘I will fix this’, rather than ‘We will work together to solve this problem'.” 

During his time on the campaign, Reed tried to change the language and actions, which helped the organisation work internally. 

He said that most of a CTO’s problems are related to people, not the technology. “Technology is easy, but making sure people trust you so that you can move at the speed in which you need to move is hard. A good CTO is about moving hurdles.”

Read more on Web development