Software application quality lessons on a Hawaiian yacht

Software application testing company Coverity held a press and customer breakfast meeting yesterday morning in the City of London area to analyse the code quality refinement work carried out by Adobe, a customer of its products.

After a full English, half a bottle of HP sauce and a full pot of coffee, attendees got the full spiel from Todd Heckel, director of engineering and quality at Adobe.

Heckel explained that he and his team were focused on software application development testing across the Adobe suite — the sort that leads to real and tangible actions that can be carried out with a view to removing defects that exist in any product.

This can include peer code reviews, unit testing, static code analysis and as many as four or five more stages of code “filtering”… and it is this filtering element that provides the colour for our story.

The Watermaker Lesson

Heckel and his wife are clearly outdoor adventurous types; they embarked upon a voyage to Hawaii from California some time back with all the kit they needed on a lovely vessel, which came with its own “watermaker”, a mechanical filtering system capable of turning seawater into drinking water.

It was a 18 day voyage, but on day nine, the watermaker packed in.

The system worked by removing larger pieces of kelp from the water to start with and then gradually carried on filtering to clean out smaller and smaller particles, eventually getting down to a small membrane that could remove salt and finally make fresh water.

But it turned out that a membrane was broken at the initial stage that let big kelp chunks all the way into the final clearance process! The result was system meltdown.

So what’s the lesson here for software engineering? You have to get the big problems out of the way first or they risk killing the project mid lifecycle.


So back to the real world, Adobe says that it spent considerable effort on defect testing in 2011. The company now hopes to use Coverity products to realise a $35 million dollar saving in 2012.

But, says Heckel, static code analysis is relatively new to teams at Adobe in regard to their working with Coverity’s tools, so one might expect the defect detection improvement curve to be exponentially positive as time moves forward. “So far Adobe has found 5000 defects through using Coverity,” said Heckel.

Heckel said his teams enjoy the tool’s ability to find “real and meaningful” issues and that he places great importance on metrics. So much so that testing and defect tracking without accurate metrics are not worth so much…

Heckel also had a message for staff management in the testing arena. “Team managers should not use early defect data for individual programmer performance management, this potentially leads to individuals learning to hide defects early so that they are not picked up on individually… It is always more important to analyse team performance as a whole,” he said.

All in all a good morning and a few lessons were learnt.

Lesson #1: If you’re sailing to Hawaii, check your water filters ALL the way through before you leave.

Lesson #2: Software testing early rather than later is ALWAYS the most prudent course of action.

Lesson #3: From a timeline point of view, if a software bug has been around for 10 years and it is not causing problems then it probably isn’t an issue, but triage tactics to address serious problems first are still applicable.

Lesson #4: Even Adobe’s seemingly polished products still need professional testing tools applied to them.

Lesson #5: The Mercer Restaurant on Threadneedle Street knocks up a decent Full Monty and even serves HP sauce. A slightly poor show on the teeny mustard pots though; will flag this as core defect if I am invited again.

Image: Spectra Watermakers