Stop bugs before they stop you

There is no such thing as bug-free software. According to most studies, the average number of software bugs per 1,000 lines of...

There is no such thing as bug-free software. According to most studies, the average number of software bugs per 1,000 lines of code hovers between five and 20. Most are errors in syntax that never surface as problems. But with applications ballooning to millions of lines of code, the chance of a show-stopping mistake affecting any given application rises precipitously.

Examples are easy to find. Take Microsoft's server vulnerabilities, Oracle's first release of its E-Business Suite 11i, or Netgear's router firmware, released last May, which continuously pinged the University of Wisconsin's public Network Time Protocol servers, resulting in an inadvertent DoS (denial of service) attack.

The perception among many IT customers is that bugginess has reached crisis proportions.

The effect of bugs on productivity is high. The National Institute of Standards and Technology in 2002 released a report stating that software errors cost the US economy $59.5bn a year.

The study found more than a third of that expense could be eliminated by improved testing that enabled earlier and more effective identification and removal of defects.

To tackle all manner of software quality problems, enterprises are establishing best practices in the development phase, using third-party testing software to catch errors, and hiring third parties to inspect code after the fact.

Build quality into software code

According to Jeff Payne, president and chief executive officer of code-assessment services provider Cigital, software failure occurs for three reasons.

"First, software is probably the most complex thing we try to build today," Payne says. Second, the nature of software is such that no foolproof set of rules can be created that will absolutely eliminate bugs. The third reasonis "the fact that developers and people who build software just do a very poor job of testing, validating, and building what they're doing", according to Payne.

Most analysts agree that although a separate QA procedure should always be in place, the best way to increase software quality is to have developers test as they go - and to establish procedures that ensure business-side requirements are well understood.

Bugs often spring from common human error, but poorly conceived or poorly conveyed business requirements are equally culpable. When something does not work as intended, users do not care whether the cause was a programmer's slip of the finger or a misread requirements document.

"Best practices is to build quality in. Don't try to test it in," Payne advises.

For Cigital, proper software engineering means specifying what is to be built, and then architecting and designing before coding and testing. Using test-driven development, code is tested early in the process, rather than waiting to test the entire system when it becomes more expensive to fix problems.

In addition, software quality reviews and artifact analyses help companies that build software cut costs by eliminating expensive human hours for reworking and late lifecycle testing costs.

"You cannot catch all of the bugs through QA," says Alberto Savoia, co-founder and chief technology officer of startup Agitar, developer of Agitator bug-testing software (due for release in early 2004). Savoia also advocates getting developers more involved in bug detection rather than leaving this function to QA personnel.

"Really, the issue of software as a whole is, essentially, that software is still handmade. It's developers getting together and still hammering it out by hand," says ZapThink senior analyst Jason Bloomberg.

He advocates XP (extreme programming) and "agile" software methodologies which "more tightly link developers to the users who will use the final product".

Agile methodologies are specifically intended to ensure software meets business-side requirements, especially when requirements are changing, Bloomberg says.

But the practice loses effectiveness when scaled beyond small project teams. "Often, the project is too large to have a small team of developers with some users on it," he says. The requirements are too numerous and the repeated evaluation of applications by the business side becomes too heavy a burden.

Besides, many developers are naturally resistant to feedback. "To a large degree, developers still see themselves as artists," says Alexander Falk, president and CEO of Altova, an XML tools developer. He stresses that software development should be more like engineering and less like art so that developers can be open to different approaches.

Getting the business side on board

Management must be more attuned to software quality issues, says Jeff Klagenberg, director of product management at Reasoning, a code inspection service.

"When you get to business management, there's often a disconnect with the software development side and the fact that services and tools exist out there to make it easy to remove these defects," he says.

Reasoning revenues have increased 50% or more each quarter this year, according to the company, and the number of lines of code inspected has increased more than 200% per quarter. The company's prices start at 18 cents per line of Java, C, or C++ code examined in a process that mixes manual and automated techniques.

The business risks of lax inspection can be high. Through code assessments, Cigital customer MasterCard International has uncovered security issues in applications, according to Simon Pugh, vice president of infrastructure and standards at MasterCard.

"Certainly, as a result of their services, we have found and prevented a number of problems that otherwise would occur," says Pugh, such as flaws in software that could have been exploited by a hacker.

For example, in a smartcard application developed by a third-party company and subsequently analysed by Cigital, the application contained a back door that would have allowed a rogue website to interact with a card and obtain a PIN, he says.

Empirix, which provides load and performance testing, has found code problems such as e-commerce site users' pages getting transposed so each received the other's personal information, says Colin Mason, performance consultant in the Empirix hosted testing service group.

The company always finds at least one problem when it tests websites , says Bob Eldridge, manager of remote hosted services at Empirix. "We have yet to test a site that didn't have an issue with it before we started testing."

Paul Krill writes for InfoWorld

Read more on IT risk management