Application development just a weak link without metrics

Collect the right metrics at the right time, and the speed of software development will improve its chances at keeping up with the business.

Like the cobbler whose children have no shoes, CIOs toil away at making business processes more agile. But when it comes to IT work processes, the job isn't getting done.

"CIOs have been pushing the concept of business process management and business process optimisation to the business. But they themselves don't have the advantage of a systemised, measured and optimisable process for the single most expensive business activity they undertake -- systems development," said Brian Kilcourse, managing partner at research firm (RSR) and self-described recovering CIO. "If you can't put metrics around it, how can you possibly improve?"

Or keep your job, for that matter. "In the setting I work in -- retail -- there is a great deal of dissatisfaction with the IT organisation's ability to respond quickly to changing business requirements," Kilcourse said. "In today's business cycle, the notion of an 18-month backlog within IT is not acceptable."

Collect the right metrics at the right time, said Kilcourse and other experts, and the speed of software development can improve its chances of keeping up with the business.

Postmortem metrics = risk dead on arrival

Application development organisations lag behind their business peers, as well as IT peers, in using metrics to improve their work processes, according to Forrester Research. Metrics collected tend to be the wrong ones. In a study on application development metrics commissioned by application lifecycle management (ALM) provider Borland Software, Forrester found that application development teams rarely collect "in-flight metrics," or metrics gathered while the project is under way.

"Lacking real in-flight project data, app dev teams rarely have objective data about the true status of their projects or about the root causes of their problems," the consulting firm concluded. The July 2007 study was based on 20 in-depth interviews with IT professionals in charge of application development and project management.

Instead, application development teams are more apt to use postmortem metrics in judging a project's relative success. Unfortunately, on-time, on-budget and on-scope measures "say more about estimation accuracy than about whether a project took as long as it ought to have," the Forrester authors note. Metrics like "business value delivered" and customer satisfaction are far more valuable, but also far more difficult to collect.

RSR's Kilcourse agreed. "We measure things like did it arrive on time, or on budget. We calculate the number of errors per lines of code. But the fundamental question that keeps getting missed is: Did the effort deliver what the business asked for?"

Marc Brown, vice president of product marketing at Borland, said the Forrester study underscores the reality that app dev is the "weakest link in the chain" and software delivery a "black box." In December, Borland announced that it would introduce a number of new "open ALM" platform products in 2008 that will track the development process from idea to delivery. The products promise to help customers determine which metrics to choose and how to automate the collection of those metrics, thus optimising development processes through real-time or "in-flight" metrics that provide insight into things that matter to the business, such as risk, cost and value.

"We wanted to demystify it and make it more of a science," Brown said. "The open platform allows us to help organisations make that transition to a managed business process by utilising and leveraging their choice of technology process and best practice."

Direct competitors in the field include VMware, Microsoft, HP, IBM Rational and Compuware, but not all offer an open platform.

That's an important consideration, said Kilcourse, who serves on the CIO advisory board for Borland Software. Most CIOs at large companies have invested in some kind of application lifecycle management tools. The Borland approach of selling an integration platform for disparate tools, as well as its own tools, makes "good sense," he said.

"It sounds like what Oracle is doing with its AIA [Application Integration Architecture], but applied to the IT function," Kilcourse said, adding that IT deserves the same kind of advantages and opportunities as the business.

Costly and complex

What prevents app dev organisations from benefiting from the metrics their peers use?

It costs a company to collect data, for one, the Forrester study found. And because the metrics that organisations use are often superficial, or just wrong, the benefits are questionable -- and questioned by the people being measured.

According to the Forrester interviews, the No. 1 obstacle cited was the manual effort required. Even development shops that obtained metrics from tools told Forrester the collection was dependent upon someone, usually a project manager, entering the raw data. Manual metrics collection is tedious, error-prone and expensive. Tools that automate collection are also expensive.

Reporting and analysing app dev metrics is also technically complex, the study found. Nearly half of those surveyed by Forrester said they were unable to trend or aggregate the metrics they collected. Heterogeneous collection tools account for some of the opacity. Tracing the development lifecycle from start to finish often spans multiple development tools and tool vendors, the study showed.

Then, there is the fear factor. "It takes courage to stop working and measure all the potential sources of trouble when you're afraid that the very act of measurement might slow you down," the authors noted.

The reality is that software development remains a difficult process, Kilcourse said, despite advances made in automating the delivery of code. And the difficulty is something many "non-ITers" don't get. He offers the example of a COO who presents the CIO with an idea that could give the company a real competitive advantage: identify the company's top 10 customers and at the point of sale direct them to those offers of most interest to them. Don't bother them with the other stuff.

Sounds great, the COO is excited, he can see it, but then begins the long process of breaking it down into components: How do you define the top 10, what are relevant offers, how do you get those offers to customers when they need them, etc. And oh, by the way, the COO adds, can you get those messages to them on their cell phones in an SMS message?

"As an IT-er, you're rolling your eyes, thinking, 'My God, he just came up with a 36-month, US$50 million project,'" Kilcourse said.

Read more on Software development tools