Forrester: Modern app development requires a modern set of metrics

IT executives responsible for application development have a great opportunity to develop apps that improve customer experience or help businesses sell more

Application software development is evolving into a strategic business process for many enterprises that develop customer-facing systems of engagement. Systems of engagements in the Age of the Customer have a big responsibility in providing great customer experience where simplicity, ease of use and speed are only a few of the demanding expectations of customers. 

IT executives of this new age, who are responsible for application development and delivery have a great opportunity to shift conversations with business stakeholders toward value delivered (eg How apps can improve customer experience or help business sell more), instead of zero-sum haggling over the more traditional cost, scope and deadlines conversations. But that's easier said than done.

The truth is that software development is not a predictable process, and project managers and development teams can’t forecast how long any given development really takes, at least not for complex projects. We often see naive comparisons between software development and cooking or a factory assembly line. 

But we know that we can follow a cooking recipe step by step and get a predictable outcome, and industrial processes are scripted to the smallest degree. With software development, it's more like ordering the salmon and getting calamari instead. In modern application development, the outcome is not known beforehand and might be different from the planned result because the ingredients keep changing! (Think Iron Chef, with surprise ingredients at the start that may even change partway into the process).

Agile projects use different measures

What have we learned from the first 50 to 60 years of measuring software development projects? For many organisations, it seems like the answer is "Not much". Changing deeply rooted metrics that were created to support waterfall projects is like convincing Italians to replace fusilli and linguine with hamburgers and ribs. But take heart! 

More on application development

An increasing number of organisations are thinking differently: they accept that, no matter how good a team is, it just can't estimate all development projects with the same precision. That's especially true when new technologies, such as mobile apps or scale-out public cloud infrastructure, are involved.

These teams are also changing their measurement practices to reflect the belief that it's a waste of time to spend weeks or months specifying requirements up front. No matter how good the team is, requirements will change frequently - especially with customer-facing systems of engagement during the initial stages of development. In fact, many of these teams are the ones adopting Agile practices most aggressively. In the process, they are changing the focus of the metrics they use to measure success. 

During our research we have found an emerging set of metrics grouped around measuring progress, quality, efficiency, and the realisation of value and benefits (see figure).

Agile teams' metrics commonly span progress, quality and efficiency — but rarely value

Within those four categories, we see that:

  • Every Agile project measures velocity. Teams usually define velocity as number of story points from backlog delivered per sprint. Properly sizing user stories and tasks is the key to effectively determining velocity, but there is no standard sizing approach. But be careful when measuring velocity; as with any metric, it can create its own measurement antipattern.
  • Agile projects extend quality metrics in new ways. A good sign of an effective Agile team is a high ratio of automated test coverage. This requires an increased focus on quality metrics, especially those that relate to the level of test automation.
  • Using burndown charts creates a proxy for progress. Burndown charts track how fast the team is delivering user stories and tasks from the backlog. Product owners also attach value information to the user stories, showing how the software development team is actually delivering value as it progresses through sprints and releases.
  • Tracking technical debt creates a basis for tradeoffs. As projects progress, development teams discover defects, improper designs, new requirements, and places where it can improve the code. Collectively, these create technical debt that the team needs to address.
  • Measuring business value metrics is an emerging art. Thought leaders in some of the organisations we spoke with are busy tracking business-specific value metrics as part of their efforts to measure value delivery. 

Determine your metrics according to project complexity

Our research makes it clear: one size metrics does not fit all. In a complex, fast-changing problem domain, it will be hard to establish a clear value stream and align all of the project resources with it. In such a situation, determining metrics such as the unitary cost of a business process, like the cost of processing a claim or the cycle time from concept to cash, will be difficult. In contrast, if you work at an organisation where transparency and trust predominate and there's a culture of developer discipline, then more ambitious and effective metrics are possible. Before you start determining which metrics matter to you, ensure that you have a clear idea of what goal you want to accomplish. To select the right metrics, you should first determine your project's complexity profile and then identify metrics that align with the profile.

Determine your project's complexity profile . . .

Complexity theories have been around for years, but David Snowden has created a framework that we think works especially well when thinking about software development. The Cynefin framework [LINK] defines four types of complexity domains: simple, complicated, complex, and chaotic. We've used the domains in the Cynefin framework to help define and select metrics appropriate for each problem domain you're likely to encounter. Our approach takes into consideration three key factors: the level of certainty or uncertainty of requirements, team cohesiveness (how long the team has worked together and whether teams are working in a cross-functional or siloed organisation), and the project team's technology capabilities.

. . . And then identify metrics that align with the profile

Once you've determined the complexity of your project and mapped it to the domains of the Cynefin framework, it will be easier to select the best metrics to measure success, progress, and necessary improvements. As your projects move from simple to complex, the use of value-based and progressive metrics will vary quite significantly.  

Diego Lo Giudice is Vice President, Principal Analyst, serving Application Development & Delivery Professionals at Forrester Research.   

Read more on IT efficiency and sustainability