Chef: Now we know DevOps, how to understand DevOps metrics

This is a guest post for the Computer Weekly Open Source Inside blog written by Mandi Walls, technical practice manager / consulting director for EMEA at Chef Software.

As a piece of software, Chef is best described as a configuration management tool for handling ‘machine setup’ on physical servers and virtual machines, typically (but not exclusively) in cloud computing environments.

Walls writes as follows:

So you want to get started with DevOps, yay!

Instead of starting a potentially very long, conceptual conversation about what DevOps means, it’s more effective to identify a small but non-trivial project or area of your business that would benefit from being able to develop and deploy software faster, at scale… and more easily.

This means getting teams and people from across the IT stack together and getting them working on that one project or area, because the results and benefits become rapidly apparent.

Which all sounds marvellous, but what then, are the key metrics for DevOps success?

It’s time to ask ourselves some key questions here.

Q – What is your ultimate DevOps goal?

Consider this statement, “A lot of peoples’ ultimate goal is Continuous Delivery (CD). So measuring how long it takes to create and deploy software is of critical importance. This ‘Time to Delivery’ metric shows how long it takes to get new product from code checking to production.”

Ideally you want to see your Time to Delivery drop from months, to weeks, to days… and then literally to hours and minutes.

For many enterprises who are just getting started on their journey to the holy grail of Continuous Delivery, the truth is that Time to Delivery can still be months. While this isn’t necessarily a cause for panic, it’s certainly true that any business focused on delivering value through software, should be looking at how to significantly increase velocity.

Q – Is DevOps working, is the Q-for ‘quality’ there?

Alongside our measure of Time to Delivery, Quality of Release is another key metric. It shows how many bugs and errors are appearing in each iteration of the product. Businesses should be looking to rectify these, and ideally, develop a pre-production environment and workflow that uses automated testing. To really improve Quality of Release, managing it has to be ingrained in the workflow.

Q – How do we get to the guts in DevOps Metrics?

Over the past five years, we’ve seen cost savings, as a key metric, be talked about less and less. This is because cutting costs or headcount, is rarely a transformative approach or way of thinking. As the saying goes, “You can’t cut your way to the top”. This means the best metrics are those which help you measure the extent to which you are creating and providing greater value.

The format for such metrics will vary according to your individual business and industry.

Chef's Wallis: "You wanted DevOps defined again? Pul-ease... I thought you prefer we got down to the guts of the matter."

Chef’s Walls: “You wanted DevOps defined again? Pul-ease… I thought you prefer it if we got down to the guts of the matter.”

Arguably, DevOps metrics should always, ultimately, connect to a measurable improvement in the customer or staff experience, or at least the effects of it. This could be greater engagement, improved Net Promoter Score, or increased customer utility, such as online billing and account management.

In any industry, when getting started with DevOps and deciding on your metrics, it’s essential to begin with an accurate, current or near-future understanding what matters most to the customer. The same applies if your customers are internal.

Why are DevOps metrics so taboo?

Operational metrics such as Time to Delivery, and Quality of Release are starting to be more widely understood and discussed beyond hardcore DevOps audiences. However, even in DevOps circles and broader IT and company leadership, there’s a lot more fuzziness when it comes to having well defined business goals and working towards them.

This is getting better, there’s been a marked improvement over the past three years in particular. I’d say about 45 per cent of the clients we speak to don’t have a direct measurement for business success in mind, when we begin speaking to them. So there’s still a lot of opportunity for progress.

Key business metrics for getting going with DevOps?

To build on my previous point that the best DevOps metrics are those which measure the extent to which you are creating value… It’s worth considering this question in context of the wider shift towards seeing IT as a source of competitive advantage, rather than as a cost centre.

If you can keep this goal in mind, and choose metrics that will act as supporting evidence for this much larger organisational mind-shift, you will likely start, and stay, on the right track.

  • How about key project metrics?

“For any project or software product, defect tracking is an essential metric and an issue that’s easy to hammer out with automated testing. This is really a branch of Quality of Release, so together with Time to Delivery, these are highly salient metrics for virtually all projects.”

How will metrics will differ, depending on business size?

It’s important to remember that until you are properly collecting data from your business in the first place, it’s hard to use it to inform the correct choice of metrics. You can’t know what’s missing until you start looking.

Even once you have the data, you can’t choose your metrics until you know what business goals you’re driving towards.

For larger businesses, there’s often so much information stored inside legacy systems, where the data goes in but never out – setting up the mechanisms to collect and review it, is the first building block of success.

Once this has been done, some of the most valuable metrics are emergent, in that they arise from the metadata showing how the organisation is interacting with itself (as in, between teams and departments) and the customer.

Again – if you can pull ‘goal X’ from ‘data set ‘Y’, and share it within the company – you’re taking a big step towards raising awareness of the power of IT to act as a business value multiplier.

Help! How do I start on transformational goals?

For smaller organisations, it can be hard to begin with transformational goals and metrics, or set up company-wide dashboards and monitoring, because the IT department has even less capacity than in bigger businesses.

So it’s usually best to carry out some automation projects first, to pull out the manual work and free up IT staffs’ time. This effectively creates the capacity you need to drive larger change projects.

There now… aren’t you glad we got into the guts by now and didn’t try to provide you with a dictionary definition of what DevOps is?