Driving blind? IT dashboards, instruments and controls

For a technically astute department, IT can sometimes struggle with metrics, performance indicators, management dashboards or balanced scorecards. They are known by many different names and the business of managing IT becomes so much more difficult without them, but unfortunately they can often be derailed by politics, suspicion or lack of continuity.

Having worked on various flavours of IT performance management in several banks, I have seen what worked well and where the pitfalls became apparent. All too often, organisations can lose sight of the fundamental purpose behind performance measures and expend significant effort producing numbers that serve no further purpose. Why spend money producing the figures if they do not pay back to the organisation in some way?

One answer is to develop a pragmatic IT management dashboard, but the key is to include both the instruments and also the controls. A number of IT dashboards I have witnessed over the years focused on "output only" and were not linked to management or technical processes which drive the measures displayed on the dashboard. Thinking about a car dashboard, this is much like providing all the dials and warning lights, but no steering wheel, pedals or switches. Sure, you can see what is happening but you have no control; you are just along for the ride. This is one of the recurring problems with metrics: you tell management you are doing 45 miles an hour heading south and they ask you to go 60 miles an hour heading east. The metrics make it obvious you are not doing what they want and it will probably be a struggle to change direction and accelerate at the same time. A good set of metrics can, therefore, be a double-edged sword if you do not also have control.

So how do you go about selecting the right measures, communicating them and then using them to improve how you manage IT? The main indicators are often based around money, service delivery, problem areas and risk. Each organisation will be different, but tracking budget and incidents, such as outages or security breaches, are almost universally recognised as important things to do. A good way to identify key measures is to review what management information you actively use today and what you would like but do not yet have. Then think about problem areas which you want to address and aspects that you either need to change or expect will change around you. It is a good idea to start with a small set of measures as you can add more later, rather than start with so many the task becomes daunting from the outset.

Another good technique is to poll key managers and staff for measures they find useful and then prioritise these. Each proposed measure should be reviewed against a selection checklist to confirm it is suitable, where it comes from and how it will be used. (see Table 1 - Selection checklist)

Table 1 - Selection checklist

While developing the list of measures, consider whether each is just a metric or could become a more useful key performance indicator (KPI) which can be assigned targets and influenced by a management or technical process, (see Table 2 - Measurement types). For example, you cannot control how many e-mails people send inbound to your firm, rendering this an uncontrollable metric. However, if you use a spam-blocker you can quote the percentage of inbound e-mails rejected as spam before reaching staff. By tweaking the spam-blocker settings you can adjust what gets through and hence the value of the KPI. Having said that, the number of inbound e-mails may provide useful forecasting information if systems are reaching capacity and signal that upgrade work will be required in the foreseeable future. This is a simple example but illustrates the main point, which is to ensure each measure has a clear and valuable purpose.

Table 2 - Measurement types

Having identified a good starting set of KPIs, (see Table 3 - Popular indicators), these can be assembled into a balanced scorecard by grouping them into categories. Four groups representing the key attributes of an IT department often work well, these being financial, service delivery, operating processes and staffing. Taking this one step further and identifying the mechanisms which influence the scorecard leads us to the fully functional dashboard. The dashboard itself may well be identical in appearance to the scorecard, but it earns the new name because the processes which influence each KPI have been identified. This enables a cause and effect feedback loop to evolve around each KPI giving IT management a good level of control over the measures which matter most. It also allows meaningful targets to be set for each KPI so continuous improvement can be demonstrated over time. Once this regime is in place it is relatively straightforward to add or delete KPIs as priorities change over time. A problem area solved this year may drop off the dashboard next year and be replaced by a hotter issue.

Table 3 - Popular indicators

Another key point sometimes overlooked is to ensure different audiences are recognised and presented with the most appropriate level of information. The same base set of measures can be used to do this, but several slices may need to be produced to satisfy different groups, (see Table 4 - The four key audiences).

Table 4 - The four key audiences

General release content should be restricted to the least sensitive information which would not cause problems if it got out into the wild. It is always wise to agree this internally before publication.

The most important aspect is the IT management information layer which is key to enable efficient operation of the IT department and services it provides to the organisation. This does not need to be published widely, but should be available to senior management if necessary.

The senior management dashboard is an extract of the KPIs most important to the organisation beyond IT. These are not necessarily the items which IT would most like to share as the firm is more likely to be interested in problem areas and aspects that need to show tangible change or improvement.

The three levels described so far make use of quantitative measures and do not rely on opinion or speculation, but the final level introduces an interpretation of the overall position and outlook: the "CIO Superboard"

The CIO Superboard (see Table 5 - The CIO superboard) is the ultimate summary giving a very concise indication of the overall status, health, risk and outlook for IT. It is an interpretation by the CIO based on the numerous measures below it. The purpose here is not to give the numbers themselves, but to provide a considered opinion based on the underlying data. The CIO must remain accountable for this opinion and be prepared to discuss or defend it in front of the rest of the firm. Top management will often implement a reporting regime of this type so they can see at a glance the overall status of each department in the firm, so it's quite likely you already contribute to a report of this nature. Introducing or reviewing the KPIs below it will allow the CIO to produce the Superboard quickly, consistently and with confidence, backed up by the dashboard programme which sits behind it.

Table 5 - The "CIO superboard"

This has been a very brief introduction to a mainstay of good IT management practice, but I will conclude with a simple thought encapsulating the essence of an effective IT performance management dashboard: do not forget the steering wheel.