Driving blind? IT dashboards, instruments and controls

For a technically astute department, IT can sometimes struggle with metrics, performance indicators, management dashboards or balanced scorecards. They are known by many different names and the business of managing IT becomes so much more difficult without them, but unfortunately they can often be derailed by politics, suspicion or lack of continuity.

For a technically astute department, IT can sometimes struggle with metrics, performance indicators, management dashboards or balanced scorecards. They are known by many different names and the business of managing IT becomes so much more difficult without them, but unfortunately they can often be derailed by politics, suspicion or lack of continuity.

Having worked on various flavours of IT performance management in several banks, I have seen what worked well and where the pitfalls became apparent. All too often, organisations can lose sight of the fundamental purpose behind performance measures and expend significant effort producing numbers that serve no further purpose. Why spend money producing the figures if they do not pay back to the organisation in some way?

One answer is to develop a pragmatic IT management dashboard, but the key is to include both the instruments and also the controls. A number of IT dashboards I have witnessed over the years focused on "output only" and were not linked to management or technical processes which drive the measures displayed on the dashboard. Thinking about a car dashboard, this is much like providing all the dials and warning lights, but no steering wheel, pedals or switches. Sure, you can see what is happening but you have no control; you are just along for the ride. This is one of the recurring problems with metrics: you tell management you are doing 45 miles an hour heading south and they ask you to go 60 miles an hour heading east. The metrics make it obvious you are not doing what they want and it will probably be a struggle to change direction and accelerate at the same time. A good set of metrics can, therefore, be a double-edged sword if you do not also have control.

So how do you go about selecting the right measures, communicating them and then using them to improve how you manage IT? The main indicators are often based around money, service delivery, problem areas and risk. Each organisation will be different, but tracking budget and incidents, such as outages or security breaches, are almost universally recognised as important things to do. A good way to identify key measures is to review what management information you actively use today and what you would like but do not yet have. Then think about problem areas which you want to address and aspects that you either need to change or expect will change around you. It is a good idea to start with a small set of measures as you can add more later, rather than start with so many the task becomes daunting from the outset.

Another good technique is to poll key managers and staff for measures they find useful and then prioritise these. Each proposed measure should be reviewed against a selection checklist to confirm it is suitable, where it comes from and how it will be used. (see Table 1 - Selection checklist)

Table 1 - Selection checklist

Aspect

Considerations

Audience

All staff, IT management, senior management, external

Effort

Must be cost effective to produce on a regular basis

Frequency

Real-time, daily, monthly, quarterly, annually

Relevance

To IT and to the rest of the firm

Producer

Who will be responsible for producing each measure

Sources

Availability, reliability, accuracy, durability, automation

Consistency

Essential for performance management over time

Independence

Avoiding conflict of interest, opinion, spin and political influences

Culture

Open culture or more guarded

Response

Impact, reaction and further work caused by publishing the measure

Delivery

Presentation, spreadsheet, intranet, meeting, webcast

Dependency

Co-operation from Finance, Procurement, HR and other information holders

Tools

Do you need specialist tools or can you use what youÕve got

Linkage

Impact on staff pay, benefits or promotion

While developing the list of measures, consider whether each is just a metric or could become a more useful key performance indicator (KPI) which can be assigned targets and influenced by a management or technical process, (see Table 2 - Measurement types). For example, you cannot control how many e-mails people send inbound to your firm, rendering this an uncontrollable metric. However, if you use a spam-blocker you can quote the percentage of inbound e-mails rejected as spam before reaching staff. By tweaking the spam-blocker settings you can adjust what gets through and hence the value of the KPI. Having said that, the number of inbound e-mails may provide useful forecasting information if systems are reaching capacity and signal that upgrade work will be required in the foreseeable future. This is a simple example but illustrates the main point, which is to ensure each measure has a clear and valuable purpose.

Table 2 - Measurement types

Type

Description

Example

Metric

Just a number in isolation

We supported 48,000 transactions today

Key performance indicator (KPI)

An important metric with a target value or range and its trend over time

We supported 48,000 transactions today against a maximum capacity of 65,000 which is 74% of our current capability and currently increasing at an average of 2% per month

Balanced scorecard

A collection of KPIs giving a wider view of IT

Includes financial performance, service delivery, operating processes and staffing KPIs

Dashboard

Balanced scorecard linked to management processes

A dashboard is backed up by the processes required to influence or control the KPIs presented in it

Superboard

Executive summary of the dashboard

The CIOÕs interpretation of overall IT health, performance and risk

Having identified a good starting set of KPIs, (see Table 3 - Popular indicators), these can be assembled into a balanced scorecard by grouping them into categories. Four groups representing the key attributes of an IT department often work well, these being financial, service delivery, operating processes and staffing. Taking this one step further and identifying the mechanisms which influence the scorecard leads us to the fully functional dashboard. The dashboard itself may well be identical in appearance to the scorecard, but it earns the new name because the processes which influence each KPI have been identified. This enables a cause and effect feedback loop to evolve around each KPI giving IT management a good level of control over the measures which matter most. It also allows meaningful targets to be set for each KPI so continuous improvement can be demonstrated over time. Once this regime is in place it is relatively straightforward to add or delete KPIs as priorities change over time. A problem area solved this year may drop off the dashboard next year and be replaced by a hotter issue.

Table 3 - Popular indicators

Actual & forecast cost vs budget

Development vs support ratios

Headcount stats vs targets

Cost savings achieved

Transaction, email & other volumes

Staff turnover / tenure

Procurement value committed

Server count and capacity

Staff survey ratings

Contract / SLA breaches

Helpdesk stats and performance

Training days and targets

Project portfolio status

Outages / incidents

Recruitment performance

Cost allocations to user groups

Storage space and capacity

Floor space utilisation / charges

BCP / DR coverage and readiness

Server utilisation average and max

IT security incidents

Depreciation locked into budget

Data centre utilisation and capacity

Number of users supported

Another key point sometimes overlooked is to ensure different audiences are recognised and presented with the most appropriate level of information. The same base set of measures can be used to do this, but several slices may need to be produced to satisfy different groups, (see Table 4 - The four key audiences).

Table 4 - The four key audiences

General Release

Suitable for wide publication across the firm and potentially externally

IT Management

Required by IT management to run the department effectively

Senior Management

Summary of relevant information to demonstrate IT status, performance and progress

Executive

The CIOÕs interpretation of ITÕs overall health, performance and risks

General release content should be restricted to the least sensitive information which would not cause problems if it got out into the wild. It is always wise to agree this internally before publication.

The most important aspect is the IT management information layer which is key to enable efficient operation of the IT department and services it provides to the organisation. This does not need to be published widely, but should be available to senior management if necessary.

The senior management dashboard is an extract of the KPIs most important to the organisation beyond IT. These are not necessarily the items which IT would most like to share as the firm is more likely to be interested in problem areas and aspects that need to show tangible change or improvement.

The three levels described so far make use of quantitative measures and do not rely on opinion or speculation, but the final level introduces an interpretation of the overall position and outlook: the "CIO Superboard"

The CIO Superboard (see Table 5 - The CIO superboard) is the ultimate summary giving a very concise indication of the overall status, health, risk and outlook for IT. It is an interpretation by the CIO based on the numerous measures below it. The purpose here is not to give the numbers themselves, but to provide a considered opinion based on the underlying data. The CIO must remain accountable for this opinion and be prepared to discuss or defend it in front of the rest of the firm. Top management will often implement a reporting regime of this type so they can see at a glance the overall status of each department in the firm, so it's quite likely you already contribute to a report of this nature. Introducing or reviewing the KPIs below it will allow the CIO to produce the Superboard quickly, consistently and with confidence, backed up by the dashboard programme which sits behind it.

Table 5 - The "CIO superboard"

Aspect

Status

Comments

Financial

Green

At 98% of year-to-date budget and forecasting 1% favourable for full year

Service Delivery

Green

No major outages or incidents

Operating Processes

Amber

Inefficiencies in 3 core administrative processes being addressed

Staffing

Amber

Increase in leavers this month + reduction in target average tenure

Risk

Amber

IT security risks continue to evolve, being addressed with software vendors

This has been a very brief introduction to a mainstay of good IT management practice, but I will conclude with a simple thought encapsulating the essence of an effective IT performance management dashboard: do not forget the steering wheel.

Read more on IT strategy

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.

-ADS BY GOOGLE

SearchCIO

SearchSecurity

SearchNetworking

SearchDataCenter

SearchDataManagement

Close