European CIOs have made progress in measuring software quality but there remains work to do, according to speakers at a recent event in Brussels.
The European CIO Forum, held by software quality measurement specialist Cast Software – which attracted about 100 IT leaders from across Europe – covered the challenges and benefits of measurement. When attendees were asked about their own progress, not one said they had implemented quality assurance testing across every area of software development.
Andrew Agerbak, associate director at The Boston Consulting Group, said more effort was required on the part of European CIOs and their suppliers.
“We've seen a lot of success when organisations use key performance indicators across vendors to agree a set of performance outcomes. It doesn't mean the tensions go away, but every party knows what they’re working towards. Conversations about value don't often happen in regards to IT and knowing what the priorities are is really beneficial."
Danielle Jacobs, general manager of Belgian IT leadership association Beltug, said European CIOs need to pay more attention to testing. Her organisation has undertaken detailed research, focusing specifically on the Belgian market. Jacobs said the issues surrounding software testing are complex.
While software sits at the core of most modern organisations, there are significant differences in how measurement is used in organisations, such as those that are regulated and those that are not. "Many people believe testing is now a mature discipline but we believe a lot of opportunities for change exist," she said. “There is a big gap between what companies want to do and where they are in reality."
Balancing software measurement and cost
The Forum did, however, demonstrate that some big name European firms have made progress. Paul Thysens, CIO at BNP Paribas Fortis, said the bank is introducing a performance measurement system that measures the quality of all applications. He said the cost of software is always important, but CIOs should not use testing to simply track total outgoings.
Productivity measures, for example, are crucial. Thysens said BNP Paribas Fortis analyses certain key elements, such as cost per function and the maintainability of software. "The benefits of testing should be related to making people great, not your tools,” he said. “If your workers aren’t motivated, and they see testing as a way to monitor their roles, then you won't be successful. You must position testing as a support mechanism."
Thysens said workers should be trained to interpret results and prioritise key actions for the business. "The objective of testing should be to improve the overall quality of your software,” he said. “There is a business impact because, once you link design into quality, you can start fine-tuning strategy in regards to where you put your money and your efforts.”
Gerdy De Clercq, head of integrated solutions and transformation at mobile telecommunications specialist Proximus, is another IT leader who believes organisations that use testing can be fixated on cost. "I only have one expectation – and that is that my team delivers 400 to 500 initiatives to the business in a year,” he said. “Every project is important and cost can create emotional debates.”
De Clercq is currently evaluating Cast Software. The key lesson learned so far is that measurement must be embedded deeply in the processes of the organisation. “Any figures that can be used to help defend your position are really helpful,” said De Clercq. “That's why software measurement is so important for CIOs."
Francois Bonifacio, managing director of IT services at laboratory specialist Eurofins, said his firm uses Cast Software to test software created through an agile development approach. Testing is used in an attempt to improve software quality. "If you're driving a fast car, you can't drive it with your eyes closed,” said Bonifacio.
Read more about software quality
- Ever more customer-touching systems will force organisations to ensure their underlying software is robust enough to prevent IT problems becoming public and causing brand damage.
- Businesses outsourcing SAP customisation often receive an end product that does not meet quality standards, an extensive benchmarking exercise reveals.
- Developers have adopted agile methods to keep up with the pace of the market, but how can they ensure code stays in good shape?
“In agile, you need a lot of discipline because you are working directly with the business every day and they will not accept it if you don't work in a disciplined way. If you don't measure, the business will lose track. If you do measure, you will increase confidence across the business, even if you change direction. You have to prove that you're moving forwards in the right direction.”
Bonifacio said Eurofins had an issue with a low-performing web system. The firm used measuring to isolate modules and decide which areas needed to be modified. "Testing saves costs in terms of development and refactoring,” he said. “It can be difficult to measure business benefits, such as cost. But for the customers, the benefits of testing come in terms of service quality."
How to benefit from testing
Geert Ensing, former CIO at ABN Amro, said there are three crucial ways that European technology chiefs can benefit from an investment in testing. First, guarantees around high quality technology operations are now essential because modern IT systems are highly integrated in the way businesses serve their customers.
Second, ever-rising system complexity means IT managers must be aware of how modifications to one platform can affect other applications. Finally, technology departments will have already invested money in testing and it is crucial CIOs exploit this. “It takes major effort for businesses to know the end results are reliable,” he said.
When it comes to progress on testing in Europe, Ensing said CIOs across the region are broadly in-synch with the efforts of other IT managers around the globe. Where variations exist, he said, they are between individual firms and sectors of operation. CIOs looking to hone their approach should ensure testing is viewed as an IT and business discipline, rather than something simply performed by technology professionals.
“Take an integrated approach and make sure you industrialise your process in regards to the tracking and tracing of modifications,” said Ensing. “Work closely with your external partners, too. Every change must be translated and its potential impact on the business and its customers considered.”