On average 40% of activities conducted by application development teams are associated with application support, according to a Gartner Benchmarking Report.
Any executive in charge of application development should be concerned by that figure. And every executive in a software business should be alarmed.
In its 2004 Benchmark Study, the Service and Support Professionals Association reported that the percentage of support cases closed at first contact continues to decline, and the length of time a case is open continues to increase. Escalation to a broader resolution team leads to skyrocketing personnel costs. Longer problem resolution times can cause customer dissatisfaction and delayed or lost revenues.
These facts are not that surprising.
The software development process (presumably acc- ounting for the other 60% of a developer’s time) has changed greatly in the past few decades, but it continues to rely on the same manual, labour-intensive, iterative approach.
Supporting applications means solving application problems. And although systems to manage the “trouble ticket process” have been in place for many years, little progress has been made in technology to handle the heart of the issue: figuring out the root cause of the application problem.
The challenge is that the symptoms of a software problem rarely reflect the root cause. Finding the glitch is not easy when you do not know where to start looking. A single business transaction may kick off a sequence of complex processes, each of which may involve events that happen on a dozen potential servers. The root cause of the problem could be a software issue, a hardware fault, a configuration issue or even an end-user’s mistake.
A survey by Market Dynamics found that 75% of the application problem resolution cycle time is attributed to determining the root cause of the problem. This can be especially difficult when problems occur at remote user sites.
Support teams typically go through a lengthy and costly process that includes conference calls, attempts to gather information, trips to the user site, and multiple attempts to recreate the user’s environment and the problem scenario.
Although Gartner research has shown that application faults are responsible for only 40% of all unplanned downtime, any software support veteran knows the supplier is guilty until proven innocent. Clearly, a change in the application support model is long overdue.
A growing number of software suppliers and in-house teams are using low-impact application recording technologies. These look inside a running application to collect the technical information needed by support specialists for resolving problems.
The most sophisticated recording technologies can capture the user’s actions preceding a failure, the system configuration, events, application performance parameters and even the related code execution flow, and can synchronise this information on a single timeline. They can provide powerful automated analysis to dramatically accelerate root cause determination.
In this way the most costly and cumbersome steps of the problem resolution process are reduced or eliminated. The process of gathering information from end-users and system administrators at the customer site drops significantly since the capture of all necessary information is automated.
Time spent trying to replicate the problem is eliminated, since the actual problem history has already been captured. (According to the Market Dynamics report, this problem replication step is repeated seven times, on average, in the resolution process.) Since the problem history can simply be replayed, the problem resolution team can proceed directly to delivery of the fix.
Remote application recording technology has been shown to reduce problem resolution cycle times by up to 80%, with labour savings of nearly 60%. Software suppliers that use this technology have seen dramatic results.
An ERP payables application kept mysteriously crashing on start-up, but only at one customer site, and on two of six seemingly identical servers. After investing more than a month and £35,000 in a multi-engineer analysis, the support team still had not discovered a cause or cure.
After deploying remote application support technology, the team was able to solve the problem in less than two hours. Analysis of problem history logs pinpointed the root cause: access rights to the required Windows registry key, due to an inadvertent and undetected change.
A hospital management organisation faced recurring, intermittent processing delays in a web-based application. The application supplier, facing stiff service level penalties, deployed a 15-person crisis team which spent some 3,000 hours – costing £200,000 - without solving the problem.
When the company turned to application support technology, the first insights came in just a few hours. By isolating the root cause to specific components in the application server tier, the supplier was able to return most crisis team members to their normal responsibilities. Within the next few days, the remaining engineers pinpointed the problem to threading issues with their Com object, and solved it.
Had this company deployed such technology from the beginning, it could have reduced its investment in engineering hours from 3,000 to less than 100, saving more than £175,000 in this incident alone.
The characteristics that make enterprise applications so powerful and productive when all goes well makes them nearly inscrutable when things go wrong. The time has come for a change in how we support these applications: a technology approach that can accelerate the problem resolution process and reduce the number of escalated issues, so the development team can stay focused on development.
Yochi Slonim is chief executive at Identify Software