Yet the market for more regular types of testing is still buoyant, for two main reasons. In the first place Y2K alerted many companies to the need for proper testing. Secondly, the rise of e-business in all its forms is compelling enterprises to ensure systems run at optimum levels.
In particular, the AS/400 user base made good use of Y2K testing solutions due to its widespread legacy investment. The AS/400 market will remain a strong market for testing tools, due to the growing sense that it will play a major part in the B2B market. This is despite the fact that many of the new generation of test software are being released for web environments. The suppliers selling test suites into the AS/400 market see synergy here between the server's original testing requirements and the new demand for web performance evaluation.
In fact, leading testing tools supplier Mercury Interactive has preserved all the functions of its earlier AS/400 products within new web suites. This means that tools such as functional and load testers now extend to cover Corba, Dcom and the RMI (a Java client technology), as well as all current planned versions of browsers, according to Andy Crosby, European field market manager for Mercury. At the back end, all existing scripts and services for platforms like the AS/400, and other IBM servers, have been retained. Tools for software like SAP's enterprise resource planning environment have also been subject to the same treatment.
In terms of the Mercury toolsets, this applies to products like LoadRunner 6.5, which Crosby describes as the de facto standard in the marketplace for load balance testing. He further claims that the user base for LoadRunner is greater than all other testing tools combined. He also points out that by taking a spread of market figures from analyst firms, such as the Forrester Group, Gartner Group, and Jupiter, it can be shown that Mercury's overall share of the worldwide testing market ranges between 45-90 per cent, depending on the different categories.
Mercury claims a 90 per cent share of the SAP testing market. It is currently seeing a 300 per cent growth in sales for IBM platforms deployed for e-business. Crosby could not break out the AS/400's contribution to this growth, but reckons that it is a box to watch in the e-business space, with a lot of activity being witnessed.
Beyond the test drive
The most common technique for systems testing has been something called record and playback. This involves a macro style script defines workloads for a system, which can then be verified against predicted results. It provides a kind of benchmark for a system's raw performance characteristics. Yet the suppliers have started to realise that record and playback, however valuable, is not capable of covering all testing needs.
'Expected results represent only a fraction of testing requirements. There are database and other programs being executed, for which you have no pre-conceived idea about performance. How do you prove that these are performing well?' says Colin Armitage, managing director of Original Software, a testing tools provider specialising in the AS/400.
Original is a relatively new company and has focused all its initial efforts on the AS/400 market. Armitage explains that a good testing tool should be a push button affair that checks through the application and the database until the end of the transaction is reached. He claims that the company's TestBench400 product for the AS/400 was originated on this basis. TestDrive is the company's product for record and playback, with the name of this toolset revealing much about the useful but ultimately limited role of this technique.
Crosby admits that the record and playback tools are often the lead sale into an account for Mercury. But the rise of complex IT environments and the extra demands of e-business mean that deeper forms of testing are necessary. Crosby prefers to describe this as measuring the execution of business processes against parameters which, combined with other scripts for record and playback, will allow you to track performance across the whole lifecycle of a transaction.
Mercury is busy developing its LoadRunner product to include extra so-called monitors for various platforms. These monitors provide a lifecycle view of a given system and can deliver this information in real-time to the relevant managers. One recent monitor to be added is for WebSphere, IBM's premier e-business infrastructure and development platform. Such functionality will be needed for testing the demanding e-business applications to come, such as online industry exchanges and other high-volume transactional environments. Crosby summarises this approach as root cause analysis, where highly complex processes are dynamically tested over multi-tier hardware and software configurations.
To match the increasing complexity of IT environments, the test tool suppliers are developing some highly specialised products. For instance, Mercury has added a low-end B2C web site testing product called Astra to join its earlier suites, such as WinRunner and LoadRunner. The company is also looking at the emerging business-to-employee segment, which focuses on multi-tier corporate intranets.
Original has also shifted its product focus to the web and e-business testing market, while supporting previous toolsets. In September the company starts shipments of its TestWeb product for browsers, according to Armitage. And two months later it will ship a product called TestGUI. Further out, Original is planning to release a toolset called TestJVM that will be able to check the performance of any Java code executed on a Java Virtual Machine. On the AS/400, such tools will be capable of tracking a transaction through to its termination, with all technology elements tested.
Armitage reveals that in the two years since its release, around 100 AS/400 sites have bought TestBench400, with Cable & Wireless being a good example among others (see box). He says that there is a pent-up demand for the newer web testing tools, although it is not yet the case that floodgates have opened. In fact, six of Original's larger AS/400 customers have ordered TestWeb. Armitage does believe, however, that much more interest will be generated once the larger body of organisation using AS/400s start up e-business projects by the end of the year and beyond.
While Original has been focused solely on the AS/400 up till now, the company is looking to expand its server coverage to back up its web testing tool strategy. Clearly the company cannot at present provide alternative servers with the end-to-end testing functions delivered to the AS/400 user base. Armitage says that the next platform to get the full treatment will be Windows NT, which is prevalent at many AS/400 sites.
The potential of the AS/400 testing market is obviously there, as vendors relatively new to this business are seeking to get in on the act. One example of this trend is Rational Software, a major ally of IBM in application development technology and methodologies, and also known for application lifecycle testing.
Yet the company also has a range of other testing tools, including a record and playback product called TeamTest. This represents the baseline testing function offered, according to John Watkins, UK product manager for testing at Rational. Higher up the chain, the vendor offers extra features, with products like TestStudio and RequisitePro. PerformanceStudio is at the high end of the Rational testing software portfolio, which measures things like load stresses, volume testing, and the scaleability of systems.
Watkins claims that the differentiator for Rational is its integration of test tools with a view to the lifecycle of an application. This includes the design for test philosophy at Rational. This means taking a client-server to web architecture design, producing test scenarios and running these against the server logic before the system is constructed. Watkins observes that Rational's activity in the AS/400 testing tool market does not represent its bread and butter business. Its involvement is often with companies that want to test legacy applications for use as the back end to e-business systems.
The increasing complexity of IT environments, in particular those geared to e-business, means that testing in all its forms will become even more important. One of the main tricks with e-business is that systems must be more than highly available and capable of sustaining a wide range of stresses. And one of the only ways these criteria will be met is through testing - before, during, and after the live launch of applications. l
Automated systems testing can bring many benefits to end-users. Take the example of Castrol UK, part of the Burmah Group, and one of the leading suppliers of lubricants. As part of a global operation, the company experiences regular changes to its business processes - reflected in the need to alter and subsequently test its IT base.
As an extensive AS/400 user, Castrol UK looked around at the available tools and selected the TestBench400 suite from Original Software. This toolset has helped the company successfully cope with the demands of enhancing its systems and supporting the business processes.
One of the main reasons TesBench400 was chosen is the software's capability to test both batch and interactive programs with rules based cases, in addition to the data protection feature that helps avoid data corruption. In another example, also a Testbench400 site, the Royal Bank Invoice Finance company claims to have reduced test cycles by 83 per cent.