In the race to get online, companies are often tempted to skimp on the amount of testing they do. The assumption is that speed-to-market is the primary consideration. Not true. In a competitive environment, speed to market is important, but nowhere near as vital as holding onto your customers. If your site crashes, or runs slowly, you've got no chance of this.
There's nothing new about developers rushing the unfinished article onto the market. It's practically a pre-requisite in the software industry. This is why few IT managers buy the first version of any Microsoft product that's launched. Practically all IT projects run over their deadline for system development and often some time is recouped by skimping on the amount of testing that takes place before the system goes live.
Actually, the slogan-mongers who say e-business is a totally different ball game are half right when it comes to testing. You can skimp on the testing of a client server project development and, if anything goes wrong, it's unlikely to be a major catastrophe. Launch an untested e-business on the market and, should the system not work, it can be fatal.
"If you've gone live with an internal application and it falls over, your users will report it and you'll be able to fix the problem," says Gareth Evans, marketing director of testing tools maker Cyrano, "If it's an e-business, your customers won't tell you and they won't come back."
Doing business on the Internet involves much wider risks because you've no idea how many users you are likely to have and, given the range of platforms that will run your system and the number of devices used to access it, "guesstimating" performance will be incredibly difficult. This is why it is far more important to get an e-business platform tested as thoroughly as possible before launch.
"You only get one chance in e-commerce," says Evans. "Blow that and you can kiss good-bye to all that money you spent trying to make people look at your site."
In the business to consumer e-commerce sector, considerable amounts of money have to be spent on marketing to gain customers. Often, the marketing campaign has booked TV and radio advertising that is time-dependent, which explains why some companies have been tempted to launch their sites without rigorously testing their capability.
Theconsumer goods site Jungle.com, for example, ground to a halt on the day of its launch. Its founder, Steve Bennett, attributed this to the company being a victim of its own success. Having spent an estimated £10 million on advertising and marketing, more users than had been predicted flooded the site and thousands abandoned their purchases in disappointment. Since Bennett has a background in IT - he once headed an IT distribution company - it's probably kinder to say that the marketing spend forced him to launch the site on schedule, even though it clearly hadn't been tested accurately.
"E-businesses that flop on their launch aren't victims of their own success," says Evans, "they're victims of their own stupidity. Even if you don't know how many customers you have, you can fairly accurately gauge the demand and prepare for it."
The essential disciplines of testing an e-business's robustness are the same you'd apply to the previous generations of computer system. There are four key issues, argues Evans: function, performance, availability and security.
Jungle.com failed because it did not have enough bandwidth to cope with all the customer enquiries that were flooding onto its servers. The computing resources and network capacity needed could have been calculated more accurately if the company had made a better guess at the number of users expected on the site.
The Halifax share dealing Web site is a good example of an e-business that didn't pay enough attention to function. After making a change to the application, the site's managers went live without testing the impact of the new routine on the system. Consequently, end users found that they could suddenly access, and even trade, other customer's share portfolios. A breach of security that is hardly likely to inspire confidence among Halifax's customers, though they claim to have only lost three customers after this episode.
"This was a fundamental mistake. You have to test a site every step of the way," says Evans. "Every incremental change necessitates the site being tested again."
If this sounds laborious, it also offers the key to making the test process more streamlined. You can test every module of a system as it is being developed. When you think you have the finished article, all that needs testing is the integration of all these modules, says Nange Yianni, technical manager for testing at Compuware.
But isn't the testing of software modules in isolation meaningless? Surely it's the finished article, where they are all integrated and interrelated, that is relevant.
"Yes, the finished article is important, but you can get 90 per cent of your system testing finished before you reach the final stage," he says. "That way you don't have to enter into a lengthy period of testing and risk holding back your launch date."
One of the keys to assessing performance lies in not just guessing how many users will hit your site - which can never be an exact science - but what type they will be. Casual visitors, for example, make much less of a demand on your system than committed buyers. Once a visitor starts to make a purchase, they move from being a passive observer of published information, to someone interacting with the company database and order purchasing system. If you played safe and assumed all visitors were buyers, you would be making an expensive mistake, because the amount of processing power and server capacity bought in would far exceed your needs. "E-businesses are loath to spend their budget on capacity they don't need," says Yianni. "After all, there's so much more they could be spending that money on. Like marketing and security."
Before going live, he advises you to allow selected customers to use the site and observe patterns of usage. This will tell you, for example, what percentage of users are involved in transactions at any one time, which pages are the most popular, which products most in demand and which parts of the application seem to function most laboriously. Some companies launch their sites without making their existence widely known. This gives them time to study the site in action before their marketing machine starts to buy them customers.
The real challenge lies in integration of a traditional business system with its new Web presence. If the clicks-and-mortar traditional businesses enjoy the advantage of having an established culture of system testing, this is neutralised by the complexity of their e-commerce systems. "The trouble with adding a Web front end to a traditional back end business system is you end up with a network that's an eclectic mix of computing environments and platforms," warns Chris Ambler, global test leader of Tanning Technology.
In these circumstances it becomes very difficult to identify where a bottleneck is building up as orders are taken on a Web server, passed to an inventory system that resides on a mainframe and then made to link up with, say, a finance system on another completely different type of server.
This is why many of the testing tools for e-commerce fail to highlight problems. They don't look at the big picture, according to Colin Armitage, MD of Original Software, which claims to tackle the real problem facing clicks-and-mortar e-businesses. "It's all very well testing the user interaction of your site but that doesn't tell you how good it is as a business tool, " says Armstrong. "What good is a two second response time if the goods don't actually get delivered for two weeks?"
The present generation of testing tools aren't good enough, argues Armitage. They were developed for the less complicated world of client server, and modified for e-commerce. Unfortunately, they don't take into account the complexity of the network over which a typical e-commerce application runs.
Most testing tool vendors offer record-and-playback facilities, he says, where every possible transaction that can be made on a site is mimicked by a software routine, and response times monitored. What businesses really need is accurate operational feedback, he says. "The new generation of testing tools will now give an accurate picture of the whole chain of events, from initial submission, to order processing to despatch of the goods," says Armitage.
If Tesco's home shopping experiment hadn't relied on record-and-feedback tools, he says, it wouldn't have run into the problems it experienced on launch. The problem Tesco had was that half its information seemed to get lost between the Web server and the mainframe used to control stocks. Armitage advises companies to simplify the process by using the same platforms to run their Web server as they would to carry out the heavy duty number crunching at the back end, the traditional mainframe and mini computers. Using an IBM AS400 at both ends, for example, will allow you to see everything from initial submission to order processing to despatch, says Original, which sells AS400 testing tools.
In an environment where customer loyalty is nil, the integrity of a system is critical. The further down the development line you get before identifying problems, the more expensive they are to rectify.
Catch a bug in design, says IBM, and it will cost you $1. Catch it in testing and the cost is $10. Catch it in systems testing and it's $100. Once it's made it onto your system it will cost you $1,000. They haven't arrived at a figure yet for how much it'll cost you if it's on your e-business system. Which in a way, sums up the whole testing process in e-commerce. It hasn't been able to keep up with the pace of change.
The ten disciplines of E-system testing
1. Architectural planning
2. Site reporting
3. Database tuning
4. Code analysis and tuning
5. Representative staging
6. Load and regression testing
7. Server tuning
8. Spike readiness
9. Recovery planning
Tips on testing
The UK's three biggest testing failures so far
Type: Business-to-consumer, goods
Failure: Goods not delivered for weeks, lost orders
Result: Lost customers
Reason: Lack of integration between Web server and mainframe
Site: The Halifax
Type: Share dealing online
Failure: Application bug allowed clients to access other clients' accounts
Result: Lost customers. Potential fraud
Reason: Lack of application testing
Type: Business-to-consumer, home entertainment
Failure: Slow response times
Result: Lost revenue estimated at £100,000 - according to MD Steve Bennett
Reason: Lack of capacity planning