How much testing is needed before an e-commerce site goes live?

The solution

Know when you've done enough

Roger Marshall

IT director at the Corporation of London

They are right, of course. Systems should always be tested thoroughly. The main problem is to know when you have done enough. The more business-critical the application, the more thoroughly it should be tested, so how does your e-commerce application measure up on that basis?

Not very critical, you might say. The new system will only be processing a tiny proportion of total sales for some months and perhaps years. The existing processes are the critical ones in terms of volumes and will still be there to act as a fall-back in the event of problems. All of this may be true, but it misses several important points.

The e-commerce system will be used directly by your customers. There is no human interface to respond when things go wrong. Customers can be put off your e-business offering for a host of reasons, such as design, ease of use and speed of response. They are likely to desert you straight away and find another e-supplier at the click of a proverbial mouse if they are dissatisfied.

There is much public distrust of the security of e-commerce operations, such has been the poor performance of some of the biggest players and the resulting bad publicity. Either the pressure to get to market has caused corners to be cut or the developers simply didn't recognise the risks and failed to test adequately. Whichever is the case, you have got to go that extra mile to make sure it doesn't happen to you.

How much of your development time should be taken up by the testing? It is impossible to give an answer. There are too many variables. A better question to ask is how much effort you should put into planning the testing. The answer is a great deal, right from the start, with your best people working on it. Make sure you get commitment from the highest level to carrying through the plan and don't allow the inevitable pressure to get to market deflect you from completing it properly.

Build it into the project from the start

Roger Rawlinson

Head of e-business, The NCC Group

The pitfall for not testing thoroughly before you go live is that your application simply will not work.

Testing should not be an afterthought but built into the project from the start. It should incorporate (pre-release) development testing, functional testing, performance testing and security testing. Coupled with your testing programme, robust development procedures should also be put in place such as version and release control and stable platforms.

It is important to understand that testing detects errors, but does not detect their absence, so think carefully about your testing strategy and what you want to achieve. Make sure that testing activity can be mapped to the specification. How much you test ultimately depends on the level of assurance you need, this will depend on the business impact of failure. It is, however, not a good idea to get your customers to do the testing by releasing poor product! Two key benefits of e-commerce are ease of communication and speed of access to customers, they must however be balanced with the impact of failure. There is no hiding place, it is there for all to see and your brand will feel the effects.

Set aside 25% to 30% for testing and "time box" this so that it does not get squeezed as the project is delayed and the day live date approaches.

Allocate most of the development time

Andrew Davies

Visiting professor in information systems, Cranfield School of Management

You should set aside much more than the average organisation allocates at present.

Our experience is that traditional applications need to have at least 50%, and often much more, of software development time devoted to module and system testing, with at least 40% of total project time allocated to user acceptance testing. Many project "over-runs" have been caused by underestimating the time required for testing.

"E" has raised the stakes, with so much more scope for things that go wrong to cause major business issues. I am sure that most readers who have used the Internet have a dossier of horror stories about performance or software problems, and the media have described many awful cases. The only way to stop these problems is very thorough testing, with much more detailed planning of the scenarios to be tested.

I have used automated testing tools and was delighted by their effectiveness and the productivity gains we achieved, despite the up-front learning and set-up. With e-applications, my view is that the question is not "should we use these tools?" but "how do we ensure that they are used enough?".

I think that we will find that the time to be allocated to testing e-applications should be most of the development period.

Should take place on an ongoing basis

David Taylor

Certus

E-commerce applications have to be fast if they are to be effective. However, they also have to be robust. Application testing is, therefore, important, and it should take place on an ongoing basis.

In terms of who does the testing, many of these testers dress up what they do - e-commerce developers will be able to test that most systems work correctly. Personally I would only use external specialist testers for mission-critical applications that must work 100% right from day one. The pitfalls are obvious, but these must be balanced with time to market. So if you are going to use such a tester, then be clear what they will do, and when they will deliver.

Overall, about 20% of time should be taken up testing, but this should be built in to the overall cycle, and not left to the end.