The Specmail2001 benchmark has been devised to measure the throughput and responsiveness of e-mail systems and predict the scalability of the system in terms of the number of users supported.
The system was developed for Spec by Compaq, Mirapoint, Openwave Systems and Sun Microsystems. The tests have been developed over the past two years with input from Netscape, Critical Path, SGI, IBM, Hewlett-Packard and the University of Pava in Italy.
As the database of results builds up, Specmail2001 will provide a comparison of performances between systems. However, Ed Brill, senior manager for messaging and collaboration at IBM's Lotus Software group, warned that the results should be seen as a guide and not a directory. "It is like measuring a car's performance between 0 and 60mph - it shows raw performance figures but does not offer a comparison of other features within the system," he explained.
"Manufacturers in several areas have played the benchmarking game for years and, though the Specmail2001 figures may be useful to ISPs in the hosting space, it is not a market that Lotus Domino targets."
To date, only a few companies have posted validated results for their e-mail systems - primarily Mirapoint, but also Novell and IBM. It will be some time before the figures show sufficient data for meaningful conclusions to be drawn. Even then, Brill claimed it will only offer a speed comparison in broad terms.
"Highest speed does not equal best performance because it depends what is done to the e-mail other than routing," he said.