Spec (Standard Performance Evaluation Corporation), the non-profit-
making benchmarking organisation, has expanded its range of tests
to encompass e-mail server performance, writes Eric Doyle.
The Specmail2001 benchmark has been devised to measure the
throughput and responsiveness of e-mail systems and predict the
scalability of the system in terms of the number of users
supported.
The system was developed for Spec by Compaq, Mirapoint, Openwave
Systems and Sun Microsystems. The tests have been developed over
the past two years with input from Netscape, Critical Path, SGI,
IBM, Hewlett-Packard and the University of Pava in Italy.
As the database of results builds up, Specmail2001 will provide a
comparison of performances between systems. However, Ed Brill,
senior manager for messaging and collaboration at IBM's Lotus
Software group, warned that the results should be seen as a guide
and not a directory. "It is like measuring a car's performance
between 0 and 60mph - it shows raw performance figures but does not
offer a comparison of other features within the system," he
explained.
"Manufacturers in several areas have played the benchmarking game
for years and, though the Specmail2001 figures may be useful to
ISPs in the hosting space, it is not a market that Lotus Domino
targets."
To date, only a few companies have posted validated results for
their e-mail systems - primarily Mirapoint, but also Novell and
IBM. It will be some time before the figures show sufficient data
for meaningful conclusions to be drawn. Even then, Brill claimed it
will only offer a speed comparison in broad terms.
"Highest speed does not equal best performance because it depends
what is done to the e-mail other than routing," he said.
www.specbench.org/osg/mail2001/docs/whitepaper.html