Feature

The benchmarks don't work

There are few things more pitiful than having to listen to a vendor whining about how benchmarks are "fixed" then to hear the same vendor use the next breath to proclaim its superiority in (you guessed it) Benchmark X!

Curiously there's one set benchmarks that no-one seems to want to talk about any more. The TPC ( www.tpc.org) benchmarks were once the darling of vendors like Oracle and Sun. The problem now is that Microsoft dominates the TPC benchmarks.

Naturally, it stands to reason that if Microsoft's technology can take positions 1, 2, 3, 5, 6 and 9 in an all out performance benchmark (TPC-C) then there must be a problem. Quite a problem, if you're Oracle or Sun. It gets even more unpleasant if you look at the price performance rankings for the same benchmark where Microsoft holds every single one of the top ten price performance slots. Well naturally you'd smell a rat wouldn't you? But the real shocker is that Microsoft didn't submit these results - IBM, Compaq and Dell did.

However there is some good news for those Java app servers and non Microsoft databases that might be feeling a little left out. ECperf ( http://ecperf.theserverside.com/ecperf/)is on hand, specifically designed to test J2EE application servers. There's also a comprehensive document explaining ECperf entitled Eight reasons why ECperf is the right way to evaluate J2EE performance ( www.theserverside.com/resources/articles/Why_ECperf/why_ecperf.pdf). There is a ninth reason why vendors might want you to look at ECperf, which I am surprised wasn't mentioned - perhaps it goes without saying: The ninth reason is Microsoft doesn't have a J2EE app server.

Another interesting titbit is to be found in this document: It seems that database vendors that brag about their ECperf benchmark scores are misleading us just a little bit as ECperf isn't actually designed to measure database performance. The Eight reasons document explains in section 3.3 (and I quote verbatim) "ECperf is not designed to test DBMS performance and scalability. These are adequately measured by other standard workloads such as TPC-C, TPC-D and TPC-W[2]" . Now I'm sure that this is an oversight on the part of the braggers - after all they would never seek to deliberately mislead us would they? Don't you think that if the Java app server and database vendors were so sure they could beat Microsoft in price/performance they'd pick a field on which Microsoft could play?

Further information:
Related Ovum research : SoftwareArchitectures@Ovum .
For more information e-mail: info@ovum.com or go to www.ovum.com .

Email Alerts

Register now to receive ComputerWeekly.com IT-related news, guides and more, delivered to your inbox.
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

This was first published in July 2002

 

COMMENTS powered by Disqus  //  Commenting policy