The ECPerf benchmark, released last year by Sun, was supported by a number of Java application server suppliers including BEA Systems, IBM, Sun Microsystems, Oracle, Sybase, iPlanet, Hewlett-Packard, IONA, Informix Software, Borland and ATG.
But according to Giga, Java application server suppliers have been unwilling to publish an "apples to apples" (comparable hardware and software configuration) ECPerf benchmark.
Giga advised any company with both the required expertise/resources and funding to perform the ECPerf benchmark on their own corporate hardware, database and operating system specific configurations in order to get the most accurate benchmark comparisons.
It noted that to perform such benchmarks users would need at least one to two J2EE technologists, one system/application server administrator and one database administer (DBA) for a minimum of two to three weeks.
Giga analyst John Meyer said: "Once the initial set of internal benchmarks have been completed, companies should invite [in] the application server vendors to allow them to attempt to tweak the performance of the configurations, assuring the most accurate benchmark possible."
Meyer said that in most cases the tier-one application server vendors would be much more likely to provide resources to tweak an existing ECPerf benchmark for their application server compared to running a complete ECPerf benchmark for a prospective customer.
The entire ECPerf benchmark can be downloaded from Sun at java.sun.com/j2ee/ecperf/download.html.