Benchmarks need a pinch of salt

Last week IBM announced it had pipped Hewlett-Packard to the top spot on the TPC benchmarks for non-clustered servers.

Last week IBM announced it had pipped Hewlett-Packard to the top spot on the TPC benchmarks for non-clustered servers.

In essence, the results show that, where price is no object, IBM would be able to configure a datacentre server based on its pSeries 690 which could beat last month's leader, the HP Superdome.

HP and its erstwhile partners Intel and Microsoft have a policy to beat expensive propriety Unix systems both in terms of performance and overall cost per transaction. So while IBM attained a TPC benchmark of 680,613.12 tpmC, the cost per transaction was $11.13/tpmC.

When I spoke to him last week, Richard George, HP's business manager responsible for the company's business server line, was keen to point out the fact that while IBM did indeed outperform HP by about 3%, the IBM configuration was more than $1m more expensive. Moreover, Rogers emphasised that IBM's system was proprietary (ie it used Unix and IBM's Power 4 processor) and that the system in question was at the end of its life, giving what he suggested was absolutely no upgrade path for users.

Not to be outdone, IBM referred me to the HP Superdome specification submitted to the TPC. It transpires that the HP machine is configured with 64 processors, while IBM's "only" uses 32. This has an impact on software costs.

So while IBM is more expensive, the cost of licensing DB/2 and AIX software for its 32-way system is at least $500,000 (£310,000) less than 64-bit SQL Server and Windows 2003 on the HP Superdome.

George conceded that the "street price" of DB/2 is more attractive than SQL Server, but he put this price difference down to the ongoing price war between IBM and Oracle over database licensing.

I asked Simon Roberts, pSeries product manager at IBM, why the IBM system was significantly more expensive than HP's offering, for what appears a rather insignificant 3% improvement on performance. It is all down to the storage, he said.

IBM used "proper" enterprise storage based on its SSA disc subsystem, while HP used cheap and cheerful SCSI discs. Roberts said no datacentre manager would base their datacentre storage on SCSI - most opt for expensive kit such as Fibre Channel and storage area networks.

At HP, George admitted the benchmark was based on its cheaper MSA range rather than the datacentre-class EVA family of storage products. He could not give me a reason why this was so.

So what does this prove? Is HP trying to pull the wool over our eyes? Is IBM having a $1m laugh? It certainly seems so, on both counts.

Neither server tested is currently shipping. The HP Superdome is configured with Intel's next-generation Itanium chip, Madison, and is not due to hit the market until October. IBM's machine will ship on 31 May but the software patches users will need to apply to achieve its record-breaking TPC results will not be available until October.

What is the TPC?

The Transaction Processing Performance Council is a body devoted to benchmarking transaction processing systems. To derive the number of transactions that can be processed in a given timeframe, TPC benchmarks measure the total performance of the system, including the operating system, database management system and any other related components. The TPC-C benchmark yields transactions per minutes expressed in tpmC ratings.

Read more on Server hardware