Better test for gauging supercomputer performance

Market research company IDC has claimed that a measurement system it has created could give supercomputer customers a new way to...

Market research company IDC has claimed that a measurement system it has created could give supercomputer customers a new way to measure the performance of those systems and evaluate them more accurately before purchase.

The IDC Balanced Rating is a supercomputer and high-performance computing (HPC) test that measures the performance of systems by testing their processor, memory and scalability performance, said Earl Joseph, director of research for high-performance computing at IDC.

The tests are performed only on systems that are installed and running at customer locations, rather than at vendor locations to ensure accuracy, so that the tests will return real-world performance results, Joseph said. HPC systems are used in vehicle design, drug and genome research, weather forecasting and a number of other areas.

That there was a need to document the real-world performance of these systems became clear to IDC after it formed the High Performance Computing User Forum, a group that includes 1,300 HPC users, to try to better understand what issues users were facing, Joseph said.

The members of the forum said that they needed a way to know what the actual performance of these systems was, not just the peak performance that other tests measure.

The Balanced Rating, IDC's answer to those users' questions, tests the processor performance, memory capacity and scalability of 893 HPCs installed worldwide. The processor uses the Specfp_rate_base2000 benchmark to test floating point operations, the SPECint_rate_base2000 to test integer calculations and the Linpack Rmax test which solves dense linear equations, Joseph said.

The memory system capacity test examines how much data can be moved in and out of memory in a given period using the STREAM TRIAD test. Scaling capability is determined by using total processor count and total system memory and interconnect bandwidth.

The systems tested in this way have been broken down into four categories: Technical Capability Computers, which includes systems designed for the most demanding tasks; Technical Enterprise Computers, systems that cost more than $1m; Technical Divisional Computers, systems that cost between $250,000 and $999,000; and Technical Departmental Computers, which are machines priced at less than $250,000.

The top five overall systems, as measured by the Balanced Rating, are the NEC Earth Simulator and IBM's ACSI White SP Power 3, followed by Hewlett-Packard's LANL AlphaServerSC45 P-2, PSC AlphaServer SC45 and CEA DAM AlphaServer SC45.

The rankings are available free online at the HPC User Forum Web site, located at

The site also offers a custom filter of the information to allow users to rank systems based on their needs.

The Balanced Rating includes six tests, but Joseph would like to increase that number to 12 in a few years.

The performance numbers will be updated every quarter, and, starting in November, the list will also include the price of each of the systems tested

Read more on Data centre hardware