Los Alamos National Laboratory (LANL) is building a 128-processor cluster comprising RLX blade servers running on Intel's Xeon chips and Linux.
The lab has picked Infiniband for the high-speed interconnect between the servers, as it tests out the technology for use in larger systems.
Infiniband, which is backed by many vendors, is gaining traction in the general market. It provides a high-bandwidth, low-latency interconnect between systems that is useful in high-performance computing.
The use of Infiniband marks an effort by LANL to shy away from proprietary interconnects from the likes of Myricom and Quadrics Supercomputers World, said Steve Tenbrink, group leader of the network engineering group at LANL.
"We are trying to push open standards for interconnects where we can," Tenbrink said. "This will help us evaluate if Infiniband is the right interconnect for a larger cluster."
The system is the second cluster at LANL to use the small, compact blade servers from RLX. This type of specialised server allows customers to group hundreds of processors together in a single rack and cut down on the cables that cascade down the back of more traditional systems.
LANL used low-power Transmeta processors in its other RLX-based cluster, called Green Destiny, to test ways of reducing cooling costs and improving the stability of large computers.
While LANL has yet to get the Xeon-based cluster up and running, it plans to use the system to run some of its nuclear simulation software. However, it will depend on how well Infiniband works with a large number of computers.
"You have to start somewhere," said Tenbrink. "The problem is that as you scale higher and higher, interconnect performance tends to get worse and worse. You really have to be careful how you address that problem."
Read more on Data centre hardware
Evolution in action: How datacentre hardware is moving to meet tomorrow’s tech challenges
InfiniBand networking for storage and converged data centers
Evaluating Fibre Channel, InfiniBand and Ethernet protocols for network virtualisation
Understanding NAS interconnects: Fast Ethernet and Gigabit Ethernet