Cisco may not have said it directly, but it appears to have abandoned InfiniBand.
The company is getting behind Data Centre Ethernet in a big way with its Data Centre 3.0 strategy, which is designed to provide converged networking for the datacentre. It has taken standard high-speed Ethernet and added standards-based extensions for services such as quality of service and lossless transmission.
"The adoption of Data Centre Ethernet will help enable a unified fabric in the datacentre, reducing costs and simplifying management," the firm said in a statement.
"Many high-performance compute applications already run over Ethernet. With the enhancements to Ethernet and migration to 10 Gigabit Ethernet [or 10GbE], Cisco believes that the subset of applications that require high bandwidth and low latencies can be supported effectively on Ethernet," the statement said.
That does not seem to leave much room for InfiniBand.
It is a strange position to take for a company that was piling millions of dollars into InfiniBand just a few years ago. The firm spent $250m on InfiniBand switching firm Topspin Communications in 2005, and upgraded its line of switches with that company's technology the following year.
It made it clear even then that it was unifying the two in the network, so that Ethernet and InfiniBand could be used together in the same switch and configured with the same management console. Now, Cisco appears to believe Ethernet has evolved to the point where other protocols such as InfiniBand are no longer necessary.
"Cisco is pushing it because it wants to do Fibre Channel over Ethernet. Ethernet is trying to build in things that InfiniBand already does," says Kevin Judd, senior staff product manager at InfiniBand SAN switching vendor QLogic and co-chair of the InfiniBand Trade Association's marketing working group.
Darin Stahl, lead analyst at industry watcher Info-Tech Research Group, agrees that 10GbE still has a lot to prove. "The approval happened in 2002, and the adoption rate has been much slower than I would have expected given the promise of what it could do. The biggest issue around that is the cost per port," he says.
Brian Sparks, who co-chairs the IBTA working group along with Judd, says that the port cost for high-speed Ethernet currently makes it impracticable for many datacentres. The IBTA's figures put 10Gbit/sec InfiniBand at $22 per Gbit/sec, compared with $85 per Gbit/sec for 10Gb/sec Ethernet. "We do not project 10GbE to become competitive until at least 2010," he says.
Sparks also alludes to rumours of Cisco's forthcoming entry into the blade server market. "Cisco is doing things that make the server manufacturers do a double take. The likes of Hewlett-Packard, IBM, Dell and Sun are looking for a way to compete with Cisco solutions, and see InfiniBand as a way to compete against Cisco, at a better cost," he says.
Perhaps, but server suppliers such as Fujitsu are nevertheless promoting the use of high-speed Ethernet, even in high-performance computing environments normally considered a sanctuary for InfiniBand. A white paper from the company points to the ratification of the 10GBase-CX4 10GbE over copper standard in 2004 as a critical point in time for the standard, because it dropped the cost per port.
He adds that InfiniBand's low latency design makes it particularly well-suited for the high-performance computing space. 10GbE dropped the half-duplex mode that required CSMA/CD to handle packet collisions, and moved to full duplex instead. But the latency of the ratified, unmodified protocol is still higher than InfiniBand.
"There are 10GbE suppliers applying technologies to lower the latency. Right now they are getting latency of three microseconds, but that is still three times the latency that InfiniBand is offering," says Judd.
Nevertheless, Hamish Macarthur, founder of market watcher Macarthur Stroud International, is sceptical about the protocol's chances.
"The main system suppliers are starting to close back in again to more direct attached storage. A good example of that is blade servers, some of which now have 2.5in disk drives," he says, arguing that suppliers are trying to gain end-to-end control of server and storage space, which limits the scope for InfiniBand.
"The issue of what happens to InfiniBand presents a big question mark. There may be lessons from that for the development of new architectures, but as a protocol itself it has limited time." Macarthur says.
If Ethernet does encroach on InfiniBand's market, it will take some years before it does significant damage. And the inevitable shelving of capital-intensive infrastructural IT projects for the next year (at least) will do nothing to help it. Nevertheless, the constantly evolving standard should not be discounted.
This was first published in February 2009