Does InfiniBand interconnect score over options such as FC, 10 GbE?

InfiniBand interconnect has always had a strong presence in the storage domain compared to its close competitors. In fact it has been the de facto interconnect for high performance computing (HPC) networking for a while. Not only that, in recent times, InfiniBand interconnect has been able to secure a stronger presence in the domain thanks to its expertise, leaving the competition far behind. InfiniBand interconnect’s key competition in this domain are Fiber Channel (FC) and 10 Gigabit Ethernet (10 GbE).

For understanding this comparison in detail, let us take a look at the cost factor. Here is a quick calculation to figure out the cost per Mbps for InfiniBand interconnect, FC and 10GbE. I have selected 4 Gbps FC, 10 GbE and 20 G DDR (Double Data Rate) InfiniBand interconnect for this task. Also, I have taken the price for the cheapest HBAs (Host Bus Adaptors)/cards and switches for all the three variants (see table below):

The formula is as follows: price of the HBA + (price of the switch / number of ports in the switch) / speed of the card in Mbps. This will tell us what the least possible price for a single connectivity is for each interconnect (including InfiniBand interconnect). See the table below for the results:

These figures make it clear that while the old debate over 10 GbE and FC – which one is better - continues, InfiniBand interconnect is emerging as the silent winner. In this regard, we have already taken a look at Infiniband as the interconnect and associated benefits. Now, let us quickly glance through the features as well as limitations of 10GbE and FC as compared to InfiniBand interconnect.

In order to understand the advantages of InfiniBand interconnect, let us consider FC first. FC has always been the preferred choice for large storage area network (SAN) deployments due to its native HBA capabilities. In its case, the FC card is detected as a disk controller by the client machine and can connect to the exported SAN target like a native small computer system interface (SCSI) device. You do not require any additional software initiator to connect to the client, which saves from the overheads of protocol translation at the software level.

But compared to InfiniBand interconnect, FC has its limitations. For instance, even if it’s simpler to deploy, it’s slower compared to 10 GbE by its native speed. Today FC is only available in 4G and 8G variants whereas, as the name suggests 10 GbE can give up to 10 Gbps of theoretical max speed.

Keeping aside InfiniBand interconnect, if we compare the price, 4G FC cards are more expensive than 10G cards available today. The 8Gs are, in fact, exorbitantly expensive. Another problem with FC is that it’s not well suited for IP. Internet Protocol over FC (IPoFC) is possible up to some extent but it’s not a preferred medium yet. Thus, you will need to have a separate Ethernet card in your servers for regular networking.

And that’s where 10G scores over FC. Its other advantages include the fact that it can be used for TCP/IP (Transmission Control Protocol / Internet Protocol) networking as also for storage interconnects due to high speed. But the limitation is that it works over IP and you can only use SCSI commands using iSCSI (Internet SCSI) protocol. Further, this protocol being non-native, has significant overheads. Yet, the throughput is significantly better than 4G FC and the price is still on the lower side. Moreover, if you compare it with 8G FC, the actual throughput which you get is still comparable and yet price is significantly low. The biggest advantage with 10 GbE is that being Ethernet, it can even communicate very easily with your existing network providing easier provisioning, migration and compatibility. However, the fact remains that InfiniBand interconnect scores over all others.

About the author: Anindya Roy is a tech evangelist at Netweb Technologies, and has worked on open source private cloud platforms. His specialization areas include HPC, cloud computing, and virtualization. Roy also has an extensive experience in application evaluation as well as testing. Write to him at [email protected].

(As told to Jasmine Desai)