A guide to Fibre Channel over Ethernet

Can Fibre Channel over Ethernet offer the reliability and resilience of storage area networks at a fraction of the cost?

When Joe Mathis worked at IBM, there was still one problem that the IBM master inventor had not solved. He had spent his time designing ultrafast workstations that he liked to call supercomputer desktops. The problem was that he had never been able to design an I/O mechanism that could feed the computers data fast enough. Another problem was that the high-speed communications systems available tended to work over very short distances, he says.

"Then, I heard about this thing called Fibre Channel," recalls Mathis, who now works at virtual infrastructure company Virtual Instruments. He realised that this could be a high-speed I/O system that could break the physical restraints associated with current high-speed connectivity and became one of the original architects of the standard.

He presented his paper to the Fibre Channel working committee 20 years ago. Now, Fibre Channel is evolving. The International Committee for Information Technology Standards (INCITS) has a group called T11 that works on fibre standards. In 2007, the T11 group began working on a standard that would enable Fibre Channel to be transported over Ethernet. In June this year, INCITS finalised the standard for submission as a draft to the American National Standards Institute (ANSI).

Converging networks

Why would anyone want to run a perfectly good high-speed storage networking standard over Ethernet? One of the biggest drivers is consolidation, explains Stuart Bridger, Emea service and support manager for storage and SAN systems at value-added distributor Avnet Technology Solutions.

"The objective is to converge networks," he says. "It is about simplifying the multiple networks in today's datacentres. You have your SAN network, which is likely to be on Fibre Channel, separate LAN networks, and possibly even a separate high-performance computing network, each with their own physical networks, management platforms and isolated cabling systems."

Since the major chip suppliers turned their focus to multiple cores, rather than simply bumping up clock speeds, we have experienced runaway computing power. With some vendors now offering 32 cores, the I/O requirements for the average server are increasing. The idea of running Fibre Channel and Ethernet separately becomes more problematic in this case, as customers find themselves increasingly bogged down with multiple network interface cards and host bus adapters (HBAs are the interface devices used to connect servers to SANs over traditional Fibre Channel links).

Consolidating Fibre Channel links onto high-speed Ethernet networks will reduce the complexity and management overhead of cabling, as well as cutting the number of interface devices needed in a server.

Analysts predict moving from a higher-end, higher-margin protocol to a more commoditised protocol can reduce costs significantly. "The cost will be reduced by a factor of ten, because you are moving from proprietary to non-proprietary systems," says Keith Humphreys, managing consultant at networking analyst EuroLAN.

According to the Fibre Channel Industry Association, the average PCI Express adapter uses around 25 watts. Given that heat overhead is the biggest barrier to server expansion in a rack, the ability to cut the number of adapters in a server could enable a datacentre manager to use the rack space more effectively. Although we must not forget that a 10Gbit adapter is likely to use more energy and generate more heat than a 1Gb NIC.

Fibre Channel as used in the protocol has no idea that it is running over Ethernet at layer 2. All of the things we normally see in Fibre Channel, such as low latency characteristics, security and traffic management attributes, remain when using it over Ethernet, which was not originally designed with such things in mind.

The traditional Fibre Channel protocol has five layers, known as FC-0 through FC-4. FC-0 and FC-1 represent the physical and data encoding layers respectively, and it is these that are replaced in an Fibre Channel over Ethernet (FCoE) implementation. Ethernet's physical and MAC address layers, which represent the physical and data link layers in the traditional OSI stack, take over here, leaving the upper three layers of fibre channel to run over the Ethernet link. These three layers are framing, services and protocol mapping.

Data flow

The big difference between Fibre Channel and Ethernet is that the former is lossless, so all of the data sent is guaranteed to arrive at the other end, whereas Ethernet is a lossy protocol, so data packets sometimes have to be resent. This is not acceptable in a low-latency SAN environment, so architects developing FCoE had to find a way around it.

Currently, the best way to prevent the problem is to use a feature of Ethernet that enables a receiving port to send a pause request to a sending port if it is too busy to receive traffic. However, a priority flow control enhancement is being introduced that will enable Ethernet pause capabilities to be merged with quality of service capabilities, making lossless Ethernet more aware of user priority status.

The industry has developed a name for the type of datacentre-optimised ethernet needed to fully support FCoE. It calls it converged enhanced ethernet (CEE), although Cisco has characteristically adopted its own terminology, calling it datacentre ethernet (DCE).

Until these issues are thrashed out, Bridger says that FCoE will be a "top of rack" solution, limited to the first few feet of the datacentre network, but as flow control issues are sorted out in datacentre-class ethernet, we will begin to see the deployment of end-to-end FCoE networks, especially as storage arrays begin to include native FCoE capabilities.

Although some commentators view FCoE as a protocol that will operate over an entirely separate 10Gbit ethernet network, Bridger sees the real value of the protocol when used in conjunction with an array of virtualised machines. Ideally, storage array traffic would run over a high-speed Ethernet network in conjunction with virtual machine traffic from a single server, he says.

However it is implemented, FCoE is likely to have a significant effect on the datacentre network, and on the industry in general, over the next few years. It is definitely a technology to watch.

Major FCoE players

Datacentre networking specialist Brocade set out its FCoE roadmap just over a year ago, before the standard was officially ratified. The company has launched products that bring FCoE and Converged Enhanced Ethernet together, and is working with specialist high-speed ethernet vendor NetApp to ensure their products are compatible.

Cisco believes that FCoE will drive adoption of 10Gbit Ethernet networks. The company set out its own FCoE stall with the Unified Computing System (UCS), which heralded its entry into the server market, and which uses the standard for internal communications. It has also promised FCoE communications with external devices in a future software upgrade. It also offers the Nexus FCoE switch.

Emulex sells host bus adapters that enable Fibre Channel storage arrays to connect to Fibre Channel switches. It has entered the FCoE race with converged network adapters designed to enable Fibre Channel connections to high-speed Ethernet networks. Emulex was the subject of a takeover bid by Broadcom, which failed, and is now suing Emulex over a patent dispute. Broadcom is a fabless designer of networking chips, but has little or no intellectual property in the FCoE space.

QLogic is a manufacturer of converged network adapters. It launched a second-generation FCoE-capable CNA earlier this year, featuring a single basic for multiple networking functions including FCoE.

Slow FCoE adoption

Fibre Channel over Ethernet (FCoE) was only ratified by INCITS in June, so implementations are limited. Eric Sheppard, Emea programme director for storage research at IDC, believes its adoption will be relatively flat because of the economic situation.

"We are seeing that some of the expected adoption of FCoE has been pushed out until beyond the recession," he says. He expects people to become more interested in the technology in the second half of next year.

Nevertheless, given the cost saving benefits associated with the new protocol, it is not surprising that some brave customers have taken steps to make themselves early adopters. LA County, one of the biggest counties in the US, announced in January that it had taken the plunge.

The county, which was already an existing Cisco customer, replaced eight Catalyst 6500 Ethernet switches with two of the networking supplier's Is 7000 FCoE switches in the datacentre core.

LA County, which has virtualised its servers and is redesigning its datacentre, needed a new design to accommodate its alterations. It settled on FCoE running over a 10Gbit Ethernet fabric.

Spokespeople within the county's IT department argued that the move would cut power consumption by half, while also using converged network adapters to replace ethernet NICs and Fibre Channel host bus adapters.

But that was back in January. The IT department at LA County has since gone very quiet. When we called them to find out what was going on, the tight-lipped staff simply said that there were "issues" around the implementation.

What might those issues be? One hopes that interoperability is not the problem. When Fibre Channel was initially ratified, it suffered from interoperability issues as suppliers implemented the protocol differently and customers found it difficult to bolt their equipment together. But experts argue that interoperability should not be as big a problem with FCoE because it already uses two well-established standards.

Read more on Networking hardware