Introduction to 10 Gigabit Ethernet

10 Gigabit Ethernet is faster than its predecessors, but also requires some new tricks to implement and manage.

Networks are experiencing an unprecedented demand for bandwidth, thanks to data center consolidation and a growing number of data-intensive applications like video production, managed services, storage, VoIP and streaming media servers. Network technology has responded, moving from 10/100 Mbps to 1 Gigabit Ethernet (1 GigE). Now 10 Gigabit Ethernet (10 GigE) transmission technology is on the horizon, buoyed by a suite of IEEE standards and pushed by an increasing array of vendors eager to take the next step in network evolution. This article highlights the basics of 10 GigE.

Introduction to 10 Gigabit Ethernet

10 GigE is not a new technology. It actually started development in 1999 when the IEEE 802.3 Higher Speed Study Group formed to create the successor to 1 GigE released the year before. The primary goal was to increase speed at a reasonable cost while maintaining IEEE 802.3 Ethernet frame formats and compatibility to IEEE 802.3x flow -- basically ensuring backward compatibility with previous Ethernet standards and cabling infrastructure.

Perhaps the biggest difference between 10 Gigabit Ethernet and older Ethernet standards is the abandonment of half-duplex operation in favor of full-duplex. Half-duplex network communication dates back to the original version of Ethernet, relying on packet contention on the wire (also known as Carrier-Sense Multiple Access/Collision Detection or CSMA/CD) to determine whether the cable was free for transmission. This simple solution to a complicated networking problem was a key attribute to Ethernet's early success, but it's simply too inefficient to use at 10 GigE speeds. There is no contention in full-duplex operation, and a network node can essentially send packets at any time. However, this necessitates the use of switches to interconnect network nodes since hubs no longer work to connect network devices.

10 GigE was first ratified in 2002 as IEEE Standard 802.3ae-2002, using -SR, -LR, -ER and -LX4 optical fiber to connect network nodes and switch ports. Since then, the 10 GigE standard has been amended several times to accommodate a variety of cable types. For example, 802.3ak in 2004 specifies the use of -CX4 copper InfiniBand cabling, 802.3an in 2006 specifies familiar-looking copper twisted pair cabling (also known as "10GBaseT," "Category 6" or "Cat 6" cable), 802.3aq in 2006 denotes -LRM fiber with improved signaling, and 802.3ap in 2007 includes the use of -KR and -KX4 copper backplane. Many of the amendments are expected to be consolidated into a single IEEE Standard 802.3-2008, which should be released this year.

10 Gigabit Ethernet compatibility issues

Compatibility is still a gray area for 10 Gigabit Ethernet, and there are numerous alternatives for the physical layer (or PHY) -- the means of connecting network devices at 10 GigE. With only about 1 million 10 GigE ports shipped in 2007, it's still too early to tell which physical layer standard will be accepted by the market. High-end 10 GigE networking devices (such as switches) accommodate different physical connectivity types through the use of plug-in "PHY" modules. For example, if -LR fiber connections are called for, you can try corresponding switch modules and NICs.

Backward compatibility is another unknown that you should consider before mixing Ethernet standards. Price is certainly one important factor. While a 10 GigE port should ideally sense and negotiate the slower connection, the current cost of a 10 GigE port makes it economically pointless. "You wouldn't plug a 1 GigE server port into a 10 GigE switch -- it would never happen," said Adam Gray, chief technology officer of Novacoast, an IT professional services and product development company. However, it's still too early to say with certainty that a 1 GigE NIC from manufacturer A would interoperate properly when connected to a 10 GigE switch port from manufacturer B (or vice versa).

Still, 10 GigE and 1 GigE switch ports should operate together within the same switch. For example, a manufacturer's switch with a mix of 10 GigE ports and 1 GigE ports should work properly. The 1 GigE port(s) would likely interface the switch to slower devices and the remainder of your 1 GigE infrastructure.

Ramping up to 10 Gigabit Ethernet

While the 10 GigE standard is hardly new and a limited number of products have been available for years, it's only starting to show signs of adoption. The principle limitations of 10 GigE adoption have been price, need and interoperability. Early 10 Gigabit Ethernet products cost several thousand dollars per port. The bandwidth needs of most organisations did not even begin to justify the corresponding investment.

But even while most businesses have not yet realized the full potential of their 1 GigE infrastructures, it is axiomatic that bandwidth needs are constantly growing. More organisations are adopting Voice over IP and other bandwidth-intensive tasks like streaming audio and video. And more application demands are being compounded with data center consolidation initiatives.

As data centers are consolidated, the demand on each data center's core network increases, according to Bob Laliberte, analyst at the Enterprise Strategy Group, an industry analyst firm. Laliberte pointed out that Hewlett-Packard, for instance, went from 85 data centers down to six, each with more traffic and greater bandwidth needs than before.

Server virtualisation is another bandwidth driver, forcing the traffic from multiple virtual machines (VMs) through the I/O resources of a single physical server. Even the future emergence of Fibre Channel over Ethernet (FCoE) is seen as a potential driver for 10 Gigabit Ethernet technology.

Vendors such as Extreme Networks, Foundry Networks, Alacritech. and Broadcom are supporting 10 GigE products. You can peruse a listing of manufacturers currently involved in 10 Gigabit Ethernet product interoperability testing. Product competition not only provides a wider selection of interoperable products for solution providers to recommend, it also precipitates the price pressure that makes 10 GigE more affordable and easier for clients to justify -- especially now that standard Category 6 copper cabling is supported in addition to various optical fiber types. "Now we're starting to see companies coming out with 10 GigE products that are priced at $US900 per port or less," Laliberte said, indicating that pricing is expected to decline to about $US500 per port by the end of 2008.

Making the case for 10 Gigabit Ethernet

Only the largest data-intensive organisations, such as video production houses, Internet service providers and FCoE early adopters, are currently using 10 Gigabit Ethernet technology. "For us, it's shops that produce such a volume of traffic that they need the connectivity between switches or to very large server farms," Gray said, noting that standard business applications have not been a driver for 10 GigE at this point, and it's still more cost-effective to trunk multiple 1 GigE links when additional bandwidth is needed at key points in the network.

Another driver for the standard is virtualisation, which can strain 1 GigE links. Replacing those 1 GigE links with 10 GigE technology can ease congestion, providing bandwidth for future growth through more applications or additional VMs. 10 GigE NICs with TCP/IP offload cards can help streamline I/O on busy virtualised servers.

Expect to see 10 GigE deployment follow the same pattern as 1 GigE, with deployments starting in the core network of the data center and gradually working their way to the edge of the enterprise -- though the need for 10 Gigabit Ethernet at the edge may take years to arrive.

One of the major selling points of 10 GigE is familiarity -- clients already know the technology and have seen it evolve from 10/100 Mbps to 1 GigE, and 10 GigE should not present any type of culture shock. Still, some 10 GigE upgrades will be disruptive because downtime will be needed to install new NICs and deploy new switch ports.

Be sensitive to network disruptions and downtime, and take extra care to ensure interoperability in a test setting before deploying 10 GigE devices Adequate vendor support, OS driver availability and interoperability testing to sidestep potential compatibility issues. Another way to mitigate network disruption and deployment downtime is to implement 10 GigE in small phases starting with the least critical links.

This article first appeared at

Read more on Data centre networking