Feeling the pinch with yourGigabit Ethernet network backbone?
Having a few too many packet collisions? Network congestion causing
you problems? If you are answering yes to these questions, you
should spare a thought for Joe Lawrence. The principal architect at
fibre optic wide area network provider Level 3 is dealing with
traffic an order of magnitude greater, and when he feels the pinch,
it really hurts.
Lawrence has to manage traffic volumes in his core network that
would make your hair curl. Packets flow through the heart of his
infrastructure at hundreds of gigabits per second (gbps).
"It is not a question of whether we will have a need for a
solution by 2010, it is a question of when the solution we need
right now will show up," he says.
The year 2010 is important for Lawrence because it is the likely
ratification date for the 100 Gigabit Ethernet standard, which will
let network managers deliver 100Gbits of traffic through a single
port. Currently, Lawrence has to deliver 10 times that traffic over
a single link, and has to be creative in his solutions.
He is not the only one trying to squeeze a swimming pool through
a drinking straw. Internet exchanges, large telecoms companies,
major content providers and others are all facing the same
constraints, and either need higher speeds now, or they will do
very soon.
For many of organisations, the next two and half years will be a
long wait. In the meantime, the people responsible for developing
the 100 Gigabit Ethernet standard are still in the early stages of
thrashing it out, and they have not officially got a working group
under way yet.
What is going on?
The move to 100 Gigabit Ethernet
Moving to 100gbps is a big leap. The last big increase in
Ethernet speeds happened in 2002, with the ratification of the 10
Gigabit standard. However, the internet never stands still, and
traffic volumes are increasing substantially.
Thanks to everything from internet video to the massive increase
in voice over IP communications, the people that keep the internet
running need more capacity. As a result, a collection of suppliers
and end-user organisations making up the
Institute of
Electrical and Electronics Engineers' (IEEE) 802.3 Higher-Speed
Study Group (HSSG) began the journey down the long, red
tape-ridden road to ratifying a new standard.
HSSG chair John D'Ambrosia says, "The HSSG call for interest,
which is how you get the programme started in the IEEE, happened in
July 2006. The focus at that time was on a networking perspective,
particularly in terms of aggregation.
"We had people from the carrier networks, and people from
datacentres participating. We had individuals from the Wall Street
stock exchange, and we had the video people coming in too. We saw
this recurring theme of traffic growth."
Participants were looking at ways to increase the throughput at
the level where high-throughput switches handle large amounts of
aggregated traffic from other networking equipment. People such as
Lawrence rely on running lots of links in parallel.
Problems of parallel links
This presents challenges of its own, says Val Oliva, director of
product management at Foundry Networks. "You will notice that there
is a latency difference when traffic is serialised over a big link,
as opposed to aggregating over multiple links," he says.
Sending traffic over multiple links also creates synchronisation
problems, should packets arrive out of order. "What happens if it
is out of order is that the other node will say 'I did not get the
packet, please retransmit'."
Add to that the associated management problems and things begin
to get even more problematic. Stringing multiple links together to
get the bandwidth you need means that you must monitor each
individual link for network management statistics, increasing your
overhead.
"Running eight 10 Gigabit links in parallel is an operational
issue in datacentres where you want to plug and forget," says
Lawrence.
But companies who manage networks for a living are prepared to
spend extra time tweaking their infrastructure for high
performance.
Lawrence manages the packet synchronisation problem using
flow-based load balancing equipment, but those load balancers have
to look inside the TCP header to manage the traffic, which makes it
more difficult to scale.
These are not insurmountable problems - Lawrence is aggregating
dozens of 10 Gigabit Ethernet links at a time, and the company has
capacity for growth - but it makes his job harder.
So, with unprecedented pressure from the market, why the hold
up? The HSSG's job at the outset was relatively easy. It fixed on
100gbps as its target speed and was then able to lay out a core set
of technical objectives that would form the basis of its goals. All
was going well at the HSSG until another contingent raised issues
that complicated the situation.
D'Ambrosia says, "Individuals from Sun said that it did not
service the server requirements. Here, we are not talking about
network aggregation. We are talking about the servers themselves,"
says D'Ambrosia.
Accommodating two speeds
Servers have to connect to the network, and some people did not
want them to do that at 100gbps. This is because the requirements
for core network speeds and for direct communications between
servers and networking equipment are growing at different
speeds.
The HSSG eventually decided to accommodate both parties by
including two speeds within the single standard. None of this
surprised former physicist Stephen Garrison, vice-president of
marketing at Force 10 Networks.
"The PCI bus in the server is moving from PCIe to PCI2e early
next year," he says, adding that this creates the need for a 40gbps
connection.
Experts say that whereas many networking firms are asking for
100gbps speeds now, people connecting servers to switches will not
need those speeds until about 2015 at the earliest.
So why not simply work on a 100gbps standard and let it cover
slower connections by default? "You have to realise that these
markets have different cost targets and power requirements.
"If you design for 100gbps and then use it at 40gbps, it does
not get rid of the cost and power implications, so we are really
optimising for the two rates, rather than simply using the one rate
at the slower speed," says D'Ambrosia.
This led to the development of a set of HSSG objectives designed
to support both parties. Both speeds have some things in common.
For example, they will support full duplex operation only, which,
like the 10 Gigabit Ethernet standard, gets rid of the
Carrier Sense Multiple Access With Collision Detection
(CSMA/CD) packet collision issues that all previous versions of
Ethernet suffered from under half-duplex operation.
Unsurprisingly, the traditional Ethernet frame format and frame
sizes will be preserved, ensuring backwards-compatibility with
existing Ethernet standards.
Where things differ is at the physical layer. Operations of
40gbps should support distances of at least 160km on multimode
fibre connections, and at least 16km over a copper cable assembly.
The backplane connections in high-speed server chassis will support
40gbps over a distance of at least 1.6km. For 100gbps operations,
planners are aiming for distances of 40km on single-mode fibre, at
least 160km using multimode fibre, and at least 16km over a copper
cable assembly.
These objectives clearly categorise 40gbps and 100gbps inside
and outside the datacentre, respectively, says John Jaeger,
director of business development at Infinera, which sells equipment
for high-speed network links.
Jaeger, like others that Computer Weekly spoke to, has concerns
over the inclusion of the 40gbps standard within the HSSG's
remit.
"Our customers have made it clear that their business today is
being impacted by an inability to aggregate 10 Gigabit Ethernet
connections in the network. Our first concern was that we needed to
take care of our customers and did not want to delay that
activity," he says.
Jaeger worries that accommodating the slower speed will delay
the standards process, and others have said that the debate between
those focused on server connections and those concerned with core
network aggregation delayed the process by between six and 12
months.
Including server-side stakeholders in the standard also raises
the question of what will happen to other connectivity technologies
in that area, such as Infiniband.
"There is a strong focus on using Ethernet in more of these
applications. A lot of guys were building high-performance clusters
and started using Gigabit Ethernet," says Brad Booth, president of
the Ethernet Alliance. He adds that many high-performance network
firms began writing optimised TCP stacks for their own
environment.
"There is a strong trend towards a common networking
technology," says Booth. It makes for cheaper infrastructure, which
is why, for example, the incumbent 40gbps fibre optic networking
standard, OC768, will not cut it for many datacentre operators.
"It is more cost-effective to buy four 10 Gigabit Ethernet ports
than a single OC768 port," says Lawrence.
What it means for IT managers
In the short term, all this will have little impact on network
and IT managers. All that is really known are the objectives for a
standard.
The HSSG has issued a project authorisation request - a contract
with the IEEE agreeing the scope of work on the project. Once that
is accepted, the task force can be created and proposals for the
draft standard can then be considered and reviewed.
The drafts will gradually be opened up to a broader base of
reviewers, until it eventually makes it to ratification. That will
probably take the rest of the decade and, unlike Ethernet itself,
the process is one thing that is very hard to speed up.