In trying to be all things to all men and not focusing on a
technology, Infiniband has diluted its message and been sidelined
by industry developments.
Back in the early 90s, an Informix marketing executive made a
presentation to some gathered journalists predicting the death of
the mainframe by 1999. In fact, not only is the mainframe alive and
well, but some technology concepts that emerged from the mainframe
market are now being proposed for use in more distributed data
centre applications. Clearly, new dogs are learning old
tricks.
One of the most high-profile technologies using mainframe-like
principles is Infiniband, a server connectivity architecture which
vendors are hoping will dramatically improve the way that servers
talk to other equipment in the data centre. Although this
technology is relatively new (the Infiniband Trade Association was
formed in October 1999), it uses the old mainframe principle of
divorcing I/O processing from a server's central processing
unit.
The companies steering the ITA - Compaq, Dell, Hewlett-Packard,
IBM, Intel, Microsoft and Sun Microsystems - wanted a new
technology that would enhance the way that servers connected with
peripheral devices such as storage arrays and with other servers.
Previous data centre connectivity architectures have significant
drawbacks, according to the ITA's collection of white papers. One
such drawback is that many of them were bus-based, whereas
Infiniband is fabric-based. Bus-based architectures use a single
data 'highway' that all devices share. Such data connections can
become congested, meaning that the number of devices using them can
be limited and bandwidth is constrained. With server processing
power increasing exponentially, this can cause problems.
Switched fabric
Infiniband works by using a switched
fabric technique, in which a switch or collection of switches
connects devices into the server on a point-to-point basis. The
necessary I/O processing power takes place outside the server,
rather than inside the box, and this has several other advantages.
Instead of sending all information to a single server, an
Infiniband switched fabric can take input from multiple external
devices and send them to any servers attached to the Infiniband
fabric.
Two key components of the Infiniband fabric are the host channel
adaptor and the target channel adaptor. These are hardware
interfaces used by the server and by external devices to connect to
the Infiniband fabric. The HCA sits inside the server, generally on
the motherboard, whereas the TCA sits on the external device's I/O
controller on the other side of the fabric. They talk to each other
using packets defined in IPv6, the latest version of the Internet
protocol.
Infiniband networks provide a bandwidth of between 2.5Gbit/sec and
30Gbit/sec, depending on whether you use a 1x, 4x or 12x link as
defined in the Infiniband specification. An Infiniband link can
stretch for up to 17m using copper cable, and the distance rises to
kilometres when running Infiniband over fibre.
Varied applications
The applications for Infiniband are
varied, according to Brice Clark, director of strategy and business
planning for Hewlett-Packard's Procurve range of switches and hubs.
One attractive possibility is to use Infiniband as a single
connectivity architecture to handle I/O between the server and
other devices using other protocols. He envisages a scenario in
which an Infiniband fabric is used to connect a server to other
devices that are using a mixture of fibre channel and gigabit
Ethernet connections, for example. This would give the other
devices the benefit of the switched architecture, while also
minimising the amount of cabling spaghetti flowing directly into
the server.
The 17m limit for copper-based Infiniband connections is also
enough to make it viable for storage applications, according to
Clark. A white paper from Mellanox, a member of the ITA, also
highlights storage area networks (SANs) as a suitable application
for Infiniband. Generally, fibre channel or high-speed Ethernet
connections are used to build SANs, which are collections of
storage arrays communicating through their own high-speed matrix of
switches in much the same way as an Infiniband fabric is
structured. An Infiniband SAN fabric could increase bandwidth over
fibre channel, which peaks out at around 4.24Gbit/sec.
Clustering potential
Clustering is the other area in
which the technology shows some potential. Geoff Poskitt, a systems
architect at Fujitsu Siemens, likes the concept of Infiniband in a
clustering environment because of the technical qualities of the
architecture. Ethernet clusters can run into problems because of
Ethernet's carrier sense with multiple access/collision detection
(CSMA/CD) framework, in which packets are sent across the network
in the hope that they don't collide with other packets. If they do
collide, then they are resent. The more traffic on the network, the
more likely these collisions are to occur and the slower network
traffic will be.
"Clustering doesn't really have an industry standard at the
moment," Poskitt explains. "There are some loose clusters that work
over fast Ethernet. But there are other clustering configurations
that carry low message latency requirements. With Ethernet, you can
get high latency."
Not everyone agrees with him. Geoff Barrall, CTO for network
attached storage vendor BlueArc, explains that his company was
planning to roll out Infiniband-capable products, but that it
decided to put its implementation of the technology on ice
following what it perceived as a general cooling-off among vendors.
"We've put it on hold now until we can see survival for Infiniband
and until we can see some form of adoption for it," he explains,
adding that while Infiniband may be a low-latency product, Ethernet
is getting so fast now that such considerations are often
irrelevant. "It would be possible to do the same things in [the
clustering] space with Ethernet. Infiniband has a lot of strong
competition now from Ethernet. Historically, Ethernet has always
done well. It beat off Token Ring, FDDI, ATM and more recently it's
been competing with fibre channel."
Blade servers
With clustering and SANs taken care of by
technologies such as fibre channel and Ethernet, the most promising
opportunity for Infiniband is in a new area - blade servers. This
product category, which we discussed in the 9 July issue of
MicroScope, focuses on server density and offers an alternative to
traditional rack-mounted servers at the low end of the market. A
blade server has a tightly packed chassis in which tens of 'blades'
- essentially servers on cards - can be slotted, increasing the
processing power per square foot ratio. Vendors argue that this is
an increasingly important issue for companies which have to pay
steep rental costs for space in specialist data centres.
Several companies are planning blade servers, although it's not
clear that all of them will be using Infiniband, at least in the
first instance. Fujitsu Siemens, for example, has launched some
products in this category, but its products are aimed at front-end
applications such as e-mail servers and are based on gigabit
Ethernet. Poskitt says the company will expand into the middle tier
next year, and that the next generation of blade servers will be
"full performance". "A lot of people are saying that Infiniband
products will start appearing in the second half of next year, or
2004," he predicts, adding that blades will have an Infiniband chip
on them that plugs into a backplane. An Infiniband switch will then
connect a group of blades together, bringing us back to the
clustered scenario again. "The key thing to remember is that this
doesn't make sense in a single scenario. You'd need a lot of shared
servers to want to do that," he argues.
Too risky
None of this makes things very easy for
companies such as BlueArc, which doesn't want to take a leap of
faith into Infiniband until it's sure that it has a supplier that's
going to stay the distance. Barrall doesn't want to start using a
vendor which then pulls out of Infiniband or goes out of business,
leaving it unable to get parts. The problem is that he doesn't
think the market for blade servers is enough to support all the
Infiniband vendors, making the whole thing too risky for him at
present.
Barrall isn't the only person who's concerned about Infiniband.
Nigel Lambert, marketing manager for storage equipment distributor
Zycko, is equally cynical. "No mainstream product has its eye on
Infiniband. It's going to be high-end and niche," he says, adding
that his company understands gigabit Ethernet because it's built on
well-established standards, and that it's also working with fibre
channel, another tried and tested technology. "The channel is there
to serve the needs of the mainstream."
If it were just small players such as BlueArc and Zycko that were
anti-Infiniband, perhaps the ITA wouldn't have quite so much to
worry about, but Infiniband has suffered more significant knocks.
In particular, Intel announced in May that it was not going to be
making host channel adaptors for the volume market, contrary to
earlier plans. Instead, it would leave that to third parties such
as Mellanox. This did nothing to encourage enthusiasm in the
market.
Diluted message
Infiniband sounds like a no-brainer,
technically speaking, so what gives? One problem, according to HP's
Clark, is that Infiniband tried to be all things to all men, and
has diluted its message in the process. "It's better to focus on a
technology than trying to make it all things to all people," he
says, arguing that people have already been proposing that the
technology be used as a LAN transport mechanism before it has even
met its primary objective. "Infiniband hasn't fulfilled anything
yet, even as an I/O system. In fact it's taken so long that other
I/O alternatives have shown up. The way it ends up when you try to
do everything at once is that nothing happens."
He says that Infiniband could end up suffering the same fate as
ATM, the technology that tried to serve the workgroup, LAN backbone
and WAN backbone markets in the latter half of the 90s, and which
he says is now being replaced with gigabit Ethernet installations
across the board. The reason that Ethernet has won out over so many
other technologies is that it builds on strong foundations,
establishing itself in one market before attacking another, rather
than trying to capture all markets at once, he argues.
Moving goalposts
Perhaps one of the biggest problems
for Infiniband is that because the lead time for the technology has
been relatively long, the market goalposts for the technology are
moving. Infiniband was originally designed to offer an alternative
server connection to the PCI bus, which has been the primary means
of connecting external devices to servers and PCs for the past
decade. Initially, host channel adaptors would be based on PCI
cards, but eventually they would be provided as native
implementations. The problem is that since then, PCI technology has
evolved, leading to an extended lifespan for this veteran bus
technology.
The PCI-Express and PCI-X 2.0 specifications that were ratified on
23 July by the PCI-SIG - the consortium that develops and promotes
the PCI standard - dramatically enhance PCI's speed. The most
significant of these two standards for the Infiniband market is
PCI-X 2.0, because it focuses more on external connectivity,
whereas PCI-Express (formerly known as 3GIO), is concerned
primarily with connecting components that reside inside the PC,
such as graphics subsystems.
PCI-X 2.0 is the latest in a series of evolutions of the PCI
standard. Literature on the PCI-SIG Web site explains that the
latest version of the specification offers speeds of up to 533MHz,
compared with the original PCI's 33MHz specification. The PCI-SIG
explains that it will be a complementary architecture to Infiniband
and will be able to connect to Infiniband fabrics outside the
box.
What's interesting is how PCI-X 2.0 is putting the squeeze on
Infiniband at the server I/O end of the equation. Whereas
Infiniband was intended to replace PCI, companies now have an
alternative technology based on a well-established I/O mechanism.
This puts Infiniband under the same pressure at the server I/O
level as it does at the external connectivity level, where it faces
similar pressure from gigabit Ethernet.
Lukewarm support
Given this reduction in the scope of
Infiniband at all levels, the future of the technology is far from
certain, and the lukewarm support from major players such as Intel
isn't helping. Systems integrators and VARs, faced with customers
who simply want to get the job done, may find Infiniband a hard
sell, especially as many products haven't even appeared yet.
Moreover, they will likely have more experience with technologies
such as gigabit Ethernet and perhaps fibre channel.
Infiniband has an impressive name, but it could end up being a
niche technology for very high-performance applications. There is
doubtless an opportunity for vendors and channel partners to make a
profit, but it isn't as large as the ITA may initially have hoped.
Further information
www.bluearc.comwww.fujitsusiemens.co.ukwww.hp.comwww.pci.sig.comwww.zycko.com