Bandwidth blues

For companies that require incredibly high bandwidth networks, the Gigabyte System Network (GSN) might be the best solution

For companies that require incredibly high bandwidth networks, the Gigabyte System Network (GSN) might be the best solution

Gigabyte System Network, or GSN, is the highest bandwidth and lowest latency interconnect standard, providing full duplex 6400Mbit/s (800Mbit/s) of error-free, flow-controlled data transmission. The technology is ideal wherever organisations require timely and efficient movement of large amounts of information, including scientific and technical computing, HDTV, data mining, transaction processing, video and film archiving, and storage management. The proposed ANSI standard provides for interoperability with Ethernet, Fibre Channel, ATM, HIPPI-800, and other standards.

GSN is suitable for technical computing applications such as clustering and system area networks and for enterprise applications of big data client-server functions (e.g. HDTV, post-production scanners, MR medical imaging). It can also be used for storage management backbones that need huge bandwidth, low latency, and extremely efficient CPU utilisation.

At present, Silicon Graphics is the only vendor providing support for this technology. Gigabyte System Network, also known as HIPPI-6400, is a physical layer (PHY) currently being developed within the HIPPI family of interconnect standards. The official standard number will be ANSI NCITS 323-199x.

The HIPPI link technology most commonly used today is HIPPI-800, a 32-bit simplex parallel interface defined for copper cables clocked at 25MHz. The raw bandwidth of HIPPI-800 is 800Mbit/s. GSN defines a 20-bit interface for copper cables operating at 500MHz, or a 10-bit interface for fiber-optic cables operating at 1GHz. It has a raw bandwidth of 10000Mbit/s in each direction. This provides a payload bandwidth of 6400Mbits/s in each direction after subtracting the overheads of AC-encoding and control information.

The GSN effort began with a combination of circumstances: a desire for gigabyte/s networking compatible with existing HIPPI and Ethernet networks, proof-of-concept technology for high-bandwidth links and routers and the ability of the ANSI T11 standards group to consider another PHY for HIPPI.

Computer interconnects carry a wide variety of traffic types because of the broad set of applications that utilise distributed computing resources. Examples include MPI-based parallel computation, file serving, web serving, datamining, transaction processing, video and image archiving, and distribution. This is certainly a broader range of applications than the telnet-ftp-mail scenario.

The traffic generated by these applications may vary from small messages of a few bytes to large multi gigabyte bulk transfers. In addition to the size metric one can also distinguish different message frequency distributions ranging from asynchronous to bursty to continuous. There may also be hard real-time components where a priority discipline or a completion time discipline is needed.

The proliferation of applications and their different traffic profiles has encouraged the development of optimisations. Modern architectures such as ATM and Fibre Channel represent a departure from older designs such as the IEEE 802.3 Ethernet family in a significant way: these designs attempt to explicitly accommodate a wide variety of traffic types.

ATM approaches the problem with several techniques: (1) a common cell structure with numerous adaptation layers (AAL) that are specialised framing procedures, plus (2) explicit quality-of-service parameters and means for administering them (e.g. rate control), and (3) explicit procedures for constant-bit-rate (CBR) traffic. The Fibre Channel standard approaches the issue of different traffic types by defining different classes of service and several physical media. Both ATM and Fibre Channel exceed the scope of Ethernet and HIPPI-800 by a wide margin. They also exceed the complexity of the other designs by a wide margin.

The main performance objectives for GSN are high bandwidth and low latency: 800Mbit/s transfers and 1 microsecond latency for short transfers over short distances. Other objectives include:

1. Reliable, flow-controlled links

2. OS Bypass support

3. Multiplexing of messages on a link

4. Compatibility with HIPPI-800

Flow control in GSN depends on a credit-based protocol. Having reliable links means having some sort of error detection and retransmission or possibly forward error correction. GSN uses 32-bits of cyclical redundancy check (CRC) for error detection over a 32-byte micropacket plus a sliding window protocol that supports retransmission of damaged macropackets.

Flow controlled links leading into a switch can be used by the switch to eliminate congestion within the switch. A fabric that is both reliable and flow controlled can interface directly to an application without going through the usual operating system and protocol stack software layers.

Operating system bypass support means moving data between a link and an application without OS system calls or intervention. A new upper layer protocol called the "scheduled transfer protocol" defines a standard method of OS bypass.

Other HIPPI PHY layers do not multiplex. This means that while one system is connected to a destination system, others must wait for that destination. Since HIPPI messages may be arbitrarily long, it is not possible to bound the waiting time. GSN provides four virtual channels (VCs) so that up to four messages can progress across a link at the same time.

Maximum message sizes are also specified so as to bound the busy time on each VC. Data transfers larger than the desired message size are broken into smaller blocks for transmission. The details of decomposing a large transfer and collecting the blocks within an application buffer are controlled by the schedule header.

Compatibility with HIPPI-800 is maintained by defining a representation within GSN for all the packet types defined for HIPPI-800 and by defining procedures for implementing a transparent bridge between the two PHYs.

The use of 48-bit IEEE 802.3-style addresses allows bridges to other datalink protocols to be implemented. GSN Link. There are two kinds of media defined for GSN: copper coaxial and optical fiber. The copper link is based on a 50-pair coaxial cable and 100-pin connector.

The optical media to be selected for GSN depend on the continued evolution of parallel optical fiber components. GSN link protocol is designed to accommodate links up to 1km in length. The propagation time for optical fiber is roughly 5 ns/m, so the propagation time for a 1km link would be 5 microseconds. Five microseconds of data at 800Mbit/s is 4000 bytes. Thus a 8Kb (8192) transmission buffer is sufficient to cover the round-trip delay of a 1 km GSN link with a little spare time.

Automatic skew compensation is an important part of the GSN link design. Without it a simple parallel link of this kind would not be able to cover much distance: skew buildup could easily exceed the clock period of 2ns.

The skew buildup in optical ribbon cable varies from a low of about 1.5ps/m to as much as 10ps/m. Thus skew for a 1km link could vary between 1.5 and 10ns, depending on the quality of the cable. GSN compensates for skew by defining a special bit pattern called the training sequence, which is used at the receiving end of a link to sense the amount of skew on each signal line. Receiver circuitry can use this information to compensate for skew.

( SGI 1999

Compiled by Agith Ram

Read more on Networking hardware