SAN topology guide and Fibre Channel over Ethernet (FCoE)

This SAN topology guide details the key SAN fabric topology types and looks into the future of the Fibre Channel over Ethernet (FCoE)-based converged network.

SAN topology has undergone a persistent if slow evolution. And with the advent of Fibre Channel over Ethernet (FCoE), converged LAN/SAN architectures have become possible.

But the Fibre Channel fabric is here to stay for the foreseeable future. This guide details the key SAN fabric topology types and looks into the future of the FCoE-based converged network.

Fibre Channel Arbitrated Loop (FC-AL)

The most primitive type of Fibre Channel fabric is the Fibre Channel Arbitrated Loop (FC-AL). This topology does what it says on the tin -- it's arranged in a loop (without a switch) where every Fibre Channel-connected device connects to the next, somewhat like a Token-Ring network. This topology can support up to 126 devices. While being simple and cheap to set up, it is not optimised for high levels of performance as only one port can speak to another at any one time. Also, if one device fails the whole loop goes down.

Switched fabric topologies: Single and dual-switch

Given the disadvantages of an unswitched SAN like FC-AL, the obvious answer is to add a switch to link ports on the fabric. Adding a single switch as the hub connecting the "spokes" of a fabric sidesteps the possibility of one device failure bringing down the whole fabric and also reduces latency as many ports are allowed to talk to many others simultaneously, depending on the number of ports on the switch.

The nature of the switch is key to the performance and resilience of a single-switch fabric. A director-class switch will potentially contain thousands of ports and redundant internal backplanes. A fixed-count switch will be much more limited in port count and will have no internal redundancy.

The next variation on the switched fabric is to add another switch, with every server and storage device dual-pathed through both switches. This immediately doubles port count and introduces redundancy, making sure connectivity is safe if one fabric (switch) goes down.

Adding more switches results in a loop or mesh fabric. A loop fabric is the result when you interconnect switches, with each switch connected to the next until the loop is closed. A mesh fabric is similar to a loop, but as well as connecting to the next switch, all switches are also connected to each other via inter-switch links (ISLs). Simple looping of the switch fabric means traffic has to potentially go through several hops to connect devices.

With a mesh fabric this disadvantage is overcome as every switch connects to any other device with only one hop. It is a topology well-suited to small- and medium-sized organisations.

Drawbacks of the mesh topology are that if you want to expand it you have to connect the new switch to every other switch. Also, because of the number of inter-switch links you need to ensure that your switches have adequate port count to cope with the number of connections. This results in diminishing returns as after a certain point there isn't much benefit gained by adding a switch because the number of ISLs that have to be catered to will use so many switch ports.

Core-edge topology

The port count/ISL limitation on connecting fixed switches led to the core-edge topology in which a larger port count fixed switch or director-class switch is attached as a hub to fixed switches at the edge. Core-edge fabrics can grow to hundreds of ports with devices only having to traverse two switch hops to access their storage. Small-scale core-edge deployments would leave too many unused ports in the core switch, so it is best suited for large fabrics.

The mesh topology is therefore an excellent choice for small- to medium-sized SANs vs. a core-edge topology, which would result in an excess of empty ports in the core switch. As the fabric grows, the core-edge configuration minimises the number of wasted ISL ports. Redundancy can be built into the fabric by having each host device connect to two edge switches to create two independent paths. To do this, server host bus adapters (HBAs) need multi-pathing software set up to fail over and/or load balance between paths. Adding another core switch would make the network fully redundant.

Converged LAN/SAN fabrics

The advent of Fibre Channel over Ethernet (FCoE) has brought the possibility of running LAN and SAN traffic on the same physical cabling. Ethernet was never built for the type of low-latency, lossless transmission needed by data storage traffic, hence the need for Fibre Channel. However, FCoE or, more precisely, the newly developed enhanced Ethernet standards have brought the possibility of putting SAN traffic onto 10 Gbps Ethernet cabling.

Next Steps

SAN switching: How to configure a SAN switch

SAN topologies, part 1: Know your switch options

SAN topologies, part 2: How to design your SAN

Director-class SAN switches handbook

SAN implementation: Key steps in SAN deployment


Read more on Storage fabric, switches and networks

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.