Switches manage traffic, accepting packets
from one location and forwarding that data to the appropriate
network segment or destination. This creates the network
"fabric" that interconnects servers (hosts) with storage
systems. In spite of the simple connotation, switches are
deceptively complex systems using a combination of specialized
physical hardware and onboard logic to manage data exchanges at
line speeds in real time. Consequently, switches must provide a
mix of performance, reliability and management versatility,
while maintaining a reasonable cost per port.
Switches are routinely expanded or replaced to accommodate more
storage platforms or host servers in the fabric. Switching is also
updated to achieve better performance (e.g., faster ports), enhance
fabric availability, restrict network traffic and bring intelligent
features, such as virtualization, to the fabric. This section of
the guide highlights some of the most important best practices and
goals for switch expansion or replacement.
Best Practice No. 1: Understand the physical limitations of
your switches
A switch is usually sized based on a finite number of ports so
you should know how many ports are available for use, as well as
the total number of ports that can be added to your current
switches, if any. Switches intended for small and many medium-sized
organizations offer fixed-port counts and cannot be expanded. The
Catalyst Express 500 product family from Cisco Systems Inc. and the
SANbox 1400 from QLogic Corp. are examples of fixed-port
switches.
When you need additional ports, simply buy another switch and
connect the switches together through an interswitch link (ISL).
Switches intended for large business and enterprise use, aka
director-class switches, are far more upgradeable, offering as many
as several hundred ports through modular upgrades. For example, the
48000 director from Brocade Communications Systems Inc. can handle
up to 384 ports through up to eight 48-port blade modules. Experts
recommend using fewer, larger switches. "A company that needs 100
ports would probably be better served buying a single
director-class switch instead of buying three or more smaller
switches and tying them together," says Stephen Foskett, director
of data practice at Contoural Inc.. "They just perform better than
the smaller ones."
Best Practice No. 2: Watch port speeds and look for 8 Gbps
Fibre Channel switches
Beyond the sheer number of ports, storage professionals should
also consider port speed. Common Fibre Channel port speeds include
1 Gbps, 2 Gbps and 4 Gbps. You can find ports running at 10 Gbps,
but 10 Gbps ports are not backward compatible with the
slower port speeds. Consequently, 10 Gbps ports are typically used
as ISLs to interconnect Fibre Channel switches within the data
center. You don't really need to buy 10 Gbps ports unless there is
a specific need for them. The next big push for port speeds is 8
Gbps, but it's not quite ready yet. "You can buy the [8 Gbps] HBAs
but not the switches," Foskett says. "Eight [Gbps technology] is
off the table right now."
Ethernet switches for
iSCSI SANs commonly offer 1 Gbps ports.
Switches with 10 Gbps ports are readily available, but the
actual number of 10 Gbps ports is limited. For example, the
Cisco Catalyst 4948 10 Gigabit Ethernet (GigE) switch provides
only two 10 Gbps switch ports.
Best Practice No. 3: Understand the functional limitations of
your switches
Today, switches incorporate features that go beyond simply
managing network communication, such as data migration and storage
virtualization, which are common in the large director-class
switches. The need for intelligent features, such as these, is
frequently the motivation for switch upgrades and replacements, and
storage administrators should consider intelligent features and
their role in the data center. For example, it may be possible to
justify a Brocade Mi10K director for VSAN support in the
organization. Conversely, experts warn against acquiring unneeded
features and functionality. Not only is this a waste of money, but
the switch itself is underutilized.
Factor emerging technologies into your upgrade plans. The FCoE
connectivity that is on the horizon will affect switches and host
bus adapters (HBA), requiring a forklift upgrade to implement. As a
result, some users may forego switch upgrades or changes until
FCoE-compliant switches and storage systems are available. "Do I
want to make a big 4 gigabit Fibre Channel SAN investment when EMC
tells me I'll be using a totally different protocol in three
years?" Foskett asks.
Best Practice No. 4: Standardize on one switch vendor in the
enterprise
Experts recommend standardizing on a single switch vendor to
ensure full compatibility and adequate security between switches,
especially if the switches include intelligent features. "Having
more than one vendor in your fabric can compromise your security."
says Phil Goodwin, president of Diogenes Analytical Laboratories
Inc. If you must use switches from more than one vendor, thorough
compatibility and security testing is highly recommended.
Best Practice No. 5: Architect the switching infrastructure
for high availability
It's important to architect a SAN so that redundant connections
use multiple pathways to separate switches, eliminating single
points of failure within one switch. However, the cost of multiple
directors is often prohibitive, while the use of additional
switches is typically discouraged. Storage architects are forced to
perform a balancing act to achieve a suitable mix of cost,
simplicity and reliability, and that balance is different for every
organization. "You have to decide what your tolerance for risk (and
outage) is," Foskett says. "We always recommend building dual
redundant SANs."
Best Practice No. 6: Watch for unexpected failure rates on
older switches
Older switches (circa 2002) may be subject to surprisingly high
failure rates. "If you've got a four- or five-year-old switch with
16 to 24 ports, consider replacing the whole thing," Foskett says.
Budgeting a replacement switch at the first sign of port failures
offers an opportunity to purchase a larger switch with more ports
to accommodate network growth and allow consolidation with other
small switches in the environment. The move to a new switch can
also introduce useful switch features that may not have been
available when that original switch went into service.
Best Practice No. 7: Look for opportunities to deploy
iSCSI
ISCSI technology is being deployed by more and more small and
medium-sized businesses (SMB) , as well as within the departments
and workgroups of large enterprises. While iSCSI is still not an
ideal choice for all SAN deployments, its use is increasing,
bolstered by its relative simplicity, low deployment costs, good
overall performance and reliability, and broad user knowledge of
Ethernet. When expanding a SAN, it's now valid to consider the
potential role of iSCSI in your own enterprise. "Even though iSCSI
is perfectly acceptable for any operating system, if you have 100
Windows systems that you'd really love to hook to the SAN, strongly
consider getting an iSCSI SAN instead of a Fibre Channel SAN,"
Foskett says.
Best Practice No. 8: Avoid port sharing in iSCSI
One potential problem with Ethernet performance is the common
practice of port oversubscription. Many Ethernet servers do not
depend on high performance, so almost all Ethernet switch ports are
oversubscribed. This allows more than one device to use the same
switch port, thereby maximizing each port's bandwidth utilization.
Experts note that a port may be oversubscribed as much as 10 to 1.
Oversubscribed switches may not be able to handle the excess load
in a high-performance iSCSI SAN, so avoid configuration practices
that lead to switch port oversubscription, or select high-end
Ethernet switches for deployment within an iSCSI SAN.