I've been spending a lot of time with customers lately and the subject of networking naturally comes up in many of those discussions. There are a couple take aways from these discussions:
While there are many switches to choose from, the Cisco 3750 switches get the highest marks for compatibility, performance and lack of problems. FWIW, we use them in our own development labs at EqualLogic.
The iSCSI industry lacks the sort of interoperability testing that FC customers are familiar with. There are a number of reasons for this, but the fact remains that iSCSI customers don't get the same kind of help selecting their switching equipment as Fibre Channel customers do. Sometimes customers have to change out the switches in their iSCSI networks in order to overcome performance issues. That can suck. Its also one of the reasons iSCSI sometimes gets a bad performance rap. iSCSI is not the problem - the problem is the overall SAN (storage + network) design. An iSCSI SAN is only as good as the switches that are used.
We tell customers that iSCSI SAN switches need to support jumbo frames and flow control. Another consideration is buffer memory; the more buffer memory a switch has, the better. ISL links also matter If iSCSI traffic is going to traverse multiple switches, you probably want to have high speed ISL capabilities. That's one of the reasons we like the 3750s so much. The next generation 3750 (the E model, I believe) have an interconnect that bypasses the switch's backplane for traffic that is destined to egress the stack through another switch.
There are a lot of switches that work very well. Try to stay away from cheap switches, that's where most of the problems are.
Comments (9)
We are currently testing a Cisco 3560G for our iSCSI SAN, and will be using our 4510 with a gigabit blade for failover.
Posted by Justin Campbell | November 30, 2007 1:18 PM
Posted on November 30, 2007 13:18
Good Point Mark! Thanks for bringing this important topic up. It might be worth mentioning that the EqualLogic Educational Services Organization will be offering a 1 day Workshop on Optimizing the iSCSI Infrastructure starting in January of 2008.
Although, we do not specifically call out the Cisco 3750 as the "best" switch in the Workshop, we discuss the features and criteria that should be considered when choosing a switch for an iSCSI SAN along with hands on Lab Exercises on configuring the switches and Operating Systems to get the absolute best performance.
As we all know the EqualLogic PS Series Arrays are built to give the absolute best performance out of the box, so we want to ensure that the rest of our customer's iSCSI infrastructure is optimized as well.
Posted by Marilyn Harrison | December 4, 2007 9:29 AM
Posted on December 4, 2007 09:29
When dealing with Jumbo Frames on the 3750 and 3560, it is a stack-wide setting for gigabit and a stack-wide setting for 10/100. If ingress traffic enters a gigabit interface and egresses on a 10/100 port, it is dropped (per: http://www.cisco.com/en/US/products/hw/switches/ps700/products_configuration_example09186a008010edab.shtml#c3). Is there any recommendation for dealing with a mixed stack (having some gigabit switches and some 10/100 switches)? I would like to keep it as one stack and not break it. We currently have five 10/100 switches and two gigabit switches in the stack.
Posted by Dustin | January 8, 2008 9:33 AM
Posted on January 8, 2008 09:33
I would be curious to see a collection of switch features to look for in selecting a switch solution for an iSCSI environment. From what I've seen so far, connectivity speed, STP, Flow Control, unicast Storm Control, Jumbo Frames, and VLANs seem to be some of the most pertinent considerations.
Posted by Robby | April 30, 2008 10:21 AM
Posted on April 30, 2008 10:21
Great information on the importance of switches. As an integrator, we have the tough job of selling the higher end switches. Some of the esoteric points are often lost in client discussions, especially get into things like buffers and jumbo packets. Maybe you've covered it elsewhere, but it may be worth mentioning that NIC hardware can also make a tremendous difference. Servers with NICs that do not have jumbo packet capability or receive buffers which can be adjusted to 2048 may still perform poorly even with a good switch and EQ backend.
Posted by Dave Murphy | May 9, 2008 4:17 PM
Posted on May 9, 2008 16:17
Dave, good points on the importance of NIC selection.
We've been talking about starting a series of online chats on the topic of iSCSI networking on the Dell Techcenter. I'll announce them here on this blog and at Inside IT later this month.
Posted by marc farley | May 9, 2008 4:47 PM
Posted on May 9, 2008 16:47
When configuring flow control on a 3750 Gigabit switch, what flow control option is preferred? I am currently using the flow control option desired.
If you have a sample configuration for the iSCSI and Cisco Switch configuration, that would be very helpful.
Posted by Jim Dixon | June 2, 2008 7:58 AM
Posted on June 2, 2008 07:58
Jim, there is a document on our web site listed on this page (under the category of network switching). You can download after registering.
http://www.equallogic.com/resources/technicaldocumentsview.aspx?id=364
There's a lot of stuff in the doc, but the section on flow control has this:
The following commands shows how to configure Gigabit Ethernet interface 0/1 on switch 1 to
auto-negotiate the correct Flow Control setting with the device to which it is connected:
Switch> enable
Switch# configure terminal
Switch(config)# interface gigabitethernet1/0/1
Switch(config-if)# flowcontrol receive desired
Switch(config-if)# exit
Switch(config)# exit
Switch# copy running-config startup-config
To view or confirm Flow Control status on a port, use the following command:
Switch# show flowcontrol interface gigabitethernet1/0/1
Posted by marc farley | June 2, 2008 3:05 PM
Posted on June 2, 2008 15:05
We just installed a pair of Cisco 3560Gs. We enabled flow control and jumbo frames, per the doc mentioned above.
However, in testing throughput with IOmeter, I'm seeing better throughput when *disabling* jumbo frames on the client-side NIC. I've confirmed this with several different servers, using different IOmeter test setup configurations.
I've confirmed that the MTU size is set on the switch:
SAN3560G-1#sh system mtu
System MTU size is 1500 bytes
System Jumbo MTU size is 9000 bytes
Routing MTU size is 1500 bytes
SAN3560G-1#
Anyone have any clues as to why I'm seeing this behavior?
Thanks,
--Raleigh
Posted by Raleigh | June 9, 2008 6:53 PM
Posted on June 9, 2008 18:53