Storage @ Work

Deni Connor at Network World published a column yesterday, capturing the thoughts of technologist Joel Snyder. Here's the link to the Network World article: Network World Column: Is Fibre Channel Dead? I actually saw the story first on Hartmut Wiehr's blog here where there is a permalink: Hartmut Wiehr's posting on "Is Fibre Channel Dead?"

Joel is not a storage guy per se, but a technology guy with a long career who has seen a lot. Here is his profile at Network World: Joel Snyder bio

I'm not sure that I agree with Joel altogether. I do feel that iSCSI will eclipse Fibre Channel relatively soon, but I think Fibre Channel is going to have a longer tail and slower decline than other dinosaur technologies like FDDI and Token Ring- only because the market for storage tends to be extremely risk averse and the process of swapping infrastructures is both risky and expensive. Still I talk to customers every week that have jettisoned Fibre Channel in favor of iSCSI because of the flexibility shortcomings and the relative high cost of owning Fibre Channel.

A new Ethernet standard called CEE (Convergence Enhanced Ethernet) is being proposed as a way to unify networking technologies in the data center. This blog discusses it, as does this article in EETimes.

Of course, we've all seen standards come and go before, but there are certainly good reasons to think that Ethernet standards have a little bit better chance of making it through the ringer than most other standards. If CEE gains critical mass, Fibre Channel and InfiniBand will quietly evaporate and iSCSI will become the de-facto SAN standard technology. It seems inevitable sooner or later.

Lucas Mearian at Computerworld published an interview yesterday with Greg Scherer, formerly the Chief Technical Officer at Emulex. Greg saw a lot of changes in the storage industry in the course of his 24 years career at Emulex and has the catbird's view of the Fibre Channel industry. Now he is putting his experience into a startup, Neterion, which is working on high speed iSCSI technology. Lucas did a great job in this interview and asked a number of intriguing questions. Definitely worth a read:

http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9015082

If you still haven't linked over, here's a quote from the interview to whet your whistle:

If you think about where Fibre Channel is deployed, it's really mainly in tier three of the data center. And it never jumped into being a real channel product. It's really an OEM-driven product. The OEM channel is an expensive channel. The channel [vendors] will make maybe 15 points. It's a pretty slim margin, but there's also very little support. The channel [market] is really just a facilitator to get the product from point A to point B. In the Fibre Channel market, which is OEM driven, there is anywhere from a 40-to-65-point margin. It's very lucrative.

I know there were times when some of the OEMs where making more money off the sales of Fibre Channel adapters and switches than they were off the system sales. It's one of those protected ecosystems where people were willing the pay it and they [the vendors] were willing to take it.

Today's ANSI news: .......... FC vendors propose FCoE. Fibre Channel Protocol over Ethernet or something like that. Serial SCSI running over Ethernet without TCP or IP. Let's see, that has been tried before with Nishan's SOIP, Adaptec's EtherSCSI and other less well known technologies - and killed off by whom? Most of the companies proposing FCoE. Ironic.

So, if you are a FC vendor and it looks like 10Gb Ethernet is going to be part of the tiger (along with iSCSI) that eats your lunch, what do you do? You resurrect old ideas that you hung five years ago, dust off the corpses, insert zombie brains and call it the new shiny FCoE (Frankenstein Convergence over Ethernet?). The only difference between what you killed yesterday and your unholy experiment today is that this bad boy saves your business. It doesn't really have that much to do with customers, except that you get to squeeze them a little longer.

Here is a link to a TechTarget blog that I agree with.

And in honor of this abomination, here is a link to Tim Burton's Frankenweenie on YouTube .

The employees of Brocade are really tired of the shadow Greg Reyes' trial is casting. I think they deserve a break.

Click To Play

In this podcast Farley talks about the problem of managing changes to storage infrastructures.

Craig Randall wrote yesterday about an experience he had at a Starbucks recently and made a point of how controlling your destiny is such a valuable thing. The topic is aligned with a video I made yesterday and I like what he says:

" when you “outsource your experience” to someone else (e.g. to grocery stores such as Safeway or Albertsons), you run the risk of damaging your brand. The people running the Starbucks kiosk are employees of the grocery chain, not Starbucks. They don’t buy into the brand from an experience stand point, and apparently are not encouraged (e.g. via incentives) to do so by their employer (or indirectly by Starbucks)."

Managing storage products is a core competency that most IT aministrators should have. There are definite risks similar to those Craig mentioned above when you turn the "storage keys" over to somebody else. It's a no-brainer to buy technologies and products that allow you to control your own destiny.

Of course, that's why iSCSI is taking off the way it is - because people feel they can understand and control the networks better than they can with Fibre Channel. It's also the design goal of all our products at EqualLogic - making administration easier and faster so that IT professionals can take charge of their own storage infrastructures and move onto other tasks.

John Webster wrote a piece yesterday in praise of FCoE. Now John is somebody I like to talk to and I usually think his arguments are sound, but I can't disagree with him more on this one. Here's what he says: "it leverages the Fibre Channel protocol’s use of SCSI directly over Ethernet, whereas iSCSI uses a different encapsulation method to send SCSI over TCP/IP, which then usually goes over Ethernet."

The glossing over of the FC protocol is important: FCoE uses the FCP protocol, which is an encapsulation of serial SCSI developed for Fibre Channel networks and mapped instead onto non-routable Ethernet frames. It is not somehow more "natural" as John's statement implies.

There are two key takeaways: 1) FCoE is going to cost a lot of money in fork lift upgrades of data center switches. It doesn't run on your existing switches and its going to be disruptive and expensive to purchase and deploy. 2) John Webster mentions that FCoE developers are missing the ease of management message. It's probably more realistic that they are trying to hide from the management question because management doesn't improve much with FCoE. It uses Ethernet MAC addresses, which are roughly equivalent to Fibre Channel's worldwide names, which are one of the root causes of so much pain installing, configuring and changing a Fibre Channel SAN.

Customers with lots of FC storage want FCoE because they want to maintain their investment in their FC storage systems. That's understandable, but they ought to think through what that means for their virtual server plans. It will probably work, but it might not be as simple to manage or as flexible as one would hope for. After all, who would have predicted that FC was going to be so taxing to manage ten years ago when it was first being deployed? And who would have thought those fundamental management problems would still be unsolved ten years later? FCoE is not pain relief, its the reason SAN administrators will need pain relief.

The best course of action for those customers might be to stick with 4Gb FC and avoid all the disruption and cost.

I've received a couple emails now about a bizarre web slideshow that eWeek is linking to in their editor's pick newsletter. It's called "10 Things You Should Know About iSCSI"

There's so much wrong with this that its difficult to know where to start. Maybe, there should be parallel slideshows pointing out the hazards of eating meat (choking, high blood pressure, pets disappearing, etc.) or gardening (bee stings, falling tree limbs, impaling oneself on rusted pitchfork tines, etc.) It has a blatant FC bias. BTW, just imagine if someone tried to enumerate the things you need to know about Fibre Channel. Could they do it?

One slide says: "Don't believe the ease-of-use claims. Yes, the vendors will claim that there's "one-button" setup. Even if everything works exactly as it should, it still takes more than one button." Ummmm....... and careful dialing those 10 digit phone numbers!

Then there is the slide suggesting that the Microsoft iSCSI initiator is risky (remember to chew that meatball before you swallow it). If you haven't used it, you might not know that it is one of the easiest and most stable pieces of SAN host-based software ever developed.

The last slide I'll comment on is the one titled, "It may not save you money". It says: " But even under ideal circumstances, you'll need four iSCSI connections for every Fibre Channel connection, unless you're using 10G Ethernet. And, of course, 10G Ethernet is really expensive, too." OK, and which is faster - a car going 15 miles an hour or a train going 15 miles an hour? People often make the mistake of confusing bandwidth and transmission speed. Even though iSCSI over Gigabit Ethernet has a quarter of the bandwidth of 4Gb FC, there are rarely bandwidth or performance problems with iSCSI. That has been a very pleasant surprise to nearly all iSCSI customers. In addition, automated load-balancing with iSCSI acts as a bandwidth multiplier.

The value of the press in the Internet era is vetting. This piece should have been sent back for more scrubbing and objectivity.

Lucas Mearian at Computerworld wrote about Dell's acquisition of EqualLogic and concludes that it will force a split in the Dell/EMC partnership.

I think Lucas is an astute observer of the industry, but my previous opinion posted on the VMTN forums and my agreement with Chuck Hollis' blog posting yesterday weren't swayed by what Lucas had to say.

Dell and EMC both have a lot more at stake here than the iSCSI business that EqualLogic brings to the table. They do business with a much broader set of products than iSCSI storage - such as VMware. Dell is a very valuable sales channel for EMC and EMC is a very important technology provider for Dell. There will always be some level of competition among the largest companies in the technology sector. It creates a bit of a mess sometimes, but having interdependencies between companies provides stability and efficiencies that ultimately result in broader solutions at lower costs.

I suspect this deal will actually end up creating additional opportunities to expand the business between Dell and EMC. I'm not always right about these things, but that's what my intuition is telling me.

I just got done sitting in on the Wikibon conference call about storage clusters. Wikibon is an idea that I like: open discussions and content that intends to engage subject matter experts, vendors, analysts and customers.

But noble goals do not make for satisfying group gropes. The call today was a mashup of perspectives and expectations that never came together. I think the problem is that the words "cluster" and "storage" conjure up images that are as consistent among people as ink blot interpretations.

This is going to be Wikibon's challenge. They don't want to control the discussion as much as facilitate it. Unfortunately the storage industry has a serious penchant for willful fragmentation. Its one of the reasons why the vocabulary of storage is so confusing. Anyway, the call ran down many different threads without much fusion of ideas across them. One of the more interesting threads involved a discussion of Google and Amazon style storage, with Robin Harris insinuating that all large companies were going to need to adopt such designs. Hmmmm. Not sure about THAT one. Then there was the usual confusion about whether or not we were talking about clustered file systems or clustered SAN storage and whether or not that meant Fibre Channel and whether or not Fibre Channel had much of a future and yadda yadda yadda. Yikes!

I give Peter Burris a lot of credit for attempting to create a recorded summary out of the preceding discussion. His EMC-centric analysis of the storage universe came through a bit strong, but its not difficult to understand considering his many years of breathing and drinking EMC details and their role in the industry.

EqualLogic sells distributed iSCSI SAN storage systems that are built by adding storage modules that work together in an Ethernet network. We don't call in "clustered" - we don't call it "grid". Its distributed, its modular, its very easy to install and manage and it's performance scales as you add systems to the group. I really believe that the future of storage is already available from EqualLogic, its just that most industry observers have not taken the time to understand it.

Storage blogger Stephen at the Ruptured Monkey site wrote this on his last posting:

Its amazing how much time you can waste with troubleshooting in the SAN if you don't have any tools. Especially when you decide to offer some of your knowledge to staff who just joined your team and have some basic idea of what a LUN but don't have a clue what a WWpN or a flogi might be.

And that's exactly why I wrote in yesterday's chat session that I believe iSCSI is going to relegate FC to a sustaining role in the infrastructure. You don't need to learn all that strange, arcane stuff to run an iSCSI SAN. Of course, I have a strong bias towards the Dell Equallogic flavor of iSCSI.

Just saw this article on SearchStorage's Australia/New Zealand site. The customer in the article, Austcorp, talks about the advantages of going with an Equallogic iSCSI SAN. They also describe how it facilitated the adoption of server virtualization (VMware) too.

The article ends with this quote:

"Utilisation of the SAN improves through virtualisation, "We are using 70% the of SAN's capacity and the ROI from these two projects is initially all coming from the SAN."

A customer of ours, John Wilder, talks about his experiences working with our iSCSI SAN products. Besides liking the ease of use and installation features of his PS series array, he is using thin provisioning and snapshots too. Here is a quote from his post:

Here’s a real life example- currently we have a thin provisioned volume which is taking up 96 GB of actual disk space (the host OS sees it as a 500 GB volume), and we have taken 10 daily differential snapshots of this volume which are currently utilizing 1.36 GB of space.

Chris Evans, of Storage Architect blog fame, posted on something Chuck Hollis wrote in a recent blog.

It basically goes like this:

Chuck: "Everybody buys everything from EMC, at least that's what most of the smart people I know are doing."

Chris: "Yeah right, all us smart guys love being locked into EMC - NOT!"

Questions about the future of both iSCSI and Fibre Channel. are popular with analysts and the press.

Eric Schott, Director of Product Management gives his opinions in the interview below, which was filmed by UberPulse. Many readers will likely be surprised by what he has to say, although to me it just demonstrates the kind of clear, realistic thinking he always has.

If you want to catch Eric in person, he will be presenting at SNW next Wednesday. Here is a link to a description of his presentation.

My most recent post on Inside IT.

This blog was first posted here on Inside IT.

This FCoE thing is probably going to last for some time as a difference of opinions and perspectives. For those who wished I had kept my mouth shut (or keyboard locked), I was probably a little nastier a year ago with this post on my Storage @ Work blog.  Being a CREATURE OF HABIT,  I responded to last week's news from SNW with my usual open-minded and fair approach. 

Having said these snide things about FCoE, I am quite sure that it actually will be an excellent solution for lots of FC customers that need a migration path onto something less mortal than Fibre Channel.   The move to Ethernet-based SANs struck me as an obvious evolution a long time ago, after I heard the first FC bigot explain that FC was a channel and Ethernet was a network.  So I guess I shouldn't be too hard on FCoE, because it is a move in the right direction. Being an iSCSI technology bigot, it just seems like a unnecessary, cumbersome step, but to be fair, I tend to see the world through medium-sized business glasses.

I believe that most of the concerns people have had with iSCSI are based on the implementations that are available. They either don't exploit the iSCSI standard very well or they do not scale up to be a good fit for large-scale data processing.  I'm not buying the protocol arguments for FCoE. At the end of the day, I believe the brute force power of 10Gb Ethernet will be sufficient for iSCSI and that I would rather deal with the rare tuning problems that occur than the crushed-veggie-juice-on-papyrus methods of managing storage in an FC SAN.  I know Greg Ferro (see his comment on Dante's post)  agrees.

So where the heck do I stand today?  FCoE is as imperfect as FC, but it gets people headed where they need to be, which is Ethernet - and a lot better than FC.  There are going to be situations where taking a leap of faith to iSCSI is going to be a bit like riding a zipline.  You know its safe, but you don't want to attempt too much all at once.

Here

His views on Fibre Channel, iSCSI, FCoE and competition in our industry. FWIW, we are kicking butt with our channel partners, despite the FUD our competitors are trying to spread.

PCL Construction in Edmonton is one of the largest construction companies in North America - and they are employee owned, which means employees there are always looking for ways to be more efficient.

They became EqualLogic customers 3 years ago after having been a Fibre Channel shop prior to that. They can run everything they need to on Dell EqualLogic storage and like the ease of use and flexibility that our advanced virtualization architecture gives them. I have a video on my InsideIT blog where Derek Knox from PCL speaks at the VMware forum in San Jose about doing live data migrations while upgrading his systems.

In video below, Derek and David Howse talk about the simplicity of using Dell EqualLogic storage and SRM. Derek talks about how they get performance equal to that of their Fibre Channel SANs. Some of that comes from having a smart SAN infrastructure, but a lot comes from the design of EqualLogic arrays. FWIW, most of the new systems they are buying today have SAS drives, to give them even more overhead for their high performance applications.