Storage @ Work

Hi, I’m mfarley and I’m a storage geek. I’ve been in the network storage industry ever since the early 90’s when I worked for Palindrome, a Netware-oriented backup and recovery company. I wrote a couple books including Building Storage Networks (Osborne-McGraw-Hill) and Storage Networking Fundamentals (Cisco Press) that put me in the position of being some sort of storage expert – something that I’ve always been uncomfortable with. There’s simply too much to know and I don’t think any one person can be an expert about all this stuff. The thing I have been good at is articulating storage problems, solutions and architectures.

I started working with EqualLogic a couple years ago as a part of their corporate advisory board. Over time, I came to really like what they were doing and had the chance to speak with over 100 of their customers to see what they thought. The customer loyalty EqualLogic has really got my attention. Last year, I started considering going to work for one of the analyst companies where I could be part of a team and do my writing and analysis work with others and a sales staff with real office administration, as opposed to being a lone wolf. The whole thing looked very appealing after years of being sales, marketing, analyst, writer, accounting, and operations all in one. I was really, really close to making that decision, but I knew I wanted to talk to EqualLogic because I felt there were very special things going on there.

To be brief, it turns out that I got hired by EqualLogic to work on customer programs. Things like advisory boards, communities, forums, and this blog. I’m looking forward to carrying on blogosphere conversations with people about EqualLogic and other storage stuff that I find interesting. I hope the story that comes out of this will be interesting – sort of a chronicle of the trials and tribulations of being in a storage startup that introduces disruptive technology and a new business models and how the market reacts to it.

An interesting post from Tony Asaro: Invisible Storage and Strategic IT. Virtualization means many different things to many different people, but it is an essential underpinning of any advanced system and storage infrastructure. EqualLogic customers get to experience storage virtualization on all the levels Tony is talking about.

I saw Steve Duplessie's post today that he isn't going to be at SNW in San Diego. Wahhhh! OK, so it won't be the same event, but I'm still going to go anyway and I know I'm not as much fun as Steve, but I'm looking for chill hanging opportunities. Yo Tony? Wassup?

Lucas Mearian at Computerworld published an interview yesterday with Greg Scherer, formerly the Chief Technical Officer at Emulex. Greg saw a lot of changes in the storage industry in the course of his 24 years career at Emulex and has the catbird's view of the Fibre Channel industry. Now he is putting his experience into a startup, Neterion, which is working on high speed iSCSI technology. Lucas did a great job in this interview and asked a number of intriguing questions. Definitely worth a read:

http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9015082

If you still haven't linked over, here's a quote from the interview to whet your whistle:

If you think about where Fibre Channel is deployed, it's really mainly in tier three of the data center. And it never jumped into being a real channel product. It's really an OEM-driven product. The OEM channel is an expensive channel. The channel [vendors] will make maybe 15 points. It's a pretty slim margin, but there's also very little support. The channel [market] is really just a facilitator to get the product from point A to point B. In the Fibre Channel market, which is OEM driven, there is anywhere from a 40-to-65-point margin. It's very lucrative.

I know there were times when some of the OEMs where making more money off the sales of Fibre Channel adapters and switches than they were off the system sales. It's one of those protected ecosystems where people were willing the pay it and they [the vendors] were willing to take it.

Two posts ago I sent my props to Robin Harris (Storagemojo), now I'm feeling the bloggers biteback. Robin - what are you doing?

In his most recent post, Storagemojo links over to a post of Jon Bach's (dated Feb 5, 07) from Puget Custom Computers where Jon discusses the problems of using cheap RAID controllers in desktop systems and he gives failure numbers for drives that he has been working with in his business. Jon argues that RAID in desktop system is a royal pain because repairing an array on these systems is more of a problem than restoring from backup. I'm not going to argue much with Jon here and I certainly don't have a bone to pick with him, but I can tell you with some certainty, that my own experience on my own desktop systems has been different. I would much rather replace and remirror a disk drive than restore an entire system. Then again, I only support my family and me, a conundrum of requirements and systems that would probably be incomprehensible to anybody but me. (I know there are readers out there that know what I'm talking about it). Jon apparently has a lot of good customers and I'm sure he does what is right for them.

My heartburn is with Robin for implying that Jon's numbers might be relevant for enterprise storage and implying (by reference) that RAID is a problem too. Geez, Robin, you know better. I'm not even going to address the RAID thing because that is so wrong. On the disk side....engineering teams at all major storage companies spend many, many hours qualifying drives and working with drive manufactuers to increase reliability. Then there is the by now famous burn-in process where drives are rigorously tested to weed out those likely to experience early life failures. Then there is some likelihood that these drives live in server rooms with better environmental conditions. Then there are advanced functions like background scrubbing to find and relocate bad blocks, yadda, yadda yadda. And it is not a bunch of whooey- it is a necessity for staying in business in the ultra-conservative enterprise storage market.

There are 3 main reasons the "big guys" like EMC, EqualLogic, Hitachi, HP, Netapp and everybody else don't publish their drive failure numbers.

1. Contractual restrictions
2. High percentage of NTFs (no trouble founds) Storage system vendors and drive manufacturers agree to disagree that a drive failed (which is probably why contractual restrictions about failure disclosures are in place). The drive manufacturers will always claim a lower percentage of failures than the storage system vendors and customers. This doesn't make the disk drive manufacturers less honest by the way - there actually are a high percentage of NTF drives returned where it appears there is nothing wrong with them. I'ts rumored that some of these drives show up in retail outlets and other channels - something that, if true, could possibly impact the numbers Jon Bach is seeing.
3. The first vendor to state their numbers loses - big time. Considering the amount of slop in the analysis, if any vendor were to give numbers for drive failures, all competitors would publish lower claims and then the great wrestling match of mind numbing analysis minutiae would start and foreheads would slap keyboards and drool would seep into the cracks.

And all these things that I'm pretty sure Robin understands, but it does make for fun blogging.

Dave Hitz and I are going back and forth in our blogs, explaining our views on language - specifically the lexicon of storage. Dave's most recent post reiterates the explanation given in his previous post. Apparently he didn't like the fact that I awarded him the wagging finger of shame for willingly blurring the definitions of SAN and NAS.

My take on it still the same. The terms we use stand for something and provide a way for us to communicate complex concepts quickly. When we start polluting the definitions of accepted terms like SAN, for instance, we create confusion about things that are already complex - and this wastes a lot of time. A room full of people discussing storage can make a lot of progress if everybody clearly understands the terms. That same group without a shared lexicon will have a very difficult time succeeding. You can use the word "hammer" to refer to a hammer, a pair of pliers, or a screwdriver; or you could use the word "tool" to generically refer to these things. I think the problem you are having Dave is the lack of a generic term for a network that does "storage stuff". I use the term "storage network".

It has always been a problem in storage networking and I've always thought the term "SAN" was flawed. Let's face it - it was a term adopted by the Fibre Channel industry a decade ago to get the market to identify the technology with a new storage architecture. Your customers will use whatever terms they want to use and I don't particularly care what terms you use with them. However, I think you understand the difference between your role as a thought leader in this industry and your role as a sales support dude at Netapp. If you had written about the need for a better generic term for networks with storage traffic, that would have been a good topic, but instead you wrote about how it doesn't really matter what you call it and suggested we kidnap the term SAN for that purpose. This just creates confusion, which leads to wasted time, arguments, blame and all the other unfortunate things that go on when people don't communicate effectively. The finger wags because you are contributing to the confusion.

Storage-zilla has a post where he points to an EMC White Paper that explains how Netapp Filers make their SAN (block I/O) storage work through their WAFL-based filers.

The same argument the Zilla man lays out here works for EqualLogic of course too. If you are looking for iSCSI performance, make sure you check out EqualLogic iSCSI SAN solutions as well as those EMC and Netapp boxes.

Oracle is certainly one of the most important applications for many customers and there are a lot of misconceptions about its performance as Kevin Closson points out in this somewhat humorous MANLY MAN post on his blog. FWIW, EqualLogic has a number of customers running Oracle databases and applications on iSCSI storage networks - both with SATA and SAS drives.

Considering the nature of EMC Zilla-man's post earlier today and referenced in my posting immediately below - here is an interesting report from Network Appliance showing the relative performance of Oracle over FC, NFS and iSCSI. The fact that EqualLogic doesn't have the same bottleneck built into its iSCSI products, you can see how Oracle performance on EqualLogic storage arrays can be very good - and it explains why we have so many customers that are perfectly happy running Oracle on EqualLogic iSCSI storage networks.

Jon Toigo, DrunkenData, posted on what he thought was unfair treatment on Zillaman's blog, particularly the posting where Zillaman likened JT to SouthPark's Eric Cartman character. JT apparently thinks this was inappropriate and that others (including me) were guilty of piling on. Hmmm....You can't have it both ways Jon, seeing as part of your shtick has always been ridicule. There's nothing wrong with that, by the way, its just that you really ought to roll with the punches a little better.

Stephen Foskett on the business model change Netapp is apparently considering.

For those of you at Netapp, trying to figure out how to deal with us Ankle Biters, these links may be helpful to you:
http://ph.answers.yahoo.com/question/index?qid=20070611205338AAtuZs0
http://www.supportsusa.com/ankle/
http://www.footphysicians.com/what-is/ns_smelly-feet.htm

OK, So Hu Yoshida asked a question that seems a bit silly to a lot of us storage geeks. I guess this just means that Hu's storage interests are mostly at work and that he doesn't relish the stuff at home as much as some of the rest of us storage sickos. So whether you work at EqualLogic, EMC, ESG, Netapp, HDS, HP, SUN, iSTOR, Lefthand, Xiotech, Microsoft, IBM, Data Domain, Gear 6, Adaptec, Acopia, Ibrix, Red Hat, Compellent and all the other companies I forgot to name - here's the question for you:

Click To Play

A couple weeks ago the RupturedMonkey blog site asked blog readers about the television show Heroes. While Ruptured Monkey expressed the belief that this is off the storage topic, I'm not so sure. It seems from reading the comments to that post that lots of storage geeks have a hankering science fiction shows. I think it has something to do with our livelihoods being based on the strange phenomenon of data, its abstract relationship to our world, as well as it's storage and "livelihood". Let me know what's up - what sci fi shows can you never get enough of. Anybody besides me like Threshold?

Click To Play

I'm very happy to say we made the cut-over to Movable Type. I'm sure there will continue to be some adjustments here, but some things like commenting will be a lot easier for blog readers than the old system. Not only that, I have to believe the problems of lost posts and comments are now behind us.

Thanks everybody for your patience while we've been fixing our wagon.

Well, as promised, the new blog is up and running on Movable Type. We've had some problems with feeds and IE 6.0 rendering, but all in all, I like this environment much better than the old blog software.

OK, I've been a bit behind on a few things - well more than a few things. I finally got around to cleaning out my google blog reader (the things like a virtual garage of opinions) and came across this travelogue rant from Steve Duplessie at ESG. OMG - its more of a short story than a blog post, but it is really funny stuff and as it often is with Steve - straight from the heart.

Here's a choice excerpt: As if that's not bad enough - in mid sentence he whips out a napkin and proceeds to blow his nose - making a 15 second sound that can only be described as the same noise a very heavy table (one with a marble top) being dragged across an unfinished tile floor by a grunting Steffie Graff would make.

For a good time, click: "Lessons Learned in Italy"

Storage veteran Nik Simpson at the Burton Group writes thought provoking articles for their Data Center Strategies blog. Today he wrote about SSPs (storage service providers) – and why they didn’t succeed several years ago and what his thoughts are about how they could become more successful in the future. He concludes that the best application for SSPs is archiving and I agree with him, perhaps for slightly different reasons.

To me outsourced archiving makes sense as a corporate governance practice. If a company (and their IT organization in particular) wants to remove any implication of tampering with archived data, the best way to do it is through an SSP that provides the functionality. The idea is that once the data is sent to the SSP, the SSP more or less “owns it”, along with the responsibility of keeping it available for legal, audit and corporate reasons. Like WORM, but as a service. No writes, no updates.

It follows that the missing link for this is encryption of the data at the SSP site, which brings into play all sorts of thorny issues surrounding key management. The SSP needs to be able to share its storage resources among its customer base AND guarantee complete privacy of data. It is economically backwards to fence data on along physical storage boundaries, but customers are not keen on trusting logical fencing methods, such as partitioning, for fear of hacking and operator errors. Encryption with logical storage partitions would probably be acceptable and affordable.

I've been out and about for a couple weeks and am back now.

I want to say thanks to Lori Widmer for giving us a mention at On-Storage.com on October 2nd. She picked up on the Eric Schott video where he gives an overview of EqualLogic replication.

If there is one thing I really like, its satire. So, I have to point you to this posting:

Actually, I'm not too worried about THAT happening. For starters I expect that Dell values the capabilities of our top notch support organization and their role in building our customer loyalty. More importantly, Dell's business divisions are being recognized as having excellent support as Keycruncher points out.

Adding to the list of comments is Andrew Reichman from Forrester who concludes his posting by saying: "There will be concerns around product roadmap continuity and focus, as well questions about placement of EqualLogic products within the current Dell storage lineup, but these are likely to be minimal in comparison to the strong technology benefits and complementary fit between the two companies". His entire posting can be read here.

Steve Duplessie referred to EqualLogic employees this week as "flannel clad folks", Say that 5 times in a row. The funny thing is, its not that far off. He has a previously unseen photo of our dignified founders on his site here.

There has been a lot of speculation about the future of our channel business when the acquisition completes in a few months and I thought I'd post links to what I'm reading in the bloglands.

Robin Harris, Storagemojo: The Bastards Say Welcome
Barbara Darrow, Tech Target's Channel Market: Dell Seeks Partner Progress in EqualLogic Buy
The Var Guy, TechIQ: Will Dell Push Storage Through Partners?
Joseph Kovar, CMP: VARs say Dell's EqualLogic Acquisition Could Backfire
Edward F. Moltzen, CMP: Dell Bags EqualLogic for $1.4 Billion, Says Deal Will be Channel 'Catalyst'
Dan Bricklin, Theirway.net: Congratulations Don Bulens and Equallogic
Sharon Lisenbach, Eweek: Dell: EqualLogic Buy Validates Channel Commitment
Brad Anderson, Direct2Dell: Dell to Acquire EqualLogic

Lucas Mearian at Computerworld wrote about Dell's acquisition of EqualLogic and concludes that it will force a split in the Dell/EMC partnership.

I think Lucas is an astute observer of the industry, but my previous opinion posted on the VMTN forums and my agreement with Chuck Hollis' blog posting yesterday weren't swayed by what Lucas had to say.

Dell and EMC both have a lot more at stake here than the iSCSI business that EqualLogic brings to the table. They do business with a much broader set of products than iSCSI storage - such as VMware. Dell is a very valuable sales channel for EMC and EMC is a very important technology provider for Dell. There will always be some level of competition among the largest companies in the technology sector. It creates a bit of a mess sometimes, but having interdependencies between companies provides stability and efficiencies that ultimately result in broader solutions at lower costs.

I suspect this deal will actually end up creating additional opportunities to expand the business between Dell and EMC. I'm not always right about these things, but that's what my intuition is telling me.

Here are links to a couple pieces that showed up in the last 24 hours regarding Dell's acquisition of EqualLogic. Both reflect the writer's intent to present fresh, thoughtful perspectives.

The first is by longtime technology editor Bill Snyder in his InfoWorld blog, Tech's Bottom Line, where he combines technology and financial analysis.

The other is from ZDnet Australia, where author Liam Tung provides regional commentary. I always appreciate the candor from down under.

Chris Preimesberger wrote about the Dell EqualLogic news @ Channelinsider.com.

Tony Pearson, storage blogger for IBM, wrote about the impact on the rest of the industry from the Dell EqualLogic aquisition. As Tony likes to do (and what he's paid to do), he turned it into a platform for talking about IBM's storage products.

I want to correct him on his EqualLogic history. Apparently after surfing or talking to IBM market researchers he came across news that we had signed an agreement with Sun several years ago. Figuring that such a deal must have been an important part of our rapid growth he pieced together an odd collection of Jabberwocky on the topic. What he didn't surmise was that the Sun EqualLogic agreement turned out to be one of those worthless pieces of paper that mostly wasted human and natural resources.

The real story is that we have been 100% focussed on selling through our VAR channel for a couple years now since our fearless leader, Don Bulens, came aboard. FWIW, Don doesn't really care for the spotlight much but he needs to be given credit for making this business transition - without our committed channel business model, I don't think Dell would have paid $1.4B in cash for us. These types of transitions sound easy when you read about them, but they aren't that easy when you are the one steering the boat through the rapids.

Here we go again on a new transition - one that will be much more visible to industry observers than our apparently mysterious past.

This post is for people wondering how to set up iSCSI drivers for Solaris.

It has not been tested or approved by EqualLogic. I'm just passing along a link that looks like it would be useful to blog readers. Please send any comments about its contents to the author on the Weibeltech site.

I like Chris Wolf's take on Oracle's OVM. It looks like Oracle is trying to wedge its way into the VM business using proprietary-support tricks as their competitive advantage.

It's somewhat ironic that Oracle, a company that won the DB battles decades ago with aggressive cross-platform support, is taking this approach. Like Chris, I know several customers that are running Oracle very successfully on VMware machines and are also using EqualLogic iSCSI SAN storage.

Tony Asaro blogged about the impact that Dell's planned acquisition of EqualLogic will have on the iSCSI market. I agree with almost everything he said here, except one. He thinks there is still room for an iSCSI startup to emerge as the thought leader for this segment. I think EqualLogic is the thought leader in iSCSI and that's why we've had the successes we've had so far.

The development of EqualLogic's iSCSI SAN technology and products will continue - and will likely accelerate after the acquisition closes. There are many more things to accomplish at EqualLogic and we are going to continue to pursue them and tackle them one by one - just as we have been doing.

Arun Teneja posted today on the requirements for surveillance video storage.

His main point seems to be that there is going to be lots of data and so your storage needs to be dirt cheap. But then he says that the storage needs to scale a single image and support clustering. Oh ya, dirt cheap clustering, single image scaling and cows that jump over the moon.

I don't agree that clustering is needed, but you do need systems that can survive component failures. The ability to mix and match components as you grow is very important. You don't want a system where all nodes have to have identical configurations to work together - that makes upgrading almost impossible.

You get what you pay for. In the end you want high availability combined with storage that is the easiest to operate, maintain and expand. He thinks EqualLogic costs too much, but what he probably doesn't know is that we have people using our products for large scale surveillance operations that are very happy with it because it allows them to concentrate on the application and not the technology. It's not the purchase price, its the ownership costs that make storage valuable.

Jason Powell at Granger Community Church in Northern Indiana is hosting a lunch and demo/seminar at his site on Feb 22nd. Jason and GCC are excellent customers of ours and Jason is an active blogger about all the technology they use at GCC. I encourage you to check out their website at the link above.

Alan Hunt from VR6 Systems will be there too. Alan is an excellent resource having been a customer of ours for a couple years prior to joining VR6, an EqualLogic and Data Domain business partner. Alan and Jason both have extensive experience with our products and our company and will give you the straight scoop on what it is like to be an EqualLogic customer. Better to hear it directly from people who actually use our iSCSI SAN storage, than to take our word for it.

Thanks a ton Jason! and Thanks to you too Alan, you guys are the greatest.

Thought I'd post links to comments on yesterday's announcement.

ComputerWorld
Network World
SearchStorage
ITPro (UK)
Infostor
Internetnews
Techtaxi
Informationweek
The Register (UK)
eWeek
scaledatacom blog
bmighty.com blog
Computer Cosmos blog
infiniteadmin blog
itbusiness (Canada)

Josh Caster writes the InfiniteAdmin blog and has been exploring the business and technology involved in Dell's acquisition of EqualLogic. His most recent posting explores storage tiering and Josh asks a number of good questions related to mixing drive types within an array. I thought I'd address them here.

EqualLogic's design goals are driven by ease of use/administration. Our iSCSI SAN storage systems have a high degree of automation, including such things as dynamic relocation of data for load balancing and tiering. They also have a high level of virtualization, which makes it very easy for customers to provision storage. Keeping the drive type uniform (not mixing different types) within an array provides the simplest administrative environment by eliminating most volume management decisions. We stripe data across all the drives (except hot spares), which provides the best overall throughput. Customers don't create future problems by establishing RAID sub groups that might trap or lock-in storage capacity. When drive types can be mixed freely inside an array, it forces the customer to make volume management decisions that they might not have adequate knowledge or foresight to make. Sparing is easier too if drive types aren't mixed. At the end of the day, it's easier for customers, our support engineers and our development team to not allow drive mixing within a cabinet.

Of course, this means that our customers can't get tiering within a single cabinet, as Josh points out but he also acknowledges that tiering works with two or more of our arrays. Using the same core technology that drives our load balancing capabilities, our iSCSI SAN storage arrays automatically and dynamically locate data on the most appropriate array. In other words, customers don't have to guess where to locate data and don't have to manually move data. It's a lot more than basic disk-level tiering where customers have to provide their own load balancing expertise because Dell Equallogic arrays constantly analyze and update their data locations and tier mappings.

Finally, there are network and controller variables to consider. Storage tiers that are located within different systems use different network connections and different controllers, which means there is more network and processor overhead and less chance that I/O bottlenecks will occur.

There's a lot of speculation and analysis about what's going on at Dell, considering the company's acquisitions and movements in software, storage, services and its channel programs. I think it will help people interpret these actions better if they understand that Dell is actually very serious about simplifying IT.

It's not really about being in the software industry. It's not really about being in the storage industry. It's not really about being in the services industry. It's all about simplifying information technology. If you think that Nicholas Carr might be right about IT not being a winning differentiator for most companies, then it's extremely important to reduce the cost of IT. Organizations have unique needs and will achieve different benefits from a selection of technology options. The goal in our industry is to provide the best options.

Here are a couple more blogs that I think have a handle as to what is going on here at Dell.

from Illuminata

from ZDnet

A few postings in reaction to Dell's acquisition of MessageOne:

Beth Pariseau from Search Storage

Stephanie Balaouras from Forrester Reseach (Beth's blog references Stephanie's blog)

Howard Marks at Information Week writing about MessageOne's service offerings.

Terry Sweeney - also of Information Week on the trend for storage vendors to sell services.

Betsy Schiffman at Wired with a somewhat skeptical view based on the relationships of the players.

There will always be skeptics about M&A deals involving family members - it comes with the territory. However, where this one is concerned there are many opinions from people outside of Dell that understand why it makes sense. I've pointed out a few in my last post here.

So it turns out the S3 outage was related to a failure in authentication services. One of many things you don't have to worry about if you have your own storage. My advice is to get powerful automation storage tools in house.

Zilla-man deserves the James Joyce award for his post yesterday. Great writing Zilla-man!

As much as I like Nick Carr's writing, he has this one wrong. His writing on this reminds me of the many business plans I've seen from networking people who tried unsuccessfully to enter the storage business thinking that it would be easy. ("All those storage curmudgeons are paleolithic throwbacks"). Nope. Its a different game in storage land. There are many, many impact craters of entrepreneurs that never understood storage psych101.

Here is the link to the play by play on Amazon's Web Services site.

Nik Simpson writes about a software SAN installation that went awry.

In contrast, Dell Equallogic iSCSI hardware-based systems are simple to install and understand - especially the licensing part because there are no licenses that expire.

Marc Farley is working on some new things and some old things in a new environment. He shares a little of his experience with blog viewers here.

Click To Play

Dan Kusnetzky, in his VIRTUALLY SPEAKING blog wrote about EqualLogic's approach to storage virtualization.

He was talking to JJ and they had a senior moment discussing the technology legacy that started many years ago at Digital Equipment Corporation. FWIW, that really is the mojo of the development team and it's nice to see a nod in that direction, so thanks Dan.

But it reminds me, I think it's time to make a short video on how the storage paging works.

I've had a lot of people ask me why EqualLogic sold itself to Dell. Yes, the money mattered a lot, but it was more than the money. We didn't want to just take the money. If we were going to be sold, it needed to be someplace with a passion for what we were doing.

Matt Baker posted today on Direct2Dell about Dell's passion for iSCSI.

We are singing out of the same songbook - to quote Matt: So, when considering the economic benefits, the innovative storage models it facilitates, and the unrivaled flexibility, it is hard not to be hot on iSCSI