Storage @ Work

What is the Dell = Logic position concerning the SPC? I see that Dell was an early member and there is one benchmark posted that I found, but it seems that they have quietly removed themselves from the EMC roster. Certainly BarryB of the storage anarchist presents a compelling argument in regards to possible bias of the SPC marks.

I don't know what the Dell position is, I'll try to find out. We weren't members of SPC at EqualLogic - probably not due to a philosophical stance, but more of a resource and cost issue. We ran very lean in technical marketing at Equallogic and invested more heavily into interoperability and integration efforts.

Anarchist tends to be heavily biased towards his employer (most of us are) but he is a talented writer - as are most of the bloggers who work for EMC. He also has solid technical chops. I don't know what to think of this whole mess, but it is very dangerous for SPC to expose any type of bias. It erodes their credibility if they aren't objective.

First let me start by saying that I've always liked what SPC was trying to do, having been involved early on as the author for DataCore's full disclosure reports in 2003. That said, I have to say I was very surprised that SPC permitted this result to be published. While it *may* have been acceptable within their terms, it doesn't pass the "smell test" for me.

NetApp's claim that they tuned the EMC box using EMC's published best practices may well be true, but we all know that performance information published for customer consumption is typically somewhat conservative, because the vendors don't want to have customers doing something that leads them to lose data in some "corner case" that the benchmark doesn't stress. It was notable that while they claimed to have use EMC's public tuning information, they didn't say anything about the methods used to tune the Netapp box, i.e. did they use their own published best practices, or did they tweak the hell out of the Netapp box?

Also, I'd have a lot more sympathy for both Netapp and Walter (SPC administrator) if this wasn't the first, and so far only time that NetApp has published an SPC result. If NetApp was only trying to prove their box was fast (a rather disingenuous claim that they've made) then why didn't they just publish their numbers and compare them to existing results? If they wanted to show that managing snaps didn't slow them down, then they could simply have compared their own results with and without snaps enabled. So it's clear that this was all about marketing for Netapp, and a slap in the face to EMC from the SPC.

I suspect that SPC has dug itself a hole hear because now they've accepted one result of this type they have to keep doing it and everybody's boxes are fare game.

PS, I'm surprised that this doesn't break the EMC purchase contract, which I'm sure prohibits customers from publishing benchmark results without EMC's permission. If the contract doesn't say that, then you can bet that it will be revised, along with the contracts used by every other vendor that doesn't want any Tom, Dick or Harry publishing benchmarking results for them! It's common practice for the database vendors to include this type of clause to prevent TPC results being published without their permission, so I suspect that storage vendors will do the same if they don't already.

Excellent comments, Nik. I agree - there is something here that doesn't sit right with the olfactories. I wonder if Netapp bought the EMC system used, and without an EMC contract.

BTW, if readers aren't familiar with Nik's work, they should follow this link and check it out.

Josh, As far as the SPC goes here, there aren't any hard lines drawn here, but the product team doesn't think their tests represent the application workloads that our customers generate.

i'm with nik. also i didn't even realize that was the first publication of specs for netapp--i could have sworn they published specs for the 3000 series and got into a sand-kicking contest w/ EMC over that too, but that must've been a whitepaper, not spc...

FYI, Here is a link to Beth's writings on the SearchStorage blog site

First, thanks for the compliments. The professional respect is mutual.

Second, compliments to your product team for recognizing that the SPC workloads aren't representative of real-world customer workloads.

In fact, although marketed as an OLTP benchmark, the SPC-1 is based on an I/O trace of a now-defunct email server running on a single (very old) server platform - a workload that bears no definable relationship to what we commonly know as "OLTP" today (it's not anything like any modern email server, for that matter). The SPC-2 models a streaming multimedia server (highly sequential I/O), and the (under-development) SPC-3 models backup-to-disk.

I doubt that it is any coincidence that these are all simplistic, spindle-limited, cache hostile workloads that do more to test the raw metal than how well the platform can handle real application workloads.

Third, I tend to agree with the sentiments shared here: that the SPC has allowed NetApp to make a mockery of their organization purely for marketing benefit. Not only did the SPC change their policies regarding 3rd party testing just days before NetApp announced their results (after said policies hadn't been updated for nearly 5 years), they also allowed NetApp to modify the benchmark itself and to publish the results of that modified workload in direct contradiction to policy (by adding the unquantified additional workload of Snaps to the mix).

From my admittedly EMC-biased perspective, it is gratifying that there are others willing to raise and explore these questions of motivation and intent.

http://www.storageperformance.org/about/policies_v210.pdf

Have a look at page 30/31

The policy since October 2001 was that you were given 60 days notice before publication if someone benched your gear without your permission.

That policy was replaced with this one..

http://www.storageperformance.org/about/SPC_Policies_v3.pdf

..The weekend *after* NetApp and SPC went to the press with their results. Not before even though it's back dated. After. It was 2.10 on the Friday when I checked it, 3.0 on the Sunday when I checked it again.

What's missing? The 60 day notification window, which would have been used for the first time ever.

Hi Marc -- nice to see you back in blogland ...

Despite the fact that I agree with Marc's comment as "unrepresentative", I don't think this is about benchmarking, folks.

My cynical thinking leads me to a "follow the money" exercise. SPC makes money by getting vendors to join, as well as related PR services. Please note that no customers have joined SPC, only vendors.

The SPC won't exist for long unless it meets the needs of its paying customers, e.g. the marketing departments of storage vendors.

If you think about it, this move opens up a whole new market for Walter and the SPC, doesn't it?

EMC biases aside, I think this is a sorry situation for the industry, and I'm glad to see others who are experiencing olfactory discomfort with this recent turn of events.

And, no, I'm not gonna blog about this one again. I'm done for now.

Money is a great motivator. The things it motivates aren't always so great.

What's up with that? You mean this?

http://www.mediafire.com/?50ddff9a5rc

Does the 60 days apply to the above or only to the SPC1 results?

To Nic...SPC rules dictate the exact tunning steps be published and verified by the auditor on ALL systems tested. These steps are part of the full disclosure...

To Zilla...had EMC been a member of the SPC (or dell for that matter) not only would they have the chance to see the results before they got published but would also know as to WHO proposed the change for the 60 day waiting period...If it's convinient to claim it was netapp, then by all means you're feel free to believe that.

Eric,
The fact that the FDR contains all the tuning steps still doesn't answer the question as to whether Netapp used previously public tuning information to tune the Netapp box.

On the subject of EMC and membership of SPC. What you are basically saying is that SPC is being run like a protection racket, join them, or risk being sandbagged by your competition publishing results on your hardware. As Tony Soprano might say, "Nice business you've here, shame if it got damaged, you are insured against fire aren't you..." :-)