Storage @ Work

The University of Wisconsin has published some interesting data about the cost of dealing with SPAM. Published as a Microsoft PowerPoint file, their findings are are here: U of W SPAM costs analysis. I would have thought that SPAM storage would represent a sizable cost with a meaningful burden on the overall cost of storage. However, their findings show that the cost of storing SPAM is not all that high - given some reasonable anti-spam technology and practices in place. In their opinion, companies are best off storing some amount of spam and being able to access mail and attachments that might otherwise have been deleted as false positives. In other words, it is better to delete a bit less and not have to manage the problems of deleting too much.

FWIW, I've been using an anti-spam service called SpamStopsHere, located in Ann Arbor, Michigan. Its got a lot of flexibility and power and it works with most ISPs mail servers, but not all - because not all ISPs allow MX records to be configured as required. Their tech support folks always seem to know what's going on. If you are like me - a sysadmin for the family, I feel its definitely worth it to employ a decent spam filtering service.

Having teenagers in your household exposes you to a variety of interesting things. I hear the most interesting things when we go out driving. Check out this Muse video for their song Knights of Cydonia. (from the CD Black Holes and Revelations). At the end, the good guy takes out the bad guy by blasting him with a beam from an optical disk. Now that's the kind of storage I want!

My son showed me this silly Weird Al video on YouTube tonight and I just howled. With all the rollin' and bowlin' and pocket protectin' goin' on I didn't see any storage bling here. How can that be? Any good nerd should be showing his sto. If you see any let me know.

Two posts ago I sent my props to Robin Harris (Storagemojo), now I'm feeling the bloggers biteback. Robin - what are you doing?

In his most recent post, Storagemojo links over to a post of Jon Bach's (dated Feb 5, 07) from Puget Custom Computers where Jon discusses the problems of using cheap RAID controllers in desktop systems and he gives failure numbers for drives that he has been working with in his business. Jon argues that RAID in desktop system is a royal pain because repairing an array on these systems is more of a problem than restoring from backup. I'm not going to argue much with Jon here and I certainly don't have a bone to pick with him, but I can tell you with some certainty, that my own experience on my own desktop systems has been different. I would much rather replace and remirror a disk drive than restore an entire system. Then again, I only support my family and me, a conundrum of requirements and systems that would probably be incomprehensible to anybody but me. (I know there are readers out there that know what I'm talking about it). Jon apparently has a lot of good customers and I'm sure he does what is right for them.

My heartburn is with Robin for implying that Jon's numbers might be relevant for enterprise storage and implying (by reference) that RAID is a problem too. Geez, Robin, you know better. I'm not even going to address the RAID thing because that is so wrong. On the disk side....engineering teams at all major storage companies spend many, many hours qualifying drives and working with drive manufactuers to increase reliability. Then there is the by now famous burn-in process where drives are rigorously tested to weed out those likely to experience early life failures. Then there is some likelihood that these drives live in server rooms with better environmental conditions. Then there are advanced functions like background scrubbing to find and relocate bad blocks, yadda, yadda yadda. And it is not a bunch of whooey- it is a necessity for staying in business in the ultra-conservative enterprise storage market.

There are 3 main reasons the "big guys" like EMC, EqualLogic, Hitachi, HP, Netapp and everybody else don't publish their drive failure numbers.

1. Contractual restrictions
2. High percentage of NTFs (no trouble founds) Storage system vendors and drive manufacturers agree to disagree that a drive failed (which is probably why contractual restrictions about failure disclosures are in place). The drive manufacturers will always claim a lower percentage of failures than the storage system vendors and customers. This doesn't make the disk drive manufacturers less honest by the way - there actually are a high percentage of NTF drives returned where it appears there is nothing wrong with them. I'ts rumored that some of these drives show up in retail outlets and other channels - something that, if true, could possibly impact the numbers Jon Bach is seeing.
3. The first vendor to state their numbers loses - big time. Considering the amount of slop in the analysis, if any vendor were to give numbers for drive failures, all competitors would publish lower claims and then the great wrestling match of mind numbing analysis minutiae would start and foreheads would slap keyboards and drool would seep into the cracks.

And all these things that I'm pretty sure Robin understands, but it does make for fun blogging.

Loren Feldman at 1938Media is one of the best Internet video producers I've seen. Provactive commentary on Internet culture - especially Web 2.0. He just started a new gig yesterday producing videos for the Huffington Post. Don't expect to find much about storage on Loren's posts, but you can expect to be educated and stimulated by his direct, honest approach.

The LoneSysAdmin had a funny post today here.

I follow his blog because he's a Midwesterner, like myself and because he's funny and because he occasionally has some very interesting things to say about systems and storage.

This story says most of it:
http://www.crn.com/storage/199905754

Once upon a time Netapp was an Ankle Biter too. How does the saying go, "If you can't beat 'em...."?

So all you Netapp resellers out there wondering how to keep us at bay, I've got this to say - Doing business with a growing ankle biter might be the best decision you make this year.

OK, So Hu Yoshida asked a question that seems a bit silly to a lot of us storage geeks. I guess this just means that Hu's storage interests are mostly at work and that he doesn't relish the stuff at home as much as some of the rest of us storage sickos. So whether you work at EqualLogic, EMC, ESG, Netapp, HDS, HP, SUN, iSTOR, Lefthand, Xiotech, Microsoft, IBM, Data Domain, Gear 6, Adaptec, Acopia, Ibrix, Red Hat, Compellent and all the other companies I forgot to name - here's the question for you:

Click To Play

Processor Magazine recently published an article on iSCSI implementations that I was quoted in. That was OK, but the thing that really surprised me was that my old friend and comrade Jay Kramer from iSTOR was also quoted. Now Sixto (the author), you should have told me I was going to be partnered up with Jay because I would have insisted on a few perks, such as top billing and more quotes. Jay and I have circled the same aspects of this industry for many years: I was fabric and he was loop, he was SATA and I was SAS, but mostly he was Laurel and I was Hardy. Our best stint was probably tag-team-teaching storage networking technology to the theme of family feud. I'm not sure that Jay and I ever agree completely on anything, but I always enjoy getting together and arguing. Truth be told, I was pleasantly surprised to find him sharing this article with me and we certainly agree on the future of iSCSI.

I thought I'd reinforce a couple thoughts about best practices for iSCSI SANs that were in the article:

• Segregate SAN traffic from data traffic as best you can. Use an air-gap network, VLANs or subnets in that order. Its easier to manage changes if I/O paths can be isolated.
• Use CHAP authentication for further connection isolation. It removes a lot of the oops-darn-it risks that Fibre Channel administrators worry about.
• Minimize the number of hops between servers and storage that work together. Edge-core topologies are not necessarily the best policy. Think short for storage.

A couple weeks ago the RupturedMonkey blog site asked blog readers about the television show Heroes. While Ruptured Monkey expressed the belief that this is off the storage topic, I'm not so sure. It seems from reading the comments to that post that lots of storage geeks have a hankering science fiction shows. I think it has something to do with our livelihoods being based on the strange phenomenon of data, its abstract relationship to our world, as well as it's storage and "livelihood". Let me know what's up - what sci fi shows can you never get enough of. Anybody besides me like Threshold?

Click To Play

I'm very happy to say we made the cut-over to Movable Type. I'm sure there will continue to be some adjustments here, but some things like commenting will be a lot easier for blog readers than the old system. Not only that, I have to believe the problems of lost posts and comments are now behind us.

Thanks everybody for your patience while we've been fixing our wagon.

Well, as promised, the new blog is up and running on Movable Type. We've had some problems with feeds and IE 6.0 rendering, but all in all, I like this environment much better than the old blog software.

I'm a fan of the Tour de France. I think its one of the most intriguing sporting events every year. So many riders, lots of crazy moments, high speed thrills, crazy spectators and unbelievable demonstrations of physical endurance and strength.

Of course, we are coming to realize that some of those unbelievable displays actually are not to be believed. The leader of the race, Michael Rasmussen was thrown out by virtue of lying about his whereabouts and skipping drug tests in the weeks prior to the start of the race. When it came out that he was actually in Italy instead of Mexico, as he had claimed, his team fired him. One can only imagine how different the race would have been without him, but it is quite obvious it would have been significantly altered. Then there was Alexander Vinokurov, who received a secret blood transfusion from a donor with elevated oxygen levels. Now a blood transfusion doesn't seem like "doping" - but it is a fundamental way of changing your physiology during a long race such as this. How desperate do you have to be to swap blood with somebody else to try to get a competitive advantage.

Certain doping methods can't be detected weeks after they have been applied - even though the performance enhancing effects can be realized. This is the problem with Rasmussen's behavior before the race. He wasn't around to be tested and now there is no way to test for certain substances and methods. So he was tossed.

At the end of the day, I'm very happy to see ethics winning out. To watch what the television team of the Tour thinks, click this link and look for the video titled "Speak their mind #2" Phil Liggett, normally very a restrained professional doesn't hold back and at one point states: "May he never ride again" in reference to Rasmussen. I can't agree more.

A couple days ago, I wrote about the problems of the power distribution infrastructure as well as some thought I had about which storage technologies could contribute to easing the problems.

That got me thinking about the nature of infrastructures again and it occurred to me that we are close to hitting an inflection point in the market for data center electrical power. Here's the point - if the electrical generation and distribution infrastructure becomes too unreliable or too expensive, customers will find contingencies and alternatives to the infrastructure that give them the results they are looking for. Call it a hybrid power architecture.

If the facilities for alternative energy generation become cheap enough, why wouldn't businesses start augmenting their utility-supplied power with their own? I'm not talking about putting excess energy into the grid, but in storing excess energy locally where it is available as a backup power source to the business if the grid is unable to deliver. The challenge is storing electrical energy - this is a lot easier said than done, but battery technology is improving all the time.

Green power generation is happening already, but I think it will accelerate when businesses customers start seeing how to reduce the costs and risks of running a data center.

A world of hybrid power - am I smoking something funny?

Robin Harris, the StorageMojo, has a couple interesting posts on his ZDNet blog and his StorageMojo blog.

If you are interested in this topic, I think both are worth a look see.

First, this one from a competitor. Yikes! Somebody needs help here, but I don't think it's our customers.

Then in a strangely-related blog post, Jon Toigo, the irrepressible contrarian of storage, gave last week's news of Dell's acquisition of EqualLogic a yawn. Thank goodness he didn't say anything positive because then I would be worried. He goes on to say that SMB customers are primarily interested in cost. That's on the right track but I'd say value (what you get for your money) is actually much more important to them. Then he writes on how SMB admins prefer Harley tattoos to those from giant technology companies. Ummm, yes, I suppose that is true - definitely! But what about those tattooed, Harley-riding data center folks? Do you think they actually have three letter tattoos hidden somewhere that don't spell MOM?

As it turns out the leading tattoo computer company is probably Apple.

And for you science lovers out there, here is a bit of strangelet news. Do you think they are getting younger or older?

A couple nights ago I was watching the late news on the TV and amidst the car ads comes this eye and ear candy ad with tool-wielding models posing, working out and grinding metal and all that fun David Letterman type stuff. Very cool soundtrack. What's this? I'm thinking that maybe this will be the car of my dreams! Turns out its an ad for Dell laptops.

Maybe you've seen it too. Get me an ice water, man. Curtis, if you read this, get yourself calmed down before linking over to the video 'cause it could start fituatin' you.

Apparently, this is the first Devo song in 17 years?! Only a locust wouldn't notice. I know that everyone is not a Devo fan, but they really were the ultimate nerd band and it's great to hear the familiar sound again.

I sure hope this ad is effective in boosting Dell's lead in the desktop laptop market because I'd like to see more of the same. The imagination is full-on for what future EqualLogic SAN storage ads might someday look like. Think of the thematic material: iSCSI? RAID? Thin Provisioning? It just keeps getting better and better.

When people find out that my daughter chose to attend a small liberal arts college in Minnesota, the reaction is predictable: "does she know what she's getting herself into?" with some subsequent discussions of winter. Yes, I usually tell them, she was a child there and knows what to expect from the weather. Not much discussion about the school - I suppose there is an assumption that we packed her off to an enclave of religious fanaticism. The reality is that St Olaf College is an excellent college with a faculty dedicated to teaching undergraduates and stimulating above average thinking (sounds like Garrison Keillor I suppose) .

So when the 2008 Rhodes Scholarships were announced yesterday and two St Olaf students (out of a total of 32) were recipients it instantly became easier to talk about why St. Olaf. There is no better place for undergraduates to learn about math, chemistry, music, environmental studies and most all their programs. It’s a great place to send bright kids, despite the cold.

The annual Christmas concert is a huge deal, they've been showing them on PBS stations for years and this year they have a satellite broadcast to movie theaters around the country on December 2nd. If you want to hear the best choral music for Christmas, find a theater near you and go.

Gate agents that will help get you home for the holidays
Family and friends (who help me govern my ridiculous, ferocious chow consumption)
The economic means to be a glutton
Beautiful music and other fine things
Reasonable health
Guitar Hero
Gauntlet throwing brothers

I wonder if this idea got venture funding.

OK Sci Fi freaks, what did you think of Tin Man, the Wizard of Oz re-spin on the Sci-Fi channel?

The first movie-horror experience in my life was the flying monkey stuff in the original Wizard of Oz. Still creeps me out a bit after all these years. Tin Man so far seems to be doing a good job with the monkey thing - especially where they fly out of - the creep + sexy combo works for me.

I usually can't stand Richard Dreyfus, but he's not too bad so far in this. For the most part I saw the character and not the actor acting, which is my main gripe with him. Also, turning the cowardly lion into a clairvoyant was a nice touch - especially since there is one working for both sides. Guns are weak - why would you need guns in Oz when you can get nasty monkeys?

DG (Dorothy character) seems like a good transition, although there is something not quite right with the debutante good witch thing. She seems to take this stuff a bit too much in stride. The scene where the doting parents throw DG off the roof into the howling tornado was pretty good. Who cares if it wasn't exactly realistic to be standing on a sloped roof with a bada$# twister 5 feet away - the thought of casting your child into the vortex probably resonates with some parents.

Here is some new news regarding Dell's Channel Program. It all looks good to me and confirms what Dell has been saying all along about the planned EqualLogic acquisition: they are serious about building a VAR channel and steering their company towards growing their business through the channel.

Our business operations now look more consistent with Dell's channel direction. This ought to make things much easier for everybody - especially our customers and channel partners who support and build their businesses with EqualLogic iSCSI SAN storage. FWIW, don't assume that Dell channel program will automatically become EqualLogic's. There are likely going to be some differences for some time. But before we jump to conclusions, the deal still needs to be approved and closed.

Nicholas Carr writes very interesting stuff regularly about the impact of digital-life on society. Last week he blogged in reaction to Doris Lessing's acceptance speech for her Nobel prize. The comments that follow are an interesting read too.

However, most kids these days would probably read this and say something about how the curmudgeons just don't get it. My kids probably read lots more than I did when I was their age, they use their computers and the Internet instead of books (including comic books) and magazines.

Is their knowledge of the world less than mine at their age? Not by a long shot. We may have problems in education these days, but they don't originate with the "distractions" of technology. They have much more to do with a lack of funding and other societal issues that we are a part of.

Chuck Hollis posted on a video he saw on YouTube about EMC. It was made for a website called Wallstrip. While most of the video is a whimsical report on EMC, there is a short reference to EqualLogic in it.

I looked into Wallstrip and found they have a show on BlipTV. The description on BlipTV says: Wallstrip is where pop culture meets stock culture. Each day we take at a company who's stock is trading at or near an alltime high, and try to find the real world trend that explains why the stock is doing so well.

Its like something you'd see on the Daily Show or SNL. Be warned; there is an amazing amount of material here to divert your attention.

10 days ago, my mother had a heart attack. A minor one, but scary just the same. She hasn't exactly been great at exercising over the years and the family is concerned whether or not she will apply herself to her therapy. So I started trying to find a NIntendo Wii for her to play Wii Sports on last week. I figured she would dig the bowling. Turns out the things all sold out in the Christmas rush and nearly impossible to find - except through Wii scalpers on e-Bay That could have worked for me, but the shipping time didn't allow me to get it too her by her 81st birthday.

Then, out of the blue my brother calls me on my way to the airport to tell me he found an unclaimed Wii at a local game store (The wait-listed buyer didn't show). Sometimes fortune smiles on you. The short story is that my mom loves it and was all over the concept of Wii as therapy already. She hasn't bowled in years, but yesterday she bowled 4 virtual games - some with her sons, but some with her 4 and 6 year old grandkids. She gets the same glee from throwing a strike as she did when it was the real thing, but there aren't any wet & stinky bowling shoes to deal with.

So for home technology - the NIntendo Wii gets my vote as MVP. Now if we could just turn it into a media server too.

RED is an organization started by Bono and Bobby Shriver with the purpose of bringing life saving drugs to AIDS sufferers in Africa.

The idea is simple: companies sell products designated as RED products and when customers buy them, those companies make donations to RED. Here's a web page on the Dell site with a RED system that includes a RED version of Microsoft Vista.

You can join RED here and start looking for products. If you feel moved to make a donation, there is a link at the bottom of this page.

There is a video link on this page that shows the RED spot that Dell aired during the Super Bowl.

I saw this post yesterday from the managing director of Hosting365 in Dublin Ireland yesterday. I was impressed with the description of the event and their ability to react so quickly.

Cooling the IT infrastructure is a big deal and its why low power technology is so important. The question for the storage industry is what role it is going to play. Storage tiering can play an important role by using proportionately more lower-RPM disks than high-RPM disk drives. Technologies that utilize unpowered storage such as MAID or removable storage could also be effective, but its not clear how foolproof they are. We tend to feel better about devices that aren't turned off and on regularly.

SSDs might be an answer, but probably not with flash technology because power consumption is not much better than disk drives. Phase change memory (PCM) holds some promise, but its probably years away from being used for data center storage.

My advice for data centers that want to reduce their energy bills today is to start investigating/using tiering technology. Here is a link to a short tiered storage case study published by our business partner Virso in the UK. It's not a story explicitly about power savings, but power savings are a part of the overall money savings.

Storage guy always likes to talk about the things that are filling up corporate disk space. MP3s, 2000 copies of a media-filled powerpoint, porn disguised as spreadsheets, etc.

Then there is data clutter, like post it notes stuck to every conceivable storage surface. Today I did a search through my email for the word "lunch". Hoo boy, what a jackpot! They weren't taking up much space, but there were lots of 'em. Unfortunately, I didn't count before blowing them away. Stupid me, metrics matter.

How many lunch emails do you have in your inbox? If you could search corporate email, how big would your lunchbox be?

Loren Feldman of 1938 media has to be one of the funniest guys making web videos.

There are no sacred cows in Loren's world. An entertaining review if there ever was one.

Here is the Scoble video he cracks on in his review.

So Loren, U B de eggman?

When competing technologies emerge in the consumer space, the risks are very high for companies who challenge for supremacy.

I like coach Howland, a lot - and they have the best three extra guys!

">

BSG (BattleStar Galactica) starts up again tonight. Oh Frak!

R we all Cylons?
Does everybody die and get downloaded and re-made in a bathtub full of Vicks?
If my wife finds out I'm watching old episodes on my computer, how much grief will I take?
Do you get the support you need at home to be a true devotee?

This entry first appeared on Inside IT

Finding my stuff - real, physical stuff, not data stuff - seems to get more difficult all the time - especially the real stuff. Oh yeah and I can't find my things either. When Google House is invented, I'm buying.