Integrated Data Storage Blogs

OK, so you have decided that deduplication is the best thing ever and a must have for your backup needs. The next big question on the horizon is what KIND of deduplication is right for you. Two of the big hitters in the market today are EMC’s Avamar and CommVault’s Simpana products. Both products seem to be doing very well in the wild and both approach deduplication in completely different manners.

In the case of Avamar, the product is deduplicating at the client using variable block deduplication. Once the scan is complete on the client server and the deduplication hash is created the client actually checks back with the Avamar Data Store appliance farm to see which blocks the farm has not seen and then only the truly unique blocks across the environment (not just that server) are sent over the wire. This results is extremely high levels of deduplication AND remarkably fast backups since very little data is normally left to send after deduplication and comparison to the rest of the environment. The data is stored on the EMC Data Store appliance which presents a pretty simple GUI for the recovery. The only major chink in the armor of Avamar is that it does not have the ability to natively create tapes for those data sets you may want to retain longer than you have space to keep on the appliance farm.

CommVault came at the deduplication process a completely different way and leveraged their existing tape archive construct to create a form of fixed block deduplication. In this case the clients do the same thing they always did: run their scans, package the data, and then shoot it out. Once the data gets to the Media Agent the deduplication occurs and the data is spit out onto any disk target supported by CommVault. Since the deduplication is fixed block, the deduplication ratios are not as good as with variable block, aka Avamar, but certainly much better than typical compression. Since the deduplication occurs on the Media Agent, there is no savings in backup window time. The good news is that this is CommVault and cutting tapes is it’s forte and completely automated with the ability to have different retentions on each type of media to fit all your compliance desires within a single tool. Also, since the format of the media archive did not change, restores are just as fast with deduplicated data as they were with plain Jane backup to disk, which is huge if you have a lot of data to restore. Avamar can be sluggish in terms of restore on the smaller deployments but still certainly functional for your every day restore needs.

On the grand scheme, Avamar is the holy grail of backup speed since it only every sends fractions of incremental data over the wire to the target which reduces not only backup times but the impact of backup on both the hosts and the network. I also give CommVault a major tip of the hat in how they leveraged their existing technology and morphed it into a deduplication technology that brings huge benefits to their current customer base while staying on the commodity hardware bandwagon.

Obviously there are many more features to both products worth investigating and comparing but now you know how the two differ technically in terms of the deduplication angle.  

More: Video interview of IT manager who achieved a 20:1 ratio using deduplication technology

Chuck Hollis did a pretty decent job (in my opinion) of putting a good public face on the EMC Data Domain acquisition on his blog.  Steve Duplessie is missing the boat a little bit (again, my opinion) in his blog entry on the issue, but I do think he gets a couple of things right.

Steve's point about EMC having 82 ways to do dedupe is misguided and feeds into a FUD-Fest for all EMC's competitors in the marketplace.  Now maybe this should be blamed on EMC's marketing group for not delivering a crystal clear message around deduplication, compression, and single instance storage.  I can buy that argument.  But laying this cacauphony of competing dedupe strategies in the marketplace at EMC's feet isn't right or fair.  Every player in the market has some responsibility for that issue. 

I also don't see EMC having another way of deduplicating data as a negative.  I quite frankly get tired of hearing the mantra'd pablem of "EMC has 82 ways of doing <insert functionality here>.  Which one will they try to sell you today?"  That drivel typically comes from the same camp that tries to fix their plumbing problems at home with a hammer and then wonders why the bathroom is full of water.  Get over it.  If you had billions in the bank to buy companies that fit your long-term strategy you'd do the same thing.  Good strategies aren't built around a single way of doing things.  That's what got EMC into so much trouble in the early part of 2000-2001.  Everything was done their way (Symmetrix) or you were an idiot and not worthy to buy their technology.  EMC provides choice to their customers and flexibility in how to design a solution for their most pressing information problems.  Why does everyone beat them up for that?

Anyway, I could go on for a whole post on that topic alone, so let's get back to the topic of the moment.  The acquisition makes perfect sense on two levels:

1. It stops a major competitor in the marketplace from getting better.  So what if this is a defensive move on EMC's part?  Defense wins championships.  Good for Joe and the EMC Board for executing an excellent defensive maneuver.  As a shareholder, I applaud them for not letting NTAP walk in and change the marketplace (again).  Thank you for finally recognizing that while backup as we know it is in a death spiral, the spiral is slow, and EMC needs to do more than try to convince everyone source-based dedupe is the salve that cures all wounds.  Now, go add to the investment and turn the R & D group loose on this code and get it everywhere as fast as possible.  Don't kill the product, don't take a 5 year ramp to understand how to use it, just go get it done!  I want to see products with elements of this code embedded in them within a year.  You brag about all your R & D, now let's see something real come out of it.
2. The code is good.  The device just works, and it isn't because of the hardware.  It's because of DD's code and approach.  Why not get the best code you can for this feature?  EMC knows more about backup than just about anybody in the marketplace, and when they turn this product line loose on their sales force, watch what happens.  It will be nothing short of amazing, if history is any past indication of potential success.  EMC spends a lot of time training their folks on backup, as well they should, and the only real challenge they'll have is getting their faithful followers to stop spouting the 'Data Domain sucks' mantra.

As a reseller of both EMC and Data Domain, this is a winning combination for us.  We no longer have to hear about how bad DD is from EMC, and fend off the lesser (though workable) Quantum solution when we believe target dedupe is called for.  We will get better support from the larger EMC support org, and have everything under one roof.  That helps our business quite a bit.  It takes some of the best minds in backup in the industry and puts them in one house, which for the partner community makes life easier.  Woo Hoo!

That said, here's what I don't like: 

1. EMC paid way too much for the feature, at least on the face of it.  It's going to be tough to recoup this, even with stellar execution.  They should have bought this company at least 2 years ago.   It was a miss on their part to not have done so.  I'm sure if you add in the lost opportunity cost for NTAP, EMC will make out like bandits, but in Joe's stellar track record of acquisitions, I just think this one is at least a partial miss, only because the timing was so late.  But, sometimes you have to overpay and suck it up.  This is probably one of those times.  It just makes me wonder what else is being missed that could change the landscape of the industry?
2. I fear Quantum is dead.  This is where things get a little ugly.  Can Quantum survive on tape alone?  I doubt it.  Their engine has been soundly decried as inadequate by their largest proponent.  That's not good for business.  I like the tape products for what they do, and it will be disappointing to see Quantum flounder further, as they have some really good folks over there who work hard to do what they do.  Perhaps EMC will finally make it official and forgive the $100M they loaned them by just buying them up, finally putting a tape product directly into their catalog.  There are lots of reasons this doesn't make sense, but some that do.  We'll obviously have to wait and see.

At the end of the day, target dedupe is a technology with a very limited lifespan, just as all VTL type products.  CDP technologies, automated information movement platforms, and improved compression techniques that save replication bandwidth costs will make 'backup' as we know it a dead idea in a matter of years.  How many, I can't say.  But if EMC (and others) keep going down the path of newer, tighter, faster compression and dedupe algorithms that make it possible to transmit terabyte data sets across less than OC-level bandwidth in reasonable timeframes, and maybe even T1/T3 types of bandwidth, we won't be having a conversation about backup anymore.  This applies across the board, little AX4/NX4 companies and V-Max companies alike.  They all suffer from the same problems, and these technologies allow for the problem to be solved at both ends of the scale spectrum. 

So get on the pony and ride, EMC.  Ride fast and get this acquisition done and the product into R & D.  Get us new stuff ASAP!

Having good data storage documentation makes you and your team more effective and efficient. Most importantly, though,  good storage documentation puts all the information you need at your fingertips in the event when something goes wrong (which, at some point, it will). What does good documentation look like, then? Data storage documentation—whether you create it yourself or your vendor provides it—should possess five key elements.

1. Ideally, your documentation should start with a descriptive overview of your environment. This is important for new team members, managers and consultants, allowing them to quickly familiarize themselves with your environment and what you are trying to accomplish with your infrastructure. From there we get into the meat of the documentation.

2. The document that you will probably reference the most is the connectivity map. This document, often a MS Visio diagram, should visually describe how each of the devices (servers, storage, switches, tape/virtual tape, etc.) in your storage environment are connected. These diagrams can be accurate down to the ports on the fibre channel switches and individual devices. It is often useful to include a chart on this diagram that has management interface IP addresses and device WWPNs. An additional diagram or chart included on this or near it is a storage layout document that provides a physical perspective on the disk groups. Maintaining the accuracy of these documents is a key factor in being able to plan, troubleshoot and maintain your storage environment.

3. The configuration detail section should have all the nitty-gritty details for each storage, connectivity and host device in your SAN environment. Much of this can be gathered with manufacturer provided tools and then compiled; however, key elements to record are management interface information (IP, user(s), passwords), driver versions, firmware versions, and software versions. Other items to record include WWPN addresses and switch port connections.

4. Regardless of the size of your environment, you should maintain a change-log. At a minimum, record when a change was made, who made it, and what was done. This will prove invaluable when something breaks or something is "fixed." The change-log can provide critical insight during failure analysis or when troubleshooting performance problems. Procedural guides provide a quick way to refresh your memory or assist new team members with tasks that are not frequently performed. Whether it is configuring a new server on the SAN, setting up a new replication consistency group or adding drives to your NAS, documenting the steps and using that document as a checklist provides consistent, repeatable results.

5. The most critical part of your documentation is the support information. Having the manufacturer phone numbers, site IDs, and device serial numbers (from the configuration detail) at your fingertips will shave critical time off of problem resolution. It is also important to have your integrator's contact information in this section, as they can serve as a liaison with the manufacturer to escalate cases when necessary.

Why am I occupying real estate in cyberspace?

This is the question I'll try to answer in this, my first blog post.

Why, in the name of all that is good and holy, am I writing a blog? There are probably billions of people in the world that know more than I do about any one of the topics I might post about. So why would I do this? Well, the fact of the matter is, my boss wants me to.

I asked him not to make me do it, so if you end up getting upset over something I write, take it up with him. :-] When I asked him why he wanted me to do this, his answer was pretty simple. I tell people what I think, based upon reason and facts, and I communicate what I think with passion based on those reasons. He didn't put it QUITE that way, but that's what he meant. I think.

I'm not sure how that qualifies me to be a blogger. I really don't. I regularly read Chad Sakac's blog over at VirtualGeek, Chuck Hollis' blog, StorageZilla, Storage Anarchist, Barry Whyte, and a lot of other guys and gals out there and never once put myself in the category of people that should regularly be read by a card-carrying, dyed-in-the-wool, proud to be geek. Not once. Still don't.

That said, I'm supposed to write. What can you expect from me? You can expect a lot of opinion. I'm probably not going to tell you how to configure your Avamar system for maximum throughput of de-duplicated bits across a converged network. I'm probably not going to give you great bits of Linux tweaking advice or try to be crowned the ESX guru of the century with sage advice on how not to tune all the kernel parameters. I'll tell you what I think of things going on in the technology world. I'll tell you how I think they really matter, or really don't matter, to real customers, with real data centers, and real jobs they're trying to keep. I'll try to make sense out of the hype, the homerism, and the marketing stuff and boil it down. If I do that, then maybe people will read. If I miss the mark, probably not.

Who am I?

I'm the newest member of IDS' engineering team. I come from almost 8 years of experience at EMC. While there, I supported Sprint PCS, the Channel, the Proven Professional program (yes, you can blame me for a lot of the questions on the E20-521 exam and a few others), and the Commercial Division. I was a Systems Engineer, Technical Consultant, District Channel Manager, Technology Solutions Manager, and a Global Solutions Architect. I did a lot of things in 8 years. Before that, I spent several years in the telco space, with a CLEC and an (at the time) up-start wireless carrier.

Why does any of that matter? To you, it might not. To customers, it means that I've seen a lot of different things. I've been involved in complete project failures, and partial project successes. I've helped customers who have 'data centers' the size of a small broom cabinet to customers with data centers the size of a shopping mall do one thing: meld the technical and the business.

That's why I do what I do (besides the fact it feeds my seven kids -- do you KNOW how much food seven kids can eat? That's another blog for another day :-]). Anyway, it's what I do. I knew from the time I took apart an Apple IIe in 5th grade and made it work with the LOGO turtle robot, that making computers do things for people in ways that mattered was what I wanted to do.

So, that's what I try to do.

I'm not always right (mostly not), but I always have a reason for what I think. I can always tell you why I think what I think and endeavor to do so in terms that make sense TO YOU. You don't have to agree with me. You don't have to like me. I honestly don't care either way. I'm going to make more than a few of you upset, I'm sure. I'll probably start more than a few flame wars, and I'm okay with that. As long as things stay civil, we'll all have some fun with spirited debate (assuming anyone actually reads my drivel). I have always thought that Chuck Hollis' approach to things was the right way to go -- let comments come and allow them to be seen even when they disagree with you, but I won't allow this blog to become a sounding board of derogatory remarks or silliness. Comments will be moderated.

SO, without further ado.............

Why the V-Max was cool, but not all that exciting to me.

Now before all you EMCers go off telling me I've betrayed the family and how could I defame the good name Symmetrix, hear me out. And anti-EMCers, back off. There's no blood in the water here or reason for you to take a swipe. I could fill reams of paper with the utter ho-hum nature of your last 3 or 4 product "updates".

I'm very impressed with the technology. I'm floored by the technology, actually. I think it's quite amazing what's been done to take DMX concepts into a virtualized hardware layer that then (without too much hyperbole) can infinitely scale (there's probably still some hyperbole there, because nothing scales infinitely outside of Time, but still).  The scale we're talking about here is truly astounding for a storage subsystem.

The fact that manageability has taken a quantum leap forward for the Symmetrix is an indication that the engineering sea change that was supposed to have happened several years ago may have actually begun to occur, and Symmetrix Engineering may have begun to realize that 1) not all customers are idiots who can't manage complex technology, but 2) not all customers want to obtain a PhD in how to manage storage technology because some of them have lives. That's a whole lot of goodness as far as I'm concerned.

The geek in me looks at this thing and says 'holy cow, how'd they really do that'? The fact that this technology enables VMware to truly scale and become serious about commoditizing the compute layer is utterly awesome. This, I think, along with a truly modularized server layer and the converged network, is what VMware needs in order to become the de facto data center OS.

I'm glad they moved to commodity processors. I see a ton of potential in management and down-market mobility afforded by moving to a commodity processor platform. I think this is an infinitely cool machine.

The business guy in me says 'great, another device on the market that is supposed to save the world, but only plays to the top 5%' (or less). I'm not sure how else to see this. Part of it is that I play in a space, and have for several years now, that looks at systems like the DMX, or V-Max, where to acquire one is more than the entire total cost of their infrastructure, including plumbing, electrical, the parking lot, the soda machine, oh, and IT. I get that my perspective is a little bit skewed toward the middle to lower end of the market. I'm happy to play here, because in my experience, they listen more, are more receptive to new ideas, and generally have a passion about what they do, more than most people working in large IT shops do. They're not a cog in the wheel, they ARE the wheel, so to speak.

So what set me off about the whole V-Max thing is how this is 'the vision of the future data center'. That's great. 150 of the data centers of the world will be able to take advantage of these ground-breaking features, functions, and benefits.

Okay, I'm being hyperbolic now, but you get the point. There's no down-stream message. If you don't play with V-Max, you aren't doing anything worth talking about. If you're not going to deploy something that can support 8 bazillion drives, you don't need the uber-integration with VMware (or perhaps aren't worthy of it).

Once again, EMC has basically laid it out that you aren't going to be included in the wave of data center efficiency if your data center consists of 3 or 4 ESX hosts running less than 100 VMs. Apparently it's okay for you to still do everything manually, with no end to end visibility and management capability.

Okay, now I'm being harsh. But again, you get the point.

Don't get me wrong, I love CLARiiON. I've sold and architected a LOT of it. It's a superb platform to everything in its space, hands down, bar none; NAS, SAN, whatever. What I guess I'm upset about is that EMC thinks the only place to revolutionize the industry is in the top 5%. And no, quite frankly, I do NOT consider the CX4 revolutionary. Putting modular connectivity was a must to stay ahead, not a revolutionary idea (servers have had PCIe slots you can put different multi-port cards into for years). But it goes deeper than that. Go back and compare the CX4 launch to the V-Max. Not even close in comparison. The CX4 launch was a blip on the sub-radar of the V-Max. There was no passion to it, no earth-shattering feel to it, just another generation of CLARiiON. 'yay'.

I understand how much revenue comes from that top space, but how much GROWTH? Not much, if any. I feel like EMC missed the opportunity to REALLY turn the tables upside down. Why not go completely downstream (not to the AX4 or CX4-120 space, but that CX4-240 or 480 level and up) with this massively scalable virtualized engine and introduce GREATER than 5 9s to the mid market? Scalable doesn't only mean bigger. It means it can grow and contract. It means it can start out small, AND SMALL MEANS <96 DRIVES IS AFFORDABLE.

This architecture should support that concept. These are Intel processors now, with even more commodity parts, right? Then bring the joy of mass production TO the masses, guys. Scale this baby down and turn the storage world upside down. How could HDS possibly compete with that? NTAP, are you kidding me? They couldn't touch it. Don't even get me started on Compellent, Xiotech, or 3Par. And Equallogic, errr, Dell? Right.

I'm sold on the V-Max conceptually. It's the reality I'm left to deal with, though, and I'm not feeling it. At least not yet.

Who Needs To Backup A Centera Anyway?

With the recent demise of the ill-fated Centera Backup and Recovery Manager (CBRM), it appears that EMC is throwing in the towel on Centera backup software. So who needs Centera backup anyway? Maybe you chose to just buy one Centera and put it in a room with a sprinkler head, better known as professional Russian Roulette. Or maybe professional paranoia has kept you from drinking all the EMC Kool-Aid on replication being the final answer to data protection on the Centera.

Backup is one of those necessary evils in many IT shops. But backing up the Centera by its very nature is not trivial, since it is not your run-of-the-mill array with your run-of-the-mill operating system. Centera’s code, CentraStar, is something far more purpose built (read “black box complete with Oompa Loompas and a secret handshake”) for the ultra compliant needs of healthcare records, federal geographic surveys, good homebrew recipes, or anything else that absolutely, positively must not be changed once written.

So how do you back up this genetically enhanced WORM? That question was hard for even EMC to answer! They went the NDMP route with CBRM as a bolt on to the likes of NetBackup and Legato, trying to treat it like a NAS, but the process was wrought with challenges from the outset. With a surprising nod from EMC, enter Seven10 Storage Software.

The little company based not so far from EMC itself built the better mousetrap with the self-contained Centera backup application ALTUS. Symantec and the other juggernauts of backup need not fear this little silver bullet as it is truly purpose built to the needs of Centera owners. ALTUS will require its own server and tape drive; but for a Centera, backup is not a performance discussion, so a light (VMware anyone?) footprint is looking like the way to go.

So it won’t cure all your backup woes, but if you will sleep better with a tape copy of your Centera, this gem may bring the ZZZZs.

(If you’re having trouble sleeping at night, the IDS Blog recommends Tylenol PM, a scotch on the rocks, or any Jane Austen novel.)

More: Video interview with IT Director who implemented an enterprise-class backup and replication solution using CommVault