David's Discussions

Sometimes, though rare, you do get something for free. VMware has had API’s built within the code since the beginning, and while some of the earlier features, such as VCB (VMware Consolidated Backup), were a little rough around the edges, a new set of APIs are due out in vSphere 4.1 that are really going to impact the performance and scalability of your virtual infrastructure.

I will be the first to admit that I tend to see things from the storage side of the equation, so this latest news is particularly interesting to me. But anyone into squeezing out the best bang for their virtualized hardware investment should be pretty jazzed about this. These latest APIs are targeted specifically to how VMware can leverage a “smart” storage array to make virtual guests go even faster on existing hardware. The new “family” of APIs are called the vStorage APIs for Array Integration (VAAI). This is to differentiate them from existing APIs such as the ones for data protection, multipathing, and Site Recovery Manager.

I have long said that we in IT don’t fix issues, we push them around, and this is exactly what these three new APIs do. More specifically, they take tasks that your server hardware is doing and move those tasks to the storage array hardware. This has two major benefits: The first is that the server resources such as CPU and RAM can now serve the tasks specific to the virtual machines rather than the “administrative” work of the underlying VMFS care and feeding. Secondly, these tasks take up considerable network traffic (IP or FC depending on your storage array networks flavor) between an array and the server infrastructure; so again, more of the network’s resources actually go to serving the needs of the business applications and less to the underpinnings of vSphere.

There are a few caveats (I know you saw it coming). It’s all about the block level access, raw device mappings and VMFS for now, so if you were one of the early NFS adopters you are going to have to sit on the bench for a while yet. Secondly, you will have to be using an array that supports the APIs (kinda obvious). The good news is that while EMC will be the first kid on the block with the new toys, it is based on standard SCSI commands, so other manufacturers should not be too far behind.

OK, now onto the goodies…

First up is the hardware accelerated locking. One of the features that makes VMware the data center tool it is, revolves around the ability for multiple physical systems to work together in a cluster. Since all the machines see all the guest files at the same time, file locking is a big deal. If any two servers try to write to the same file at the same time, well, bad things happen. This locking process takes commands before, during, and after an actual update. When you have many machines performing these updates, this amounts to millions of commands. The new API reduces a large number of lock commands to a single SCSI command. While this will have some performance impact, the reduced instruction set will allow VMware clusters to become much larger due to reduced effort to arbitrate all this locking.

The next new feature is called hardware accelerated zero. VMware zeros out blocks inside a virtual machine’s file when it expands. This means that as you add data to a virtual machine there are often two or more writes necessary to actually write your data to the file. This is a huge overhead. With the new API, the host only needs to tell the array how much space to zero out, and the array performs the task rather than the host. This can reduce the IO overhead from 2 to 10 fold. Data writes are already expensive in terms of parity calculation, so this will be a huge improvement in overall array write performance.

The final, and perhaps most obvious, tool introduced is hardware accelerated copy. Here, instead of the VMware hosts moving the files for storage vMotion and the creation of machines from VM templates, the copy request is given to the array which simply moves the data internally. While not an everyday occurrence in most shops, the savings in network, array, and server resources are huge.

Personally, the most amazing thing about these new tools is that (assuming your array supports them) you will be getting them for free when vSphere 4.1 comes out later this year. I guess we can chalk it up to our maintenance dollars at work deep in the confines of VMware. So keep those maintenance contracts up to date—the upgrade is going to be worth it!

OK. I admit it. Some of the best ideas (and epiphanies) I bring to my customers don’t come from me, but instead come from other customers. Case in point, I was recently having a conversation with a long time customer of mine who has, over the years, become a good friend, so we routinely bounce ideas off one another to make sure our math works out and we are not falling prey to the continual “marketecture” produced by most manufacturers out there.

He has a midsize shop: somewhere between 50 to 100 machines, SAP, Exchange, and the rest of the usual suspect applications and platforms one would expect from a “current” IT shop. He has been planning his big move into VMware and, in his typical fashion, has done maniacal research on the hardware and the processes to do the job right for the right price. He is not one to cut corners. So, my cell rings…

"Why would I buy 8 core Intel processors over 12 core AMD processors?" I started rattling off the typical reasons along the lines of comparing the new cores to what he already has on the floor, potential performance differences between the cores themselves, recent market movements (good or bad) of the two companies, etc. The reality is that VMware posts the relative performance of most of these platforms, so the conversation was largely academic in the end, but a good exercise none the less.

But then I did the math. He was planning on buying 3 or 4 new machines to virtualize about 80% of his existing machines. Even if he bought two-way machines and three of them, that’s 72 cores! Since many of his machines are single processor, lower frequency, single core, computationally speaking, he potentially has more computational density in the 3 new machines (definitely if he goes with four new machines) than in the roughly 70 machines we are talking about virtualizing. While the RAM story may not be as straightforward, it seems like virtualization is now the only way to go to even remotely get your bang from your buck in the server space.

I appreciate the pessimists who will rattle off reasons such as application support from the software manufacturers, weird hardware requirements like license “dongles”, and… uh… OK, I can’t think of any other real reason. The reality is, the software companies are caving in every day to the masses demanding virtualized support.
So now onto the new challenges virtualization is bringing such as the required “sewer pipe” of network and storage connectivity—but that’s a conversation for another day.

Image courtesy of farleyj on Flickr via the Creative Commons

OK, dedupuplication technology is cool. It makes disk a viable target for longer term retention. Dedupe, however, is not the panacea of backup. I have been getting a lot of questions about deduplication from my customers and it's a fun topic to discuss. Being a professional pessimist, it's my job to play Debbie Downer at the dedupe party, though, and say "hey now, let’s not lose sight of the fundamentals, people."

1. Reporting – It’s almost not worth doing a backup if you can’t prove it happened, or more importantly, why it didn’t. I appreciate we are not all waiting for the SEC to kick in the doors and look for our latest backup reports to determine if you are going to jail or not. Some products such as CommVault’s Simpana have a very nice native reporting tool with an option to do some very cool statistical trending that makes decisions around necessary throughput and media not so dependent on the crystal ball. Even if you are not looking for a complete solution overhaul or have already taken the Data Domain jump or just plain happy with BackupExec, there are some tools that are remarkably functional at increasingly competitive prices such as EMC’s Data Protection Advisor that cover all the mainstream backup tools.

CommVault’s Data Growth Report from the SRM expanded reporting tools.

2. Integration with the platforms that drive your business – While all the big hitters are touting integration with the soon-to-be ousted VCB, some products have really stood out such as Symantec’s NetBackup, which allows a single backup to provide both machine and single file restore (video here). Also there are a number of VMware specific solutions that have introduced a flavor of dedupe into their technologies as well. Veeam is a standout here that is probably worth taking a look at, providing replication and backup with dedupe in a single, very cost attractive product. This product also lends well to where I see next generation data protection going. See my rant "Why Back Up Your Business Data?"

3. Speed – Depending on how you implement your backups, deduped or not, you may still be racing the sun to get your servers backed up before your users show up to change all the files again. Avamar is a clear dedupe stand out here since the method they utilize to perform the dedupe usually results in ridiculously shorter backups (see Justin’s blog "Data Domain vs. EMC Avamar: Which deduplication technology is better?"). However, being able to add media servers with a single management interface to increase the sheer brute force of your solution with or without deduplication will keep you with the old stand bys like Symantec’s NetBackup and CommVault’s Simpana.

Other criteria off the top of my head include:
1. Do you have application agents for MY applications?
2. Can you restart your backup / restore jobs from where they left off?
3. What is your bare metal solution?
4. Can you protect desktops with the same interface as the datacenter?
5. Do you have integrated archiving / compliance search?
6. How difficult is it to protect / recover the backup solution itself with history?
7. Can you multiplex / multi-stream backups for improved reporting?
8. Can I write to disk and tape at the same time?
9. How granular is your security construct?
10. Do you support my hardware with tools such as NDMP?
11. How well does your solution work with my firewall?
12. If I backup to disk, how do I cut tapes / restore from tapes?
13. Does your solution have CDP as well as regular backups?
14. Do I have to configure every server or does your solution leverage policies?
15. Does your solution manage encryption / encryption keys?
16. Does your solution push updates or do I manually update all the clients?
17. How functional is your GUI / Command Line interface what can / can’t I do from each?

This list could go on for another hundred points depending on the specific needs of your business, but I think we are now at the point when we have a deduplication conversation that extends back to what our dedupe vendor is bringing to the table beyond a compression ratio.

What do train tracks and data backup have in common? Do you happen to know why train tracks are 4 feet, 11 3/4 inches apart in England? Because they were built following the ruts left by Roman chariots.

Stop Backing Up!

I am not kidding, guys, just stop it.

One of my favorite questions to ask smart IT folks is why they backup data at all. After the unavoidable litany of doing file level restores, email restores, server restores, compliance requirements, etcetera, etcetera, we come to the real answer: the guy before me did it this way.

Who was this guy anyway? For all we know his retention scheme was based on the number of tapes that came free with the Travan tape drive he bought in the 80s. At least there is some logic to that answer.

In all seriousness, there are lots of good reasons to do backups, but far fewer IT shops have these requirements than we think. So before we spend a small fortune in deduplication, application agents, WAN acceleration, and tape infrastructures, let's figure out if what we are doing is really serving our objectives or just another (Roman) rut.

Objective 1: Restore a file / email—snapshots anyone?
If you really want to get a file back fast, utilize some host-based or (even better) array-based snapshot mechanism. In most cases, these are free and easy to implement and sure beat re-cataloging a bunch of tapes. Oversize your storage by enough to store 30 days worth of snapshots and you have met the requirements for 99% of the restores out there encountered by IT pros. In terms of email (specifically Exchange), leverage the tools already at your disposal, namely Deleted Item Retention. Not many users are using Shift+Delete in their day to day, so this covers this common restore handily. Restores outside of a 30 day window are extremely uncommon, but monthly and even annual snapshots are possible—these, though, should have a pretty good rationale before implementation.

Objective 2: Restore a server
An uncommon restore in the grand scheme of things—but definitely something we want to be prepared for—is bringing the whole server back (potentially offsite). I have yet to find a bare metal solution I really admire (reliable, inexpensive, easy to use, easy on storage, etc). My personal favorite solution here is leveraging some sort of virtualization, even if you only put one virtual machine on a physical server, along with some mainstream replication technology. The virtualization technology can be the free stuff for this to work, so all you are out is the replication software cost or array replication cost, which is generally in the same ballpark as backup software and some old server hardware that you would otherwise be using to hold up your installation manuals in the back room. This solution is far more effective toward the goal and much easier to test. Did I say test? Yeah, do it. Figuring out server level restores at 2AM sucks. In terms of retention, I have no use for a server image from a year ago. I am sure I can contrive a reason to be contrary but that’s just being obtuse. So keep an image, or a week’s worth, and move on.

Objective 3: Compliance
Okay, you got me. This one and its ridiculous retention requirements may drive us to tape purgatory, but at least I never really have to use them for operations. My compliance officer can hug the tapes at night, but I am not betting my job on a media that cannot be tested outside of an actual restore and is very susceptible to environmental fluctuations such as heat, dust, gravity, humidity, cold, air, time, quarks, palm sweat, the breath of the tape guy, etc. I would still like to have it in writing from someone who has to sign the check for this stuff (boss, lawyer, compliance officer, etc) as to exactly why they ask for the retentions they do. Again, maybe they got their guidance from the guy with the free tapes in the 80’s…

I love technology. There is nothing better than waking to the news of the next cool tool, gadget, or blinky thing. However, I have recently become a bit cynical of recent releases in the storage world. Many of these new “innovations” have been the product of the marketing teams more than the engineering teams at our storage manufacturers. Everyone out there is touting their storage “virtualization” and I have completely lost track of what that term actually means. So my apologies for the next few paragraphs which are a little more negative than usual.

Head trip: Middle managers for your storage TPS reports…

IBM, Hitachi and EMC have long had head appliances you can shove everything from ghetto RAID to ultra high end storage behind and present it all as a single storage pool from which to carve. In this case we are creating one big “virtual” array from lots of other smaller arrays. I suppose there was a time when the companies out there who needed Petabytes of storage may have wanted this to simplify their lives by having a single interface and had a few hundred thousand dollars burning a hole in their budget. I have never worked with that finance guy (rare as the Loch Ness Monster), but I would love to. These days even mid range arrays scale to just shy of a Petabyte and have a very healthy set of tools so it seems a shame to dumb them down to disk behind this “one ring to rule them all”.

Big Pool: Come on in, the storage is fine…

Here we are talking about putting all (or a big bunch) of your disks into one big RAID and slicing off storage as you need it. Wow, now that I just wrote that I have no idea what is new about this (sneaky marketers).  VMware is even doing this in software. I guess what has changed is the messaging from the storage vendors saying that all your storage is ideal for all of this. I get that more spindles thrown at an IO problem is great, but I don’t know if I want to comingle my critical Oracle data on the same spindles as my summer intern’s home drive who may very well upload his classic Ren and Stimpy collection to the public share during month end cube runs. Call me crazy. I guess if you have so many disks that this does not affect your performance why not, but again this is the classic mark of the wayward finance guy. Fencing off a little IO for my critical apps still seems like a good idea unless you have some Quality of Service tools in the array to prevent dropping business critical apps for business social tasks. These QoS tools are becoming more available all the time from the likes of EMC. Net of net, I would be wary of a salesperson or their engineer calling this strategy a technology. The worst part is that they are selling it as something to make the IT person’s life easier. I am a lazy as the next guy but I don’t know if I am going to risk my performance (and maybe my job) to save myself one more trip down a wizard to create a RAID before I carve my LUN.

Performance Pyramid: Step on up, for now…

Now we get into what I would really call a new technology. Here we get into moving away from your classic contiguous LUN idea and move to a LUN actually being an amalgamation of stripes of data from several different RAIDs and potentially from RAIDs of different performance bands. Pillar tried pulling this off in the array by moving data that needed more performance to the edge of the platters. This, in a lab, really would result in better performance. That said, the array still puts data on the inner radials so the actuator arm is still working very hard and wanders away from your performance sensitive data. It was a great concept but my experience what that this was a very “gen 1” way of going about it. Generation two has the LUN moving between multiple kinds of disks based on its temporal needs. If it does not need it to be fast, the LUN moves down to SATA, if the LUN needs heat, it moves up to 15K SAS/ Fibre Channel drives until it qualifies to move back down. Here the algorithm on what moves when is key. If the window of decision is too big, by the time the array decides the data needs more performance, the job may already be done requiring the performance. If the window is too small, minor blips from an application could result in hundreds of gigs of data flying around the array, wasting precious IOPS.  Here is where the policy engine will rule and I expect these engines to get very complex very quickly but the resulting savings from potentially overbuilding arrays for end of month style processes will be worth it.

Performance Soup: Pay no attention to the man behind the curtain…

So now that we have awesome engines making really good calls on when the data can move up and down based on educated empirical data lets reduce the LUN to smaller components and move the busy pieces all the way to flash so I don’t use any more of the prime real estate than necessary to get the job done. Most of the LUN may actually be on SATA, and still other pieces may sit on Fibre Channel or SAS. This is the end game until solid state drives are so cheap we all just buy as much as we need for capacity and walk away. This small segment migration is happening now. The array that will win the day will be the one with the smartest (not fastest) head since it will be up to the rule base to make the magic happen.

So the good news is that there are some really applicable technologies out there changing the face of storage as we know it. Just don’t let the marketing teams fool you into mistaking old school lazy for next gen cool.