David's Discussions

Sometimes, though rare, you do get something for free. VMware has had API’s built within the code since the beginning, and while some of the earlier features, such as VCB (VMware Consolidated Backup), were a little rough around the edges, a new set of APIs are due out in vSphere 4.1 that are really going to impact the performance and scalability of your virtual infrastructure.

I will be the first to admit that I tend to see things from the storage side of the equation, so this latest news is particularly interesting to me. But anyone into squeezing out the best bang for their virtualized hardware investment should be pretty jazzed about this. These latest APIs are targeted specifically to how VMware can leverage a “smart” storage array to make virtual guests go even faster on existing hardware. The new “family” of APIs are called the vStorage APIs for Array Integration (VAAI). This is to differentiate them from existing APIs such as the ones for data protection, multipathing, and Site Recovery Manager.

I have long said that we in IT don’t fix issues, we push them around, and this is exactly what these three new APIs do. More specifically, they take tasks that your server hardware is doing and move those tasks to the storage array hardware. This has two major benefits: The first is that the server resources such as CPU and RAM can now serve the tasks specific to the virtual machines rather than the “administrative” work of the underlying VMFS care and feeding. Secondly, these tasks take up considerable network traffic (IP or FC depending on your storage array networks flavor) between an array and the server infrastructure; so again, more of the network’s resources actually go to serving the needs of the business applications and less to the underpinnings of vSphere.

There are a few caveats (I know you saw it coming). It’s all about the block level access, raw device mappings and VMFS for now, so if you were one of the early NFS adopters you are going to have to sit on the bench for a while yet. Secondly, you will have to be using an array that supports the APIs (kinda obvious). The good news is that while EMC will be the first kid on the block with the new toys, it is based on standard SCSI commands, so other manufacturers should not be too far behind.

OK, now onto the goodies…

First up is the hardware accelerated locking. One of the features that makes VMware the data center tool it is, revolves around the ability for multiple physical systems to work together in a cluster. Since all the machines see all the guest files at the same time, file locking is a big deal. If any two servers try to write to the same file at the same time, well, bad things happen. This locking process takes commands before, during, and after an actual update. When you have many machines performing these updates, this amounts to millions of commands. The new API reduces a large number of lock commands to a single SCSI command. While this will have some performance impact, the reduced instruction set will allow VMware clusters to become much larger due to reduced effort to arbitrate all this locking.

The next new feature is called hardware accelerated zero. VMware zeros out blocks inside a virtual machine’s file when it expands. This means that as you add data to a virtual machine there are often two or more writes necessary to actually write your data to the file. This is a huge overhead. With the new API, the host only needs to tell the array how much space to zero out, and the array performs the task rather than the host. This can reduce the IO overhead from 2 to 10 fold. Data writes are already expensive in terms of parity calculation, so this will be a huge improvement in overall array write performance.

The final, and perhaps most obvious, tool introduced is hardware accelerated copy. Here, instead of the VMware hosts moving the files for storage vMotion and the creation of machines from VM templates, the copy request is given to the array which simply moves the data internally. While not an everyday occurrence in most shops, the savings in network, array, and server resources are huge.

Personally, the most amazing thing about these new tools is that (assuming your array supports them) you will be getting them for free when vSphere 4.1 comes out later this year. I guess we can chalk it up to our maintenance dollars at work deep in the confines of VMware. So keep those maintenance contracts up to date—the upgrade is going to be worth it!

OK. I admit it. Some of the best ideas (and epiphanies) I bring to my customers don’t come from me, but instead come from other customers. Case in point, I was recently having a conversation with a long time customer of mine who has, over the years, become a good friend, so we routinely bounce ideas off one another to make sure our math works out and we are not falling prey to the continual “marketecture” produced by most manufacturers out there.

He has a midsize shop: somewhere between 50 to 100 machines, SAP, Exchange, and the rest of the usual suspect applications and platforms one would expect from a “current” IT shop. He has been planning his big move into VMware and, in his typical fashion, has done maniacal research on the hardware and the processes to do the job right for the right price. He is not one to cut corners. So, my cell rings…

"Why would I buy 8 core Intel processors over 12 core AMD processors?" I started rattling off the typical reasons along the lines of comparing the new cores to what he already has on the floor, potential performance differences between the cores themselves, recent market movements (good or bad) of the two companies, etc. The reality is that VMware posts the relative performance of most of these platforms, so the conversation was largely academic in the end, but a good exercise none the less.

But then I did the math. He was planning on buying 3 or 4 new machines to virtualize about 80% of his existing machines. Even if he bought two-way machines and three of them, that’s 72 cores! Since many of his machines are single processor, lower frequency, single core, computationally speaking, he potentially has more computational density in the 3 new machines (definitely if he goes with four new machines) than in the roughly 70 machines we are talking about virtualizing. While the RAM story may not be as straightforward, it seems like virtualization is now the only way to go to even remotely get your bang from your buck in the server space.

I appreciate the pessimists who will rattle off reasons such as application support from the software manufacturers, weird hardware requirements like license “dongles”, and… uh… OK, I can’t think of any other real reason. The reality is, the software companies are caving in every day to the masses demanding virtualized support.
So now onto the new challenges virtualization is bringing such as the required “sewer pipe” of network and storage connectivity—but that’s a conversation for another day.

Image courtesy of farleyj on Flickr via the Creative Commons

OK, dedupuplication technology is cool. It makes disk a viable target for longer term retention. Dedupe, however, is not the panacea of backup. I have been getting a lot of questions about deduplication from my customers and it's a fun topic to discuss. Being a professional pessimist, it's my job to play Debbie Downer at the dedupe party, though, and say "hey now, let’s not lose sight of the fundamentals, people."

1. Reporting – It’s almost not worth doing a backup if you can’t prove it happened, or more importantly, why it didn’t. I appreciate we are not all waiting for the SEC to kick in the doors and look for our latest backup reports to determine if you are going to jail or not. Some products such as CommVault’s Simpana have a very nice native reporting tool with an option to do some very cool statistical trending that makes decisions around necessary throughput and media not so dependent on the crystal ball. Even if you are not looking for a complete solution overhaul or have already taken the Data Domain jump or just plain happy with BackupExec, there are some tools that are remarkably functional at increasingly competitive prices such as EMC’s Data Protection Advisor that cover all the mainstream backup tools.

CommVault’s Data Growth Report from the SRM expanded reporting tools.

2. Integration with the platforms that drive your business – While all the big hitters are touting integration with the soon-to-be ousted VCB, some products have really stood out such as Symantec’s NetBackup, which allows a single backup to provide both machine and single file restore (video here). Also there are a number of VMware specific solutions that have introduced a flavor of dedupe into their technologies as well. Veeam is a standout here that is probably worth taking a look at, providing replication and backup with dedupe in a single, very cost attractive product. This product also lends well to where I see next generation data protection going. See my rant "Why Back Up Your Business Data?"

3. Speed – Depending on how you implement your backups, deduped or not, you may still be racing the sun to get your servers backed up before your users show up to change all the files again. Avamar is a clear dedupe stand out here since the method they utilize to perform the dedupe usually results in ridiculously shorter backups (see Justin’s blog "Data Domain vs. EMC Avamar: Which deduplication technology is better?"). However, being able to add media servers with a single management interface to increase the sheer brute force of your solution with or without deduplication will keep you with the old stand bys like Symantec’s NetBackup and CommVault’s Simpana.

Other criteria off the top of my head include:
1. Do you have application agents for MY applications?
2. Can you restart your backup / restore jobs from where they left off?
3. What is your bare metal solution?
4. Can you protect desktops with the same interface as the datacenter?
5. Do you have integrated archiving / compliance search?
6. How difficult is it to protect / recover the backup solution itself with history?
7. Can you multiplex / multi-stream backups for improved reporting?
8. Can I write to disk and tape at the same time?
9. How granular is your security construct?
10. Do you support my hardware with tools such as NDMP?
11. How well does your solution work with my firewall?
12. If I backup to disk, how do I cut tapes / restore from tapes?
13. Does your solution have CDP as well as regular backups?
14. Do I have to configure every server or does your solution leverage policies?
15. Does your solution manage encryption / encryption keys?
16. Does your solution push updates or do I manually update all the clients?
17. How functional is your GUI / Command Line interface what can / can’t I do from each?

This list could go on for another hundred points depending on the specific needs of your business, but I think we are now at the point when we have a deduplication conversation that extends back to what our dedupe vendor is bringing to the table beyond a compression ratio.