Integrated Data Storage Blogs

vSphere 4.1 adds over 150 new features and improvements, many that are in preparation for the the upcoming release of VMware's Cloud offering. VMwware continues to be the innovator and isn't resting on it success in the market. I am very excited about the upcoming VMworld in San Francisco because I am anticipating many big announcement. I hope to be able to provide a blog or two from my sessions out there. Out of the loads of features in 4.1 and the numerous performance enhancements I wanted to highlight a few.

Storage I/O Control:
To me this couldn't come soon enough. I/O Storage control is DRS for Storage, it is a cluster-wide I/O scheduler that works to throttle I/O to ensure that a single VM cannot monopolize a single datastore’s capabilities. Storage I/O control is at the VMDK level for the Storage I/O throttling functionality to be activated, there needs to be some congestion on the datastore for sometime. The default settings require a sustained ~30ms response time before the code is executed.

This is enabled per datastore, and will kick in when latencies are 30 ms or greater by default, though you can adjust the latencies as you see fit via advanced settings. This feature is configured through VMware vCenter, but the ESX servers communicate between themselves via shared headers on each datastore. This preserves the ability for the cluster to continue to operate when the management interfaces are inoperative.

Network I/O Control:
Traditionally I haven't seen much need for Network QOS but with the decreasing cost of 10Gb techonology and also the number of high end applications that can now be virtualized I am glad that VMware added Network I/O control to the Resource control functionality.

There are six types of traffic that are identified:

* VM
* Management
* NFS
* iSCSI
* vMotion
* FT Logging

We now have the ability to apply limits to these types of traffic, to absolutely cap the amount of bandwidth being consumed by each host within a cluster, we also have the ability to put shares that are tied to the different types of traffic that will provide priority for all outbound network traffic.

Another new feature in 4.1 is Load-Based Teaming which is useful for the 1Gb connections. What Load Balanced Teaming does is change the physical NIC being used for a type of traffic when ESX detects that a link is more than 75% utilized over 30 seconds. You can only use these technologies for the v Distributed Switch that is incorporated into the Enterprise Plus version of vSphere.

Memory Compression:
We have been over commiting memory for a long time now, in vSphere 4 began to help us out in a few ways when you got a little too crazy with your memory over commit. First, it used transparent page sharing to try to deduplicate RAM. Second, it used the VMware Tools balloon driver to reduce the amount of RAM actively consumed by a VM. Third, the pages were sent to the physical disk.

Paging to disk is a slow and painful operation, what VMware did was they added another mechanism to stave off the need to page to disk: memory compression. ESX will take a configurable amount of memory and compress it if it needs to. We really do not want to get in this situation but we have the tools to give by us the time to add hosts or memory

vStorage API for Array Integration (VAAI):
David went into this feature in his VMware And Its Storage - Faster, Smarter, Stronger... Free? blog

Continuing this trend among us naive open systems types, VMware and a variety of storage partners are working to enable offloading of storage operations to the array. Initially there will be three operations handled by the arrays: full copy, zero-out, and locking. For operations like cloning this means that ESX won’t copy a template out over the SAN or network just to put it right back on the array.

Support for this integration on EMC CLARiiON arrays is due out with the release of Flare 30, I have not heard on release dates from other vendors such as Dell, NetApp, IBM, Hitachi, & HP. This feature is only available to Enterprise and Enterprise Plus licensees.

ESXi:
It is clear the ESXi is the future for vSphere has the Hypervisor. vSphere 4.1 is the last major release to support the classic ESX software with the service console. Future releases will be ESXi only, using APIs to control and configure the host.

VMware has added boot-from-SAN capabilities(Cisco UCS anyone?), scripted installs, enhanced Update Manager to push drivers and other modules, added built-in Active Directory support, and now fully support both local and remote Tech Support Mode.

I know that this is going to freak a lot of people out but I would get used to it because this is the way were are going. I would highly considering installing the ESXi version with this upgrade. I highly recommend checking out the ESX to ESXi upgrade center.

Good bye Service Console you will be missed.

Increased Cluster Limits:
High Availability has had its limits increased, to 32 hosts, 320 VMs per host, and 3000 VMs per cluster.

High Availability has application awareness using APIs. This lets monitoring agents work with HA to do a variety of things, including a full guest restart. This works through the VMware Tools and involves guest to host communication, which may be a security concern in some cases.

vMotion Enhancements
Some tweaks under the hood have been made to improve vMotion performance. Also the limits for number of concurrent vMotions have been increased:

* 1Gbps NICs = 4 Concurrent vMotions
* 10Gbps NICs = 8 Concurrent vMotions
* Datastore (both VMFS and NFS) = 128 Concurrent vMotio

Fault Tolerance (FT):
Not my favorite technology in the fleet, until we see multiprocessor support to the Virtual Machine I will not be on the bandwagon. I was hoping to see SMP support in this release. However, a lot of other restrictions have been lifted.

vCenter:
The biggest change is that it is 64-bit only, but for all you View users you will have to wait until the release of 4.5 since composer is not compatible with x64.

Well I hope I have some exciting news coming from VMworld this year so stay tuned.

Sometimes, though rare, you do get something for free. VMware has had API’s built within the code since the beginning, and while some of the earlier features, such as VCB (VMware Consolidated Backup), were a little rough around the edges, a new set of APIs are due out in vSphere 4.1 that are really going to impact the performance and scalability of your virtual infrastructure.

I will be the first to admit that I tend to see things from the storage side of the equation, so this latest news is particularly interesting to me. But anyone into squeezing out the best bang for their virtualized hardware investment should be pretty jazzed about this. These latest APIs are targeted specifically to how VMware can leverage a “smart” storage array to make virtual guests go even faster on existing hardware. The new “family” of APIs are called the vStorage APIs for Array Integration (VAAI). This is to differentiate them from existing APIs such as the ones for data protection, multipathing, and Site Recovery Manager.

I have long said that we in IT don’t fix issues, we push them around, and this is exactly what these three new APIs do. More specifically, they take tasks that your server hardware is doing and move those tasks to the storage array hardware. This has two major benefits: The first is that the server resources such as CPU and RAM can now serve the tasks specific to the virtual machines rather than the “administrative” work of the underlying VMFS care and feeding. Secondly, these tasks take up considerable network traffic (IP or FC depending on your storage array networks flavor) between an array and the server infrastructure; so again, more of the network’s resources actually go to serving the needs of the business applications and less to the underpinnings of vSphere.

There are a few caveats (I know you saw it coming). It’s all about the block level access, raw device mappings and VMFS for now, so if you were one of the early NFS adopters you are going to have to sit on the bench for a while yet. Secondly, you will have to be using an array that supports the APIs (kinda obvious). The good news is that while EMC will be the first kid on the block with the new toys, it is based on standard SCSI commands, so other manufacturers should not be too far behind.

OK, now onto the goodies…

First up is the hardware accelerated locking. One of the features that makes VMware the data center tool it is, revolves around the ability for multiple physical systems to work together in a cluster. Since all the machines see all the guest files at the same time, file locking is a big deal. If any two servers try to write to the same file at the same time, well, bad things happen. This locking process takes commands before, during, and after an actual update. When you have many machines performing these updates, this amounts to millions of commands. The new API reduces a large number of lock commands to a single SCSI command. While this will have some performance impact, the reduced instruction set will allow VMware clusters to become much larger due to reduced effort to arbitrate all this locking.

The next new feature is called hardware accelerated zero. VMware zeros out blocks inside a virtual machine’s file when it expands. This means that as you add data to a virtual machine there are often two or more writes necessary to actually write your data to the file. This is a huge overhead. With the new API, the host only needs to tell the array how much space to zero out, and the array performs the task rather than the host. This can reduce the IO overhead from 2 to 10 fold. Data writes are already expensive in terms of parity calculation, so this will be a huge improvement in overall array write performance.

The final, and perhaps most obvious, tool introduced is hardware accelerated copy. Here, instead of the VMware hosts moving the files for storage vMotion and the creation of machines from VM templates, the copy request is given to the array which simply moves the data internally. While not an everyday occurrence in most shops, the savings in network, array, and server resources are huge.

Personally, the most amazing thing about these new tools is that (assuming your array supports them) you will be getting them for free when vSphere 4.1 comes out later this year. I guess we can chalk it up to our maintenance dollars at work deep in the confines of VMware. So keep those maintenance contracts up to date—the upgrade is going to be worth it!

My last blog post I rambled on about the Private Cloud and how I think it can be beneficial to IT. When I start thinking about Public Clouds, though, my forecast doesn't look so positive.

I will start by saying all applications and organizations are not equal. I think there are many applications that are a good fit for Cloud applications. If you look at email, it has been in the cloud since almost the beginning of the Internet. I think for many IT organizations it does make a lot of sense to put email in the cloud. If you have ever had to manage an email infrastructure, you have to have dedicated hardware, licensing, spam filters, admins, etc. dedicated to email. The operational and infrastructure costs can cost IT a ton. There is a definite argument that you can reduce costs with the offerings Google, Microsoft Exchange, or VMware's Zimbra.

Okay, that was my pro Cloud pitch. Here is the not-so-positive spin on the Cloud and why I am not on the Public Cloud bandwagon.

The only one you can trust is yourself
The biggest concern/question that I have is when we starting putting secure information that we once had within our own corporate walls into the Cloud is, Do we gain or lose security? It is just like the feeling I got when I hired my son's first babysitter: I know I trusted this person, but do I really trust this person? And are they going to take care of him like I would? With the public cloud we just don't know what we are getting into. We do not KNOW what the methods are that are taken, we just know what we are told.

IT as Utility
This is the conversation that I have had with many of my customers: We just put our applications and data into the cloud, and we've just turned our IT department into Comcast Cable. What happens when the $99 a month triple play promo runs out and we are now paying $180 a month? We now do not know what our early IT utility bill is going to be and as businesses grow so does the IT utility bill. Also, what about customer service? Nobody has the urgency for our applications and data as much as the people who own it—which is a good segway for my final point.

Coming down from the Clouds
The one issue that I see and have discussed with other colleagues in the industry is I am not happy with my current cloud provider and I want to move my data from one cloud provider to another or even back into private infrastructure. How do I move my data? Can I move my data? What will it cost me? How long will it take? The answer is we just don't know yet. VMware is going to be making a push in the second half of this year to be the standard platform for both the Private and Public Clouds, which will help some. But what about the other Cloud providers?

So, yes, I am a bit of a Negative Nancy on the Public Cloud, admittedly not one to jump on the latest buzz or technology that jumps in the game. I tend to be a late adopter of technology and my biggest fear with the Public Cloud is the recreation of the Dot Com Fallout. If we remember back in the late 90's with the explosion of hosting providers and ASPs, how many of those companies are around today? Are we recreating history?

It appears I'm not the only one who thinks all this talk of 'clouds' is just a lot of hot air. Recently, Amazon and the VCE team have been going back and forth over the use of the term 'private cloud', and I find the comments from both sides interesting.

You can read about Amazon's shot across the bow here:

http://www.itnews.com.au/News/175953,video-time-to-kill-the-private-cloud.aspx

And the VCE team's response here:

http://www.crn.com.au/News/213820,vce-execs-defend-private-cloud-at-emc-inform.aspx

To summarize things, Amazon says that the idea of 'private cloud' ignores critical concepts like pay as you go, capacity on demand, and the idea that cloud is all about saving costs (since you still have to lay out capital to own the hardware). Their CTO argues that cloud is not about technology, it is about th advantages it brings. He also says that private cloud is a lot of marketing fluff (my assessment of his quote). Meanwhile, the VCE crowd (if you don't know, VCE stands for VMware, Cisco, and EMC) is beating the drum of revolution, and positioning the 'public cloud' as a consumer tool. It is 'hallucinatory' (a pretty interesting choice of words, IMHO) to think that enterprises who have invested in infrastructure will just throw it all out and dump their applications onto Amazon's cloud offering. To me, the truth of the matter is they're both right.

As I've said before, I think the talk about cloud is just that -- talk. No one implements a cloud. Clouds don't have any structure, any solid foundation, and reliability -- they go where the wind blows them. The metaphor doesn't just limp, it's lame. This concept shouldn't be about specific technologies, to Amazon's point, but it does require some fundamental things to work right, like virtualization (in whatever form it might take). I find it interesting that the VCE crowd points to hardware/software installs to demonstrate that the private cloud does indeed exist. It points out that though their marketing engine is in full swing, they don't understand the fundamental value of the combined technologies. The proof of private cloud isn't in how many UCS servers are deployed or how many licenses of VMware exist, guys, it's how many customers you have who adopted the combined force of your technologies and completely virtualized everything, creating a 100% uptime, flexible, constantly flexing environment. Talk to us about THAT.

To defend the VCE guys, though, they are approaching this "revolution" (another way over-used term) in a sensible way; as is quoted in the article, this is about incremental change. You don't just hear about something like this and throw everything you've built away, turning over application reliability, uptime, and security to an Internet company that can't always keep their own store online. Not if you like keeping your job, anyway.

This is about making network, compute, and storage a utility. Make the technology simple and easy to scale so you can work on the applications and the business metrics. Stop having to worry about how to design a server or your network, or your storage for a new application and just provide it resource units. That's got nothing to do with a cloud, but it has everything to do with building out a great architecture, and until Amazon and the other players can prove they can, over the Internet, provide the same level of security, performance, reliability and uptime as gear in my datacenter, they don't win the battle.

With litigation on the rise and enterprise information stores overflowing with unstructured content, many large and mid-sized companies are slashing legal costs by reclaiming data search responsibilities related to E-Discovery and Early Case Assessment. Such tasks, specifically those that seek to accurately identify, collect, and then preserve Electronically Stored Information (ESI) pertaining to active or anticipated lawsuits, have traditionally been handled by the same external law firm engaged to argue or arbitrate the legal matter in question. However, as challenges tied to system diversity and runaway data growth have driven search costs through the roof, an entire industry of specialized E-Discovery technologies and methodologies has matured. The rapidly expanding percentage of mildly-litigious organizations investing in these tools is proof that, by in-sourcing legal search tasks, E-Discovery and Early Case Assessment costs can be controlled, and in most cases, dramatically reduced.

In addition, early adopters have proven that cost-reduction is not the only reason to look inward – some of the same tools and approaches also excel at once-insurmountable data management challenges linked to compliance and retention drivers. Best of all, the capital investment required to procure and install the necessary technology seems to be coming straight out of the legal budget, which stands to see future reductions in operational expenses. So, when the right tool is selected, both IT and Legal can post simultaneous wins. Although, as is the case with most technical investments, the key to this dual victory lies in product selection - for this reason, IT departments should determine what E-Discovery and Early Case Assessment is actually costing their companies, and then, if warranted, lead the charge to bring these tasks in-house.

In the legal sense, the Discovery process occurs during the pre-trial phase of a lawsuit and provides a means by which parties can request documents and evidence from each other. For most of the history of our legal system, the process has been centered on paper records and other physical items. However, as the world changes, so do the Federal Rules of Civil Procedure, which, in 2006, were updated with specific provisions to support the preservation and collection of electronic data including email messages, Microsoft Office documents, and Instant Messaging (IM) exchanges. In short, anything electronic is now completely Discoverable and organizations are required to produce all data even remotely related to concepts or persons described within a legal complaint as part of Discovery. The process of identifying, retrieving and producing relevant electronic data related to Discovery is called Electronic-Discovery, or E-Discovery.

The practice of estimating the chances of successfully prosecuting or defending a legal matter through internally-focused Discovery efforts is called Early Case Assessment. Knowing that approximately 90% of all cases are settled before trial, Legal departments often perform Early Case Assessment in order to set a monetary break-even point for settlement. Throughout this process, while searching through their own electronic and non-electronic records, organizations keep an eye out for overtly incriminating evidence (called smoking guns in the legal world),to which opposing council will eventually be privy. With E-Discovery being a large and growing component of Discovery, it is a major component and input to the Early Case Assessment process.

Whether needed for pre-trial analysis or Early Case Assessment, getting data related to a specific person, issue or lawsuit (called responsive data in the legal world) out of source systems, document repositories and offline media can be time-consuming and technically challenging due to factors related to the way that we store and utilize information in business environments. While online search engines propagate the perception that keyword search technology is accurate, fast and easy to use, on most corporate networks, this is simply not the case. In the enterprise, electronic evidence can be scattered across multiple backup tapes, document archives, messaging hosts, and file shares. The distributed nature of our corporate networks further complicates this challenge – various versions of the same document are often stored on different workstations, laptops, removable drives and even cell phones. Email and IM conversations can sprawl out into many hundreds of related threads…solving the riddle of “who” said “what” to “whom” and “when” can quickly become a science!

For this reason, primitive E-Discovery and Early Case Assessment search efforts tend to gravitate toward a common pattern: initial collections performed by those familiar with technology (the “technologists”) are often done with a strong liberal bias, so as to not discount any potentially responsive data. This population is then reviewed manually, by those familiar with the details of the case (paralegals and attorneys), who read each document, organize documents according to relevancy, black line unrelated data within responsive documents (called redaction in the legal world), and, in general, gradually reduce the amount of data that will be used by lawyers to argue the case.

Click here to a view a larger version of the image

Combined with the compressed timelines associated with most legal actions, the largely manual nature of this work generally demands staff augmentation. Thus, the second group is almost always comprised mainly of consultants and contractors that charge by the hour (or by the page reviewed). As a result, related expenses fall in direct proportion to the amount of data gathered by the “technologists”. As it has become commonplace for the second group to be supplied by the external legal firm representing the organization in question, the exact cost of this effort may not be reported as an explicit line item. Regardless, the monetary costs of E-Discovery are traditionally the budgetary responsibility of Legal departments, which have, until just recently, benefitted from deep pockets and were rarely challenged on decisions to spend. Graphically, the relationship between data volume, relevancy and cost is represented as illustrated below:

Click here to view a larger version of the image

To put some numbers around the concept, the technology research firm Gartner says that outsourced data search and processing costs add up to $16,000 - $18,000 per GB. Considering that mid-sized cases typically require an inspection of over 100GB of data, it’s easy to see why even mildly-litigious companies should be interested in identifying and implementing alternatives to traditional E-Discovery approaches. To those in the know, the hundreds of thousands (or millions) of outsourced search and process dollars being spent by busy Legal departments comprise some of the lowest-hanging expense-reducing fruit imaginable!

Besides the benefits that come with being viewed as cost-reducing heroes, IT Departments that lead the charge to bring E-Discovery and Early Case Assessment tasks in-house can win access to cutting-edge data search and classification technology…on the Legal Department’s dime! The key to reaping these benefits lies in not being late to the party. If the project is inevitably going to happen, IT should be the champion - by initiating the conversation, and then driving the product selection process, IT can avoid being asked to support a system that was selected based solely on criteria linked to marketing and sales prowess. Conversely, proactive IT departments that take a hold of the E-Discovery steering wheel can be sure that ancillary compliance and retention requirements will be satisfied in lockstep.

If this sounds interesting, read my next post. There, I’ll explain what it takes to bring E-Discovery and Early Case Assessment search tasks in-house. After that, look for posts focused on the underlying business case as well as several that outline the ancillary features of select E-Discovery products.