Charles Makes a Point (or Two)

Of how CIOs fail.

DISCLAIMER:  This is a little bit of a rant b/c I'm more than a little peeved I STILL can't get my preorder of the iPhone 4 through either Apple or AT&T for delivery.  Standing in line for a full day at an Apple store next week (or whenever they can get stock now) is simply not an option.  I'm not THAT much of a gadget geek. 

I don't meant to be harsh, but this was a colossal failure on the part of both AT&T and Apple's IT planning, change control, capacity planning and incident response teams, IMHO.  Since both companies obviously have separate teams to accomplish these tasks, I have to place responsibility at their CIOs feet.  

If the stories going around the 'Net are accurate, basically neither organization was prepared for the onslaught that fell on them on the 15th.  Both Apple and AT&Ts web sites had minor issues early in the day, but the sites themselves stayed up (at least for me) all day long.  What never worked (again, at least for me, and reportedly for a WHOLE LOT of other people) was the promised ability to order an iPhone 4 for delivery on June 24 to my home.  That was the promise, and so that was the expectation set.  It wouldn't work from the Apple store or the AT&T store.  Why not?

From what I can gather from the reports, the issue wasn't load on the individual sites.  While there were a few issues throughout the day getting to the sites, those issues seemed to be very short lived.  The issue was the integration between the sites and all the people accessing the back end systems.  Since Apple won't allow a US customer to buy an iPhone without it being tied to an AT&T contract, we have to interface back into AT&Ts systems to determine eligibility, which in turn dictates pricing, yadda, yadda, yadda, you put a phone in the cart, agree to extend your indentured servitude to AT&T for 2 more years, bada bing, bada boom, you get a shiny new iPhone 4 on June 24.  At least that's what was promised by Steve Jobs at the WWDC, and on every banner ad AT&T and Apple could find to buy between WWDC and June 15.

The first thing that comes to mind while Apple is busy touting how wonderful they are for selling 600,000 iPhones (apparently NOT through the AT&T web site), I'm left wondering if the CIOs had mandated some true load testing and planning be done between their systems, how many more they would have sold (I know at least two would have been sold for my wife and myself for sure!).   Could it have been the first device to sell a million units on launch day?  Who knows.  We're in the Tootsie Pop commercial, my friends -- the world may never know because IT just makes the servers run for the business, they aren't part of the business.

The next thing that comes to mind is "how in the world does this happen"?  Could these IT departments be ANY MORE disconnected from either of their businesses?  This has been only the single most over-hyped and talked about device in the last decade, from the Gizmodo unveiling to the heavy-handed revenge tactics Apple used, to the exceedingly hyperbolic descriptions of how the phone is going to apparently change everything (I'm pretty sure there'll be an app for cleaning up the Gulf oil spill soon, it's all dependent on that A4 chip, so it's not backwards compatible with the 3Gs or 3G, sorry) to everyone knowing that this is the only device which AT&T is not allowed to cripple with their own branding and application placement (take a look at this Information Week piece to understand what I'm talking about if you don't already).  How do these guys miss this?  I mean, it's not like they've never been through an Apple device launch before.   They don't have any excuses here!  To say that they couldn't have predicted this is naive and untrue.  You spend marketing dollars for a reason, and given the track record of product launches for Apple in recent years, this was completely predictable.  Obviously not a specific number, but it would not have surprised me at all for them to have sold 1 million units -- if the systems had allowed it.  

I've seen some arguments made in defense of AT&T that you can't build out your systems to address theoretical unlimited transactions.  Really?  If you want to turn around the perception that you're a completely inept partner to the "stellar" Apple you do!  It's called capacity planning.  It is not an impossible task to have made some very intelligent predictions on what was going to happen here.  They should have looked at past device launches and seen that 1 million units was not out of the realm of possibility, especially given the (relatively) lackluster response the 3Gs got.  There are a lot of 2G and 3G users out there who have been waiting for this device.  There are plenty of users who have resisted coming over who are now because some of the shortcomings of earlier generations are being addressed.  How do you NOT predict this?  And if you overshoot on your capacity planning, and build out your systems to provide a stellar experience to 1 million preorders but still only do 600,000, guess what?  YOU HAVE 600,000 HAPPY CUSTOMERS!!!!  Instead, you have a muted response from those who were able to get the phone ordered, because even for them it was a pain in the backside, and you have countless others who are questioning whether they want to do business with you.  What a no-brainer.

Of course, then I'm scratching my head over the apparent lack of intelligent change control.  What in the world was the AT&T CIO thinking when it was (reportedly) allowed, the weekend before a massive, exclusive device launch, for the core customer database to undergo significant changes that had not been tested under real load?  I mean, really, didn't anyone besides the WHOLE WORLD know this device launch was coming?  Weren't vacations cancelled at AT&T so all hands could be on deck?  Seriously?  Does anyone believe in change control anymore?   In my IT past, we went into change lockdown before new launches, all end of year events, and just about any event that would drive significant orders until the event was past.  it's not a difficult concept, and it's curious why it isn't followed at AT&T, and why Apple wouldn't demand something like that from its only partner in the iPhone business so neither of them got black eyes.

This is precisely why CIOs shouldn't be providers to the business, they should be PART OF the business.  They should be sitting in these discussions making sure these things are thought of, taking a global view of these various integration points, and speaking up when there are issues.  I am assuming, of course, that they didn't do any of that, and if they didn't, they should probably be fired.  If they did, and were ignored, then they should probably quit, because they clearly aren't seen as a valuable and strategic member of the management team.   

To keep it all in perspective, it is just a stupid gadget.  But this is such a clear case of IT not serving the business well it screams for a detailed case study of what not to do if you want to serve your customers well.   

The hype machine is in full swing regarding cloud based computing, and I'm beginning to think that this juggernaut of marketing and confusion may be the biggest barrier of them all to wide-spread cloud adoption.

I said this a few posts ago, and I've had a lot of conversations about this with various people.  Cloud isn't new.  It's a repackaging of failed or quasi-failed ventures of the past.  Remember how SaaS was going to change the world and do away with the need for IT departments to manage all these complex applications and databases, and we were just going to be able to use a web browser to get at whatever application we needed to use?  I'm sure you're all beacons of productivity and your IT departments have shrunk to one person because of your heavy SaaS adoption, right?

If you'll recall, Web 2.0 (another of those meaningless marketing terms that never should have been printed) was going to change the world with its ability to ubiquitously provide seamless connectivity from any platform to any Web 2.0 'enabled' application (whatever that meant).  I bet you're all just connecting to all your apps from your cell phones and seamlessly using the same interface from desktop to laptop to phone to kiosk to <insert other technology I can't think of right now> and are just so stinking productive, right?

Remember a few years further back how Managed Services were going to bring all of our data centers into these massively efficient multi-tenant datacenters with millions of machines and petabytes of data storage in them, effectively returning all that datacenter space back to the business to put revenue generating bodies in and eliminate IT?  Yeah, never really happened.

The major barrier here for cloud computing is that it isn't new, and people are really starting to catch on to that fact.  Don't get me wrong, the adoption of virtualization technology is a paradigm shifting event for most datacenters.  I'm very pro-virtualization, and think that complete adoption of VMware should be every CIOs goal.  But trying to put a buzz on virtualization of 'private cloud' has gotten old very quickly, almost to the point of being meaningless.  You're not creating a cloud, you're creating a highly dynamic foundation of infrastructure.  Try building a house on a cloud foundation and see how far you get.  

I personally refuse to talk about VMware in the context of a cloud, as I think most people are sick and tired of hearing about it that way. 'Cloud' has become such an over-used term that it has effectively lost any meaning it may have once had.  Of course, I argue it never had any meaning, since anyone and everyone decided to put the term on every piece of technology being sold, regardless of whether it honestly had anything to do with compute workloads that could be automatically moved between data centers based on business criteria or not.  Suddenly, every USB drive has something to do with the cloud.  Every software vendor on the planet has a cloud offering (which just has to be run on a large farm of servers in your datacenter), and before long your cereal box is going to have some magical connection to 'the cloud'.  

At some point one of two things has to happen.  Either the phrase 'cloud computing' must take on meaning, and by meaning, I intend to say that everyone agrees to what it's definition is, or two, if you open  your mouth to talk about it, you're going to be told by your prospect or customer to shut it, because they're sick of being talked to about things that have no real product/solution behind them.  Remember, IT solutions should do one or more of three things: 1) reduce costs, 2) avoid costs, and 3) increase revenues.  If you can't start to show some real dollars and real components of the solution, why are people going to listen to the conversation.  Yes, the concept is cool, it is flashy, it plays well to analysts as 'visionary', but if it can't really be executed on, what's the point?

Video killed the radio star, and hype may kill cloud computing.  We'll see.

Marketing is a great thing, but sometimes it just goes too far.  All of this "cloud" talk is a prime example.

The discussion around cloud reminds me of the proverbial spring/summer activity of laying in the yard as kids and watching the clouds go by, calling out what it is you think this or that cloud is. 

"Oooh, look at that one, it's a dragon!" 
"What? that's not a dragon, that's a turtle!"
"Uh, huh, but now it's a sheep!"

The cloud is what you want it to be, because it is a giant nebulous concoction of water vapor and ice crystals, constantly changing shape and direction as dictated by the wind.  It isn't any one thing.

That's great for childhood games, but as an architect and a consultant, I've just about had enough of all this.  IT is a concrete thing.  Technology is black and white (or 1 and 0 if you prefer).  Nebulousness (is that a word?) does not fit in this environment.  You don't get to say to the CEO, "well, it was sort of online, depending on what your definition of 'online' is" (or depending upon your definition of 'is' for you politicos out there).  It's on or it's off and the business doesn't care why it was off if they needed it on. 

Every IT vendor out there has some definition of 'cloud'.  Literally, every single one.  Look at the web sites.  Everyone can tell you how they fit into the 'cloud', but not a single one of them can tell you what the 'cloud' really is.  Of course, VMware and Cisco think they have a good definition, as long as it means running everything on the Cisco unified compute platform with VMware.  That's the cloud?  Hmmm, I thought 6 months ago that was called consolidation.  Amazon thinks they have a definition, as long as it means shipping your data out to their server and storage farm to run compute cycles in their data center.  You know, I thought that was just good old-fashioned outsourcing.  SAP's got a definition, EMC has one, HP, IBM, everybody out there has one, and they are all centric to the products they've been trying to sell everyone for years.  Yes, there are some unique aspects to some of the products (EMC Atmos comes to mind), but for the most part, it's all the same with re-packaging.  vSphere and vCloud are ESX server with some really neat new features (and some completely undelivered as of yet promises), but it's still ESX Server.  The Cisco unified compute platform is servers.  Granted, an interesting architecture and management footprint, but let's be real here, they're servers.

What the 'cloud' really is in the IT context is the same thing it is in real life—vapor.  It's the second coming of the xSP model that will save all businesses from the pain of having an IT department.  I said to someone delivering a cloud pitch the other day "did you activate a time warp just before you came in here and take me back to 1999/2000?  These are the same concepts that were pitched at me about ASPs, data center outsourcing, and eCommerce 10 years ago!"  At least it sure feels like it to me.

Over the next few posts, I'm going to keep delivering my take on this topic, if nothing else to flush out some things and vent a little.  I think that there is too much hype here, and we're headed to another bubble bursting if we aren't careful.  The foundation of a technological breakthrough requires more than just a whizz-bang technology.  It requires sound planning, terrific execution, and a goal.  I don't think that anyone talking about cloud is providing those.  There's a lot of theory, a lot of talking, but not a lot of reality.  I'm all for open discussion, but could we maybe back off the child-like excitement that IT is going to be saved and suddenly align with the business based on the 'cloud'? 

I don't plan on making this a rumor mill blog, but this one's pretty good. StreetInsider.com and other financial sites are speculating that if EMC wins the battle for Data Domain (how could they not?), NTAP will go after CommVault.

Now THAT's an interesting move that has a wild chance at success!  (In case you couldn't tell, that was written with a lot of sarcasm).

If I were CommVault folks, I'd be worried right about now.  NTAP has a great track record of buying software companies and turning them into, well, bought companies.  Topio went away, Spinnaker is nowhere to really be seen outside of a roadmap slide that shows the same 'Coming Soon' graphic quarter after quarter, the Decru product seems to show up less and less every time I cross paths with NTAP -- are you detecting a theme here?  

NTAP does some things very well.  They should be rightly credited with driving the unified storage market to the point it is at today, pushing the thinking of the marketplace to places many didn't think it would go.  They have proven that 'good enough' in many situations is really good enough, and unified storage platforms can effectively solve many storage needs in the middle and upper middle marketplace.

They also do some things not so well.  They don't utilize space on the array very efficiently.  Their implementation of Fibre Channel is a little less performant than I'd like it to be at scale.  Their track record of acquiring at least moderately successful software companies and turning them into a game-changing feature or product line is, well, less than stellar.  They tend to get a little too testy when someone questions their technical methodologies.  Nothing horrific.  They aren't a bad company, and their technology is good, just not as good as others, in my opinion.

To me, this acquisition just doesn't make sense.  NTAP doesn't have a history of being able to sell software effectively.  Bringing a company like CommVault into the fold will take superior execution and understanding that the model for selling software can be quite different than that of selling hardware.  It took EMC several tries to grasp this, and it almost didn't work out very well for them (and in some cases it could be debated that they grasped it in time.  Legato, anyone?).  

How do you go from all but acquiring another hardware player that seems a perfect fit to a forced software buy?  You're not going to take CommVault and integrate it into the filers like you could the DD code.  Even if they could make it work, it would suck down so much performance on the array you'd have to buy FOUR of them to get anything accomplished in your environment.  Do you buy them and just leave them alone, letting them do their thing but add to the bottom line?  Maybe, but that doesn't really seem to gain them much.  Or do you swallow them up hoping it makes you more attractive to HP and they then gobble you both up to replace their anemic and ailing EVA line and obtain a modern backup product?  That doesn't seem to be 'in your face' enough for Donatelli to do, but it might just be enough of a poke in Tucci's eye to do it.  It would certainly stir the pot up a little more, and would make some rational sense.  It seems more likely to me that HP will pursue heavy R & D in the LeftHand product and try to turn some of that 'Invent' power loose to turn the storage world upside down.

What a weird day this turned out to be.  

Chuck Hollis did a pretty decent job (in my opinion) of putting a good public face on the EMC Data Domain acquisition on his blog.  Steve Duplessie is missing the boat a little bit (again, my opinion) in his blog entry on the issue, but I do think he gets a couple of things right.

Steve's point about EMC having 82 ways to do dedupe is misguided and feeds into a FUD-Fest for all EMC's competitors in the marketplace.  Now maybe this should be blamed on EMC's marketing group for not delivering a crystal clear message around deduplication, compression, and single instance storage.  I can buy that argument.  But laying this cacauphony of competing dedupe strategies in the marketplace at EMC's feet isn't right or fair.  Every player in the market has some responsibility for that issue. 

I also don't see EMC having another way of deduplicating data as a negative.  I quite frankly get tired of hearing the mantra'd pablem of "EMC has 82 ways of doing <insert functionality here>.  Which one will they try to sell you today?"  That drivel typically comes from the same camp that tries to fix their plumbing problems at home with a hammer and then wonders why the bathroom is full of water.  Get over it.  If you had billions in the bank to buy companies that fit your long-term strategy you'd do the same thing.  Good strategies aren't built around a single way of doing things.  That's what got EMC into so much trouble in the early part of 2000-2001.  Everything was done their way (Symmetrix) or you were an idiot and not worthy to buy their technology.  EMC provides choice to their customers and flexibility in how to design a solution for their most pressing information problems.  Why does everyone beat them up for that?

Anyway, I could go on for a whole post on that topic alone, so let's get back to the topic of the moment.  The acquisition makes perfect sense on two levels:

1. It stops a major competitor in the marketplace from getting better.  So what if this is a defensive move on EMC's part?  Defense wins championships.  Good for Joe and the EMC Board for executing an excellent defensive maneuver.  As a shareholder, I applaud them for not letting NTAP walk in and change the marketplace (again).  Thank you for finally recognizing that while backup as we know it is in a death spiral, the spiral is slow, and EMC needs to do more than try to convince everyone source-based dedupe is the salve that cures all wounds.  Now, go add to the investment and turn the R & D group loose on this code and get it everywhere as fast as possible.  Don't kill the product, don't take a 5 year ramp to understand how to use it, just go get it done!  I want to see products with elements of this code embedded in them within a year.  You brag about all your R & D, now let's see something real come out of it.
2. The code is good.  The device just works, and it isn't because of the hardware.  It's because of DD's code and approach.  Why not get the best code you can for this feature?  EMC knows more about backup than just about anybody in the marketplace, and when they turn this product line loose on their sales force, watch what happens.  It will be nothing short of amazing, if history is any past indication of potential success.  EMC spends a lot of time training their folks on backup, as well they should, and the only real challenge they'll have is getting their faithful followers to stop spouting the 'Data Domain sucks' mantra.

As a reseller of both EMC and Data Domain, this is a winning combination for us.  We no longer have to hear about how bad DD is from EMC, and fend off the lesser (though workable) Quantum solution when we believe target dedupe is called for.  We will get better support from the larger EMC support org, and have everything under one roof.  That helps our business quite a bit.  It takes some of the best minds in backup in the industry and puts them in one house, which for the partner community makes life easier.  Woo Hoo!

That said, here's what I don't like: 

1. EMC paid way too much for the feature, at least on the face of it.  It's going to be tough to recoup this, even with stellar execution.  They should have bought this company at least 2 years ago.   It was a miss on their part to not have done so.  I'm sure if you add in the lost opportunity cost for NTAP, EMC will make out like bandits, but in Joe's stellar track record of acquisitions, I just think this one is at least a partial miss, only because the timing was so late.  But, sometimes you have to overpay and suck it up.  This is probably one of those times.  It just makes me wonder what else is being missed that could change the landscape of the industry?
2. I fear Quantum is dead.  This is where things get a little ugly.  Can Quantum survive on tape alone?  I doubt it.  Their engine has been soundly decried as inadequate by their largest proponent.  That's not good for business.  I like the tape products for what they do, and it will be disappointing to see Quantum flounder further, as they have some really good folks over there who work hard to do what they do.  Perhaps EMC will finally make it official and forgive the $100M they loaned them by just buying them up, finally putting a tape product directly into their catalog.  There are lots of reasons this doesn't make sense, but some that do.  We'll obviously have to wait and see.

At the end of the day, target dedupe is a technology with a very limited lifespan, just as all VTL type products.  CDP technologies, automated information movement platforms, and improved compression techniques that save replication bandwidth costs will make 'backup' as we know it a dead idea in a matter of years.  How many, I can't say.  But if EMC (and others) keep going down the path of newer, tighter, faster compression and dedupe algorithms that make it possible to transmit terabyte data sets across less than OC-level bandwidth in reasonable timeframes, and maybe even T1/T3 types of bandwidth, we won't be having a conversation about backup anymore.  This applies across the board, little AX4/NX4 companies and V-Max companies alike.  They all suffer from the same problems, and these technologies allow for the problem to be solved at both ends of the scale spectrum. 

So get on the pony and ride, EMC.  Ride fast and get this acquisition done and the product into R & D.  Get us new stuff ASAP!