Patch Discipline, VMware, August 12th

Our Virtualization Lab VMware hosts were at 3.5 Update 2, while the ESX hosts in our Syracuse lab were still at a base 3.5 build. Guess who's in better shape this evening?As widely reported, many VMware shops are, um, uncomfortable today. The first reports started trickling in from Australia while it was still yesterday back here in the United States. Following the sun, clocks ticked over and many VMware customers who had kept up with the latest patch (ESX 3.5 Update 2, released July 27th) encountered a licensing foul-up, preventing VMs from spooling up or Vmotioning to a new host. VMs already in a live state are unaffected; the license check occurs when a VM is powered up or moved.

I'm not going to discuss the relative merits of restrictive licensing schemes. We all have our opinions; enough folks are grumbling that I don't need to be another voice in the crowd.

The folks at VMware are working very, very hard on a patch. The initial goal was sometime today; a blurb on the main VMware support page is now targeting end of day, 8/13, Pacific Standard Time for a patch to resolve the problem. The link for more information returns a loopback error as of this post. I'll cut 'em some slack on the Web issue.

Believe it or not, I was offline from Sunday through lunch today and came back to this hullabaloo via my in-box, voice mail, and RSS feeds. Looks like I picked the wrong morning to be out of the office.

Per good security policy, our two Virtualization Lab ESX hosts had been patched to Update 2 at the end of July. Coincidentally, the test lab had been completely powered down since Friday afternoon. (Yes, very green of me.) I fired up our gear and discovered the ESX error that the world is rightfully up in arms about. Mike Fratto at our Syracuse labs hadn't, er, applied the patch. So the Syracuse labs are sailing smooth and Mike is smiling. The VirtLab is shut down 'til this issue is resolved.

None of my production VM hosts are running ESX, so I'll be going to bed early tonight. I wish you well.

If you're affected by this bug, some basic advice for the time being:

1. If your guest platforms and apps aren't time sensitive, you can roll back the clocks on your host a few days. Not very realistic, but this might work for some folks.

2. If you haven't yet done so, disable Vmotioning.

3. Any unpatched hosts in your shop? You could move critical VMs to an earlier-rev ESX 3.5 box.

4. In the name of whatever you call holy, check your change management plans, nix any planned patches to 3.5 Update 2, and postpone any MS updates requiring a reboot!

As with any ad hoc solution, be sure to weigh the production impact and security risks of any short-term resolution.

5. Plan to be up late tomorrow night.

Updated 8/13

From the VMware support site:

Follow the steps below to correct this issue:

1. Read the following Knowledge Base articles first: * Fix of virtual machine power on failure issue, refer to KB 1006716 * For VI 3.5, refer to KB 1006721 for deployment consideration and instruction * For VI3.5i, refer to KB 1006670 for deployment consideration and instruction 2. Download and apply the express patch according to the product(s) you have: * VMware ESXi 3.5 Update 2 Express Patch * VMware ESX 3.5 Update 2 Express Patch

We won't be installing the ESX patch in the lab 'til this weekend. Anyone out there care to share their experience?Our Virtualization Lab VMware hosts were at 3.5 Update 2, while the ESX hosts in our Syracuse lab were still at a base 3.5 build. Guess who's in better shape this evening?