Things to keep in mind if upgrading to vSphere 5.5

If you are a super early adopter there are a few KB articles that you will want to keep an eye on if you are looking to go to vSphere 5.5 already.

Thanks to Marcel @ Lumenate for pointing these out to me… apparently they were on the download page for vSphere 5.5, but who reads the instructions right ?

KB2060511 – Upgrade from vSphere 5.1 to vSphere 5.5 rolls back after importing Lookup Service data (2060511)

KB2060901 – Active Directory authentication fails when vCenter Single Sign-On 5.5 runs on Windows Server 2012 and the AD Domain Controller is also on Windows Server 2012 (2060901)


Also if you are upgrading an existing vSphere server you will want to make sure that your SSL certificates are not expired.


  1. Locate the certificate file (rui.crt) by viewing the contents of the file (default on Windows is C:\ProgramData\VMware\VMware VirtualCenter\ssl)
  2. Run openssl against the certificate.  On Windows, the program is located in

C:\Program Files\VMware\Infrastructure\Inventory Service\bin\

openssl x509 -in C:\ProgramData\VMware\VMware VirtualCenter\ss\rui.crt -noout -text

  1. Look for the validity dates:


    Not Before: Nov 15 21:46:40 2012 GMT

    Not After : Nov 14 21:46:41 2022 GMT

And make sure that the “Not After” date is in the future.

Here is an example from my vCenter server, look for the dates about 10 lines down from the top.

ssl cert


My Frustration with HyperV, do you really save anything?

If you follow my Twitter feed you knew this article was coming…

After spending about 3 hours on the phone with a customer trying to help them get their CSV (Cluster Shared Volume) for HyperV back online (I was on the phone not because I’m a HyperV guy but because I implemented the storage they use for HyperV) I decided that maybe I should do a deep dive and learn a little more about how CSV’s work so that I can better compare them to VMFS. VMFS is the clustered file system that VMware uses to share SAN LUNs between physical VMware servers. This article also goes on to explain why HyperV is an inferior solution due to some of the other issues that I’ve seen in my limited experience with HyperV.

Anyhow, before we get to the technical stuff I need to point out that I am obviously biased to VMware… it’s what I do every day. I will try to be as fair as possible, but lets face it if you are just looking for the take away without reading the whole post it is that VMFS is far superior to CSV’s, and that while VMware might look more expensive on a bill of materials it will probably save you time and money in the long run.

On to the technical stuff.

Cluster Shared Volumes have been around a long time, Microsoft has been using them for everything from Exchange clusters to SQL clusters. They adapted CSV’s to work with HyperV so that they could allow virtual machines to more easily move from one HyperV host to another, similar to what VMFS allows VMware ESXi servers to do. Both enable high availability for virtual machines because if a host fails, other hosts can access the virtual machines the failed host was running. Additionally CSV’s are needed because under them is NTFS… which was never designed to be accessed by multiple systems at the same time, because of this something had to be put in place to allow that to happen.

Ok so the first article I come across on Technet has this to say:

“…the Cluster Shared Volumes feature included in failover clustering is only supported for use with the Hyper-V server role. The creation, reproduction, and storage of files on Cluster Shared Volumes that were not created for the Hyper-V role, including any user or application data stored under the ClusterStorage folder of the system drive on every node, are not supported and may result in unpredictable behavior, including data corruption or data loss on these shared volumes. Only files that are created for the Hyper-V role can be stored on Cluster Shared Volumes. An example of a file type that is created for the Hyper-V role is a Virtual Hard Disk (VHD) file.
Before installing any software utility that might access files stored on Cluster Shared Volumes (for example, an antivirus or backup solution), review the documentation or check with the vendor to verify that the application or utility is compatible with Cluster Shared Volumes.”
Taken from:

So to me that means that CSV’s are flaky to say the least… but lets continue.

VMFS on the other hand can store pretty much anything you can upload to it… zip files, iso files, etc etc… VMFS is almost like LVM in linux, it doesn’t care what you put on it.

On to my next point…

All HyperV nodes that are using a CSV are at the mercy of the coordinator node for that CSV. Think of it this way, you need to look for something that is in a filing cabinet, but before you can actually get the folder you need you must first talk to the secretary and ask her if its ok to look at the folder. In more technical terms this means that the coordinator node keeps track of all the metadata and file locking on folders on the CSV, after the coordinator node allows you access to the folder then IO to things in that folder happen directly to the LUN. But don’t take my word for it…  Microsoft explains it in this article

What scares me about this method is that if something gets hosed with the Coordinator node and the Failover Manager doesn’t fail over properly your CSV is inaccessible. And going by the reliability track record of Microsoft Services I would not bet my job on Failover Manager 🙂

VMFS on the other hand is a clustered file system that has no owner… there is not any one node that controls access to the file system. File locking is done at a file level by a ‘pulse field’, and in this field a host must periodically update its time stamp and let the file system know that it is still using the file. If a host crashes and another host wants to use the file the host can let the file system know that the time stamp hasn’t been updated lately, and that it is taking over ownership…. this means that each host can access files in the even of a node failure without waiting on a response from a centralized management node. If you want the in-depth answer check out this article

Next lets talk about how to get HyperV to actually work …. and I’m not talking about just getting a VM to boot up. I’m talking about setting up HA and DRS and automatic load balancing etc etc. With VMware you group all of your physical servers into a Cluster and then check two boxes… one to turn on HA and one to turn on load balancing. Of course you need to set up network interfaces for vMotion… but other than that your done. Oh and by the way this was all done from the VMware vSphere client.

On to HyperV… lets see where do we start…. no really… which interface do you want to start with. If I want to configure virtual machines I’ll need to use HyperV Manager… or maybe Systems Center Virtual Machine Manager (SCVMM). If I want to set up CSV’s to allow for HA to take place I’ll need to fire up MS Failover Manager. On and if I want to actually format a LUN, well then I’ll need to get into Disk Management.

I’m sure you get my point….

We could talk about virtual switches and distributed virtual switches (or the lack there of), or things like NIC teaming and how it is implemented, but I don’t want to write a book.

The take away

OK so normally the number one reason I hear that people are using HyperV is because it comes with Windows … Its free. But is it really free? I could argue that it takes less time to implement VMware than it does HyperV… and isn’t it known that “Time is Money”… it certainly is if you are paying someone to set it up.

I could also argue that 3+ hours of downtime trying to resolve a CSV issue where VM’s are not accessible is definitely a loss of productivity and in turn a loss of money.

Lastly SCVMM is not free and while it is not required, if you want to compare apples to apples you will want it…

So in the end VMware may be an additional line item on a bill of materials, but in the end it may be the best damn investment you will make for your virtual environment.

If you are not convinced yet here are some other fun stories about the HyperV

  • <- Failover manager required AD, if you virtualize all your Domain Controllers Failover Manger wont start.
  • <- Expanding a datastore in vmware is as simple as right clickign the lun to extend, selecting expand, clicking next like 3 times…. check out this link for the process for a CSV 🙂
  • do a google search for HyperV live migration using multiple NICs…. you wont find much, so if you buy that server with 128 or 256 GB of ram and need to take it out of production for maintenance… better have 10Gbps networks in place or grab dinner and a movie… (1Gbps network can theoretically move 125MB/sec or 7.5GB/minute so to move 256GB that would be about 34 minutes… so if you want to do a rolling outage multiply the number of hosts you have by 34 minutes to move all that ram….  just one more way VMware costs you less)
  • Look up how to do nic teaming in HyperV… Then look up how to do it in VMware

The bottom line

If you want a solution that is straight forward and easy to use then HyperV is probably not the way to go. While it may not be a big line item on a bill of materials, what it saves you there it is certainly to cost you in time and effort (not to mention frustration).

Spring Cleaning Tips for your VMware Environment

I just wanted to throw out some tips that I run into from time to time in customer environments. These are just meant to be a quick reminder of the things that I find are missed during installations. If you have any to add to the list please leave a comment below, Thanks!

vSphere Tips:

  • vSphere needs to know about your mail server or it wont be able to send you alerts. Go to Administration -> vCenter Server Settings and then find the mail section in the list. FIll in your SMTP server as well as the sender email address. This will make sure that when an alert is triggered it will be able to send an email if the action is to do so.
    vCenter Server Settings in the Administration menu
    vCenter Server Settings in the Administration menu

    Mail server settings for vCenter alerts
    Mail server settings for vCenter alerts
  • Alerts also need to be configured…. In fact vCenter will do nothing more than trigger the alarm by default. So if you want email alerts, make sure to go into each alert you want emailed to you and set it up to do so. click on your vCenter server in the left tree, and then select the alarms tab. Next click on Definitions and right click each alarm you want to send an email for, then go to the Actions tab and click add to add an email alert action. Lastly you need to specify where that email needs to go.
    If the idea of setting every one of these alarms seems like too much work your not alone. I blogged about using powershell to automate the process a while back over here…. and Lauren Malhoit blogged about it over here.
  • Alarm definitions, under the vCenter Server;s Alarms tab
    Alarm definitions, under the vCenter Server;s Alarms tab

    Actions tab where you can specify to send an email as where to send the email alert to
  • Remember to Update your hosts! OK how many of you used to brag about your windows or linux server up-time ? Well this isnt the old days so remember to update your ESXi servers and reboot them to apply those updates when needed. Your windows and linux servers will still report no downtime if you use vMotion to move the VM’s and this will help to avoid bugs and other problems in the future. To update your hosts go to the “Update Manager” tab and run a scan after attaching the critical and non critical baselines to the host or cluster.
    The update manager tab showing the attach and the scan links
    The update manager tab showing the attach and the scan links

    A VMware host that has all current updates applied
    A VMware host that has all current updates applied
  • Make sure your hosts are placed properly. I have seen customers create a datacenter object in vCenter, as well as a Cluster Object. However when they add their hosts in to vCenter they will accidentally put them under the datacenter instead of the cluster. So when things go south and they lose a host, they expect HA to kick in… but since the hosts arent in a cluster HA does nothing.

    Hosts go under the Cluster Object, unless they do not need to work with similar hosts for HA and DRS
    Hosts go under the Cluster Object, unless they do not need to work with similar hosts for HA and DRS

Storage Tips

  • Multipathing is a must for both iSCSI and Fiber Channel, but is yours working right ? The easiest way to check is to go into one of your hosts and go to the storage adapters area under the Configuration tab.  Here you will see a list of hba’s or the VMware Software iSCSI adapter. Irregardless some simple multiplication can tell us what we need to know. If the number of paths you have is not a multiple of the number of targets times the number of devices then you have a problem.

    Multiply the numbers to get the paths
    Multiply the numbers to get the path
  • Make sure you use Round Robin path selection if possible… check with your vendor to see if Round Robin is an acceptable PSP, if it is make sure to set it (on every host, and every lun). Note that ESXi 5.1 can automatically detect and set Round robin for VNX arrays. Note the picture below is a VNXe which will not have non-optimal paths, but most arrays will show two “Active (I/O)” paths and two “Active” Paths

    Round Robin Path Selection Policy, Paths Showing as Active I/O
    Round Robin Path Selection Policy, Paths Showing as Active I/O

Installing HP CIM drivers on ESXi 5.0

What is CIM?

CIM means Common Information Model, and they allow for communication between the ESXi Operating System and the hardware sensors.

Why do I need it?

Normally I am a big fan of using the HP ESXi ISO image to load servers with ESXi instead of the VMware roll if its being loaded on an HP server. The HP image is a derived from the VMware image, but they add in all the HP drivers and CIM drivers so that you don’t have to manually install them. You can always tell when an HP server has been loaded with the VMware image because under the “Hardware Status” tab there is no reference to storage devices. See Figure 1. (Click to zoom in)

Figure 1. Health Status before HP CIM drivers

Figure 2 shows the “Health Status” after the HP CIM driver install.

Figure 2 Health Status After CIM install

As you can see we now have detailed information about the Array controller and what it is doing. This is extremely important if you have local storage in the server, because without getting this information to vCenter, you wont be able to get email (or other) alerts if a driver were to fail.

There is one problem with the HP ESXi image though… it always a few days/weeks behind the official realease, and if you install a bunch of VMware servers like I do you may not be able to wait a week for it to come out. That is why sometimes we need to manually install the CIM providers on ESXi.

So how do we manually install?

In order to manually install the CIM providers we first need to download the zip file from HP’s website. In this example I am installing them on a DL360 G6 server so I went to its drivers page and looked for the following item.

Figure 3 The HP CIM providers can be downloaded from here

After downloading the drivers upload them to a datastore. If you are going to do multiple servers your best option would be to already have a SAN or NFS datastore shared among the hosts to upload to. After uploading you have two options, you can enable SSH on each of your hosts or you can use the vSphere CLI interface, I would let how many hosts I need to do determine which of these two methods I would use. Because I only had one host to update today I just enabled SSH and used the following command:

esxcli software vib install -d <datastore path><>

But if I were to use vSphere CLI I would have needed to use this command:

esxcli -s <server> -u root -p mypassword software vib install -d <datastore path><>

The advantage of using vSphere CLI is that you should only need to change the server name in the command for each host you want to do. After a successful install you will also need to reboot the host before the new CIM drivers will take affect.

Figure 4 Successful install output

VMware SRM5 and vSphere Replication

With the release of VMware vSphere 5 also came the release of many other products, one of those was Site Recovery Manager 5. Site Recovery Manager is an add-on product to vSphere that allows for a disaster recovery plan to be scripted out and followed to the “t” during a recovery process.

In older versions of SRM, storage plugins (and compatible arrays) were required for the replication of virtual machines from site A to site B. However in this latest release compatible storage arrays are no longer required, in fact you don’t even need to have the same brand of array on each side. Why? well VMware has introduced its own replication appliance which handles all of this for you. Of course if you do have compatible arrays and have already purchased replication features you are still able to use those as well.

When I first started working with VMware Infrastructure 3, SRM was considered a real pain to setup because of all of the steps involved. This is not the case in SRM5, in fact I was able to set it up from scratch in less than a day (with my vSphere backend already in place).

So the principal behind SRM is that you have at least two sites, one of them has virtual machines that you intend to protect from a disaster, and to make sure they are protected you replicate them to a Recovery Site. (Pretty graphic below) The beauty of SRM is that it automates the process of recovery and reduces (and maybe eliminates) the need for a run book. It makes it so easy that even the entry level help desk tech could do it 🙂

There are many limitations to SRM 5’s new vSphere Replication feature, so make sure to read the product documentation before purchasing to make sure its the right fit for you. But here is a quick over view of what I was able to achieve with some lab time.

Remember those annoying “Summary” tabs that you have probably turned off in the vSphere client ? Well it turns out that there is actually one that is very useful. The graphic to the right is a screen shot of the “To Do” list that is on the Summary page for SRM 5. It lists out step by step all of the configuration steps that we need to take in order to implement SRM. I won’t go through all of them in this post, but the main take away is that with 7 major steps we can have SRM up and running.

The biggest hang up I had was configuring vSphere Replication (Step 5), and that was only because I did not have SQL configured properly on each site. After resolving a misconfigurated SQL instance I was able to connect to my databases on each side and SRM was ready to go.

After going through all of the steps I was then able to pick any of my virtual machines at the protected site and tell it the Recovery Point Objective I needed, and the rest is history…. It just took off and started working. One gotcha is that I don’t think seeding your DR site is supported in this initial release of vSphere Replication (at least I didn’t see the option to). But where I was testing we have a dedicated 100Mbps point to point link, so seeding was not a big deal.



Tweaking VMware’s Round Robin Settings

One thing that should always be done when using 1Gb iSCSI is setting up Multipathing, and in all version of vSphere they have included NMP  or Native Multi Pathing. With NMP comes a path selection method called Round Robin which as with anything else with that name means that IO is send down each path, one at a time. By default, VMware has set a limit of 1000 IO’s to travel down each path before switching to the next path. However by changing that default down to 1 IO before switching paths we can sometimes achieve much greater throughput because we can more effectively utilize our links.

There has been much debate over whether this is really worth it or not, so I thought why not try it out and see what effects it would have. The test environment that I used included the following hardware:

  • Hp P4300 (Lefthand) SAN with 2 Storage Nodes
  • Hp DL380 G7 Server with 2 – 6 core Intel 5675 CPUs (3.07Ghz ;)) and 72Gb of RAM
  • Cisco 3560G switch

I setup the HP P4300 SAN with 802.3ad bonded links to the switch and turned on channel groups in passive mode on those links. As for the server, I loaded ESXi 5.0 and used two VMKernel interfaces with two NICs… I also did not use the software VMware software iSCSI initiator, but instead used the BroadCom iSCSI offload initiators because of the ease of setting them up in ESXi 5.0.

After mounting my LUN and letting it format VMFS5 I set the path selection policy to Round Robin. I then Installed a Windows 2008 R2 VM and followed the instructions from for running IOmeter (just like I have for the other SAN tests).

Here are the results with a standard ESXi 5.0 host using Round Robin with an IO limit of 1000 per path per turn. The Maximum read throughput is 129MBps, and the RealLife IOps are almost 1600.

Now lets see what turning the IO limit down to 1 IO per path per turn gets us. Below you can see that we can now achieve 207MBps on Read throughput and our RealLife IOps is now at 2091 IOps. Plus every other number went up as well!

This is an increase of 60% on the Maximum Read throughput and a 30% increase on the number of IOps!

So how do you change the limit ?

Setting the limit to a ower value is described in the HP EVA best practices guide, but I could not get their command to work.

The command for ESXi5 is:

esxcli storage nmp psp roundrobin deviceconfig set -d naa.devicename --iops 1 --type iops

and to get the device name you could use:

esxcli storage nmp device list | grep naa.600

If you want to check out their guide it can be found here:


HP ESXi Hosts continually disconnect from vCenter

Today I setup three new ESXi 5.0 hosts with the HP Branded ESXi media, all went well until I joined the hosts to vCenter. After joining and registering with the vCenter server then became disconnected. I tried to manually connect them back in and they looked like they came back… but would almost instantly disconnect again.

After some direction from @Ollfried on Twitter I did a quick Google search and found this thread

If you scroll down a bit you see where they are telling you to remove an HP VIB file that has to do with the licensing. I can confirm that this does fix the problem and after rebooting the hosts I was able to add them back to vcenter and all is well.

To fix the problem:

    1. Connect to each host with vSphere client and enable SSH
    2. SSH to the host and run the following command
      esxcli software vib remove -n hp-esx-license --no-live-install
    3. Reboot the host

What it looks like:

Building a vCloud Lab Part 5: vCD Installation

Now that we have our hardware all in place, vSphere 5 Installed, our database configured and running, and vShield deployed we are ready to build our vCloud Director virtual machine, which will be Red Hat Linux, and get vCD setup and running.

Getting Red Hat

While I’m not a fan of buying Linux, we will have to make an exception in this case as vCloud Director is only supported on Red Hat Enterprise Linux 5. So the first thing we will have to do is sign up for a demo copy of Red Hat and download the ISOs. After those are done downloading we can then install a RHEL5 server and use SCP to transfer the vCloud Director binary file over to it.

Before starting your install we should make a couple modifications to your RHEL VM’s hardware, specifically we need to add a second network adapter because vCD requires two static IP addresses. I used VMXnet3 NIC’s on my VM.

After you have RedHat installed and you are able to login, we need to run ‘setup’ so we can disable the firewall and SELinux. You can also use ‘setup’ to set the two static IP addresses here too. However, if you are using VMXnet3 NIC’s they will not yet work, first we need to install VMware tools. Since this is RedHat is is a pretty simple process:

  1. Click “Install/Upgrade Tools” from the menu
  2. Mount the CD-Rom drive somewhere (ex. mount /dev/hdc /mnt/cdrom)
  3. Copy VMware Tools to /usr/src (ex. cp /mnt/cmdrom/VMware-tools…..tar.gz /usr/src)
  4. Change directory to /usr/src then Extract the tools (ex. tar xzvf VMware…..tar.gz)
  5. cd into the tools directory cd <name of directory>
  6. run the installation Perl script ./
  7. Reboot the VM so that both NICs are found and configured

One of the two static IP addresses will be used for the web interface of vCD and the other will be used for the console proxy sessions. We will need to reboot the VM after disabling SELinux and the firewall, do that now.

Now that we have all of our IP’s set and tools installed we can now install the rest of the required packages for vCD, you can check the vCD install guide for a list of them or just copy in the following command:

yum install alsa-lib bash chkconfig compat-libcom_err coreutils findutils glibc grep initscripts krb5-libs libgcc libICE libSM libstdc libX11 libXau libXdmcp libXext libXi libXt libXtst module-init-tools net-tools pciutils procps sed tar which

Creating SSL Certificates

If you are creating this vCloud setup for public access you will want to read through the vCD installation guide to get info on how to import certificates, but if you are creating this for internal use then we can simply generate self signed certs. To do this we just need to go to the /etc directory (ex. cd /etc) then paste in the following commands:

Note if you want a more secure key store password (other then password) change it before pasting, also make sure that the quotation marks are copied in properly and not converted to a period.

/opt/vmware/vcloud-director/jre/bin/keytool -keystore certificates.ks -storetype JCEKS -storepass password -genkey -keyalg RSA -alias http -dname “cn=vcloud,  ou=vmware, o=vmware, c=US” -keypass password
/opt/vmware/vcloud-director/jre/bin/keytool -keystore certificates.ks -storetype JCEKS -storepass password -genkey -keyalg RSA -alias consoleproxy -dname “cn=vcloud,  ou=vmware, o=vmware, c=US” -keypass password

Now we will have a certificates.ks file located in /etc/ we will need that information during the vCD installation.

Installing vCD

Now change directory over to where you transferred the vCD installation file to and make sure its executable by running this command:

chmod u+x <vcd installation file name>

Now we can execute the file by typing:

./<vCD installation file name>

You will be asked which IP you want to use for the web interface and which for the console proxy, you will also be asked where your certificates are located. This is where we will enter “/etc/certificates.ks” without the quotes. Later in the installation you will then be asked for your database server information. Since we will be using vCD 1.5 we will have the option of Oracle or MSSQL. After entering the proper information it will create our database structure, this process takes a couple minutes. After it is complete we should be ready to login to the vCloud Director web interface. Which is located at https://<IPaddress> where the IPaddress is the IP of the NIC that was used for the web interface… not the console proxy ip.

Congratulations you should now have a working vCloud Director lab! Stay tuned for more articles on vCD in the future.

Building a vCloud Lab Part 4: vCD Database Prep

One part that I forgot to mention that is in the vCloud Installation Guide is preparing your database server for use with vCloud Director. This isn’t a difficult process if you know how to use SQL. Because I decided to go with MS SQL with my vCloud 1.5 installation instead of Oracle, this article will focus on how to prep the MS SQL database; Although the Oracle prep work isn’t much different. Before we get started I would like to mention that we need to install SQL in Mixed Mode Authentication, so make sure to check that and set an “SA” user password during install.

Allowing remote TCP/IP connections to SQL

By default SQL 2008 R2 Express (like the others I assume) do not allow remote connections via TCP/IP, therefore we need to enable that before we can connect our vCloud Director Server to the database we will create later on. The simplest way I found do do this is from this article on the WebECS Support pages. It will take you through step by step on how to enable remote connections, but remember to restart SQL at the end because the changes won’t take affect until you do.

Creating a vCloud Database

The first step is to create our vCloud database, to do that we need to open up the Microsoft SQL Management Studio and login. After logging in click the “New Query” button below the File menu. Then paste the following query into the window, note there are several things you may want to change, including the “vcloud” user password as well as the database location, otherwise just paste in everything below:

USE [master]
(NAME = N'vcloud', FILENAME = N'C:\vcloud.mdf', SIZE = 100MB, FILEGROWTH = 10% )
(NAME = N'vcdb_log', FILENAME = N'C:\vcloud.ldf', SIZE = 1MB, FILEGROWTH = 10%)
COLLATE Latin1_General_CS_AS
USE [vcloud]
USE [vcloud]
CREATE USER [vcloud] for LOGIN [vcloud]
USE [vcloud]
sp_addrolemember [db_owner], [vcloud]

After pasting that in you should have something similar to this, and once you do click the “Execute” button right above the code in the ribbon bar.

You should now have a new user account called vcloud with a password of vcl0ud, and a new database called vcloud, unless of course you changed the parameters before executing. These queries also set the new vcloud database as the default database for the vcloud user, and it assigns that user the db_owner role.

We are now ready to connect up a vCloud Director server to our database… Check out the next part of this series to find out how to do the vCD install on a RedHat OS.

vSphere 5.0 Ready for Download

Last night VMware officially released vSphere 5.0 for download. If you have a current support contract you can head on over to and start flooding your internet connection. Or if you are not already a VMware customer just sign up for a trial and you will get instant access to the downloads.

5.0 Licensing

If you are a SnS subscriber then you will probably want to get your 5.0 keys so that it doesnt expire. (I will break it to you now that they are not ready yet in most cases But this is the process to get them)  To do this click on “My Account”, and then on the left select “Manage Licenses” and then in the second drop down select ” vSphere 4″. Here is what that should look like:

Then click Submit.

You will then get the “error” message that I did until your keys are ready.