My Frustration with HyperV, do you really save anything?

If you follow my Twitter feed you knew this article was coming…

After spending about 3 hours on the phone with a customer trying to help them get their CSV (Cluster Shared Volume) for HyperV back online (I was on the phone not because I’m a HyperV guy but because I implemented the storage they use for HyperV) I decided that maybe I should do a deep dive and learn a little more about how CSV’s work so that I can better compare them to VMFS. VMFS is the clustered file system that VMware uses to share SAN LUNs between physical VMware servers. This article also goes on to explain why HyperV is an inferior solution due to some of the other issues that I’ve seen in my limited experience with HyperV.

Anyhow, before we get to the technical stuff I need to point out that I am obviously biased to VMware… it’s what I do every day. I will try to be as fair as possible, but lets face it if you are just looking for the take away without reading the whole post it is that VMFS is far superior to CSV’s, and that while VMware might look more expensive on a bill of materials it will probably save you time and money in the long run.

On to the technical stuff.

Cluster Shared Volumes have been around a long time, Microsoft has been using them for everything from Exchange clusters to SQL clusters. They adapted CSV’s to work with HyperV so that they could allow virtual machines to more easily move from one HyperV host to another, similar to what VMFS allows VMware ESXi servers to do. Both enable high availability for virtual machines because if a host fails, other hosts can access the virtual machines the failed host was running. Additionally CSV’s are needed because under them is NTFS… which was never designed to be accessed by multiple systems at the same time, because of this something had to be put in place to allow that to happen.

Ok so the first article I come across on Technet has this to say:

“…the Cluster Shared Volumes feature included in failover clustering is only supported for use with the Hyper-V server role. The creation, reproduction, and storage of files on Cluster Shared Volumes that were not created for the Hyper-V role, including any user or application data stored under the ClusterStorage folder of the system drive on every node, are not supported and may result in unpredictable behavior, including data corruption or data loss on these shared volumes. Only files that are created for the Hyper-V role can be stored on Cluster Shared Volumes. An example of a file type that is created for the Hyper-V role is a Virtual Hard Disk (VHD) file.
Before installing any software utility that might access files stored on Cluster Shared Volumes (for example, an antivirus or backup solution), review the documentation or check with the vendor to verify that the application or utility is compatible with Cluster Shared Volumes.”
Taken from:

So to me that means that CSV’s are flaky to say the least… but lets continue.

VMFS on the other hand can store pretty much anything you can upload to it… zip files, iso files, etc etc… VMFS is almost like LVM in linux, it doesn’t care what you put on it.

On to my next point…

All HyperV nodes that are using a CSV are at the mercy of the coordinator node for that CSV. Think of it this way, you need to look for something that is in a filing cabinet, but before you can actually get the folder you need you must first talk to the secretary and ask her if its ok to look at the folder. In more technical terms this means that the coordinator node keeps track of all the metadata and file locking on folders on the CSV, after the coordinator node allows you access to the folder then IO to things in that folder happen directly to the LUN. But don’t take my word for it…  Microsoft explains it in this article

What scares me about this method is that if something gets hosed with the Coordinator node and the Failover Manager doesn’t fail over properly your CSV is inaccessible. And going by the reliability track record of Microsoft Services I would not bet my job on Failover Manager 🙂

VMFS on the other hand is a clustered file system that has no owner… there is not any one node that controls access to the file system. File locking is done at a file level by a ‘pulse field’, and in this field a host must periodically update its time stamp and let the file system know that it is still using the file. If a host crashes and another host wants to use the file the host can let the file system know that the time stamp hasn’t been updated lately, and that it is taking over ownership…. this means that each host can access files in the even of a node failure without waiting on a response from a centralized management node. If you want the in-depth answer check out this article

Next lets talk about how to get HyperV to actually work …. and I’m not talking about just getting a VM to boot up. I’m talking about setting up HA and DRS and automatic load balancing etc etc. With VMware you group all of your physical servers into a Cluster and then check two boxes… one to turn on HA and one to turn on load balancing. Of course you need to set up network interfaces for vMotion… but other than that your done. Oh and by the way this was all done from the VMware vSphere client.

On to HyperV… lets see where do we start…. no really… which interface do you want to start with. If I want to configure virtual machines I’ll need to use HyperV Manager… or maybe Systems Center Virtual Machine Manager (SCVMM). If I want to set up CSV’s to allow for HA to take place I’ll need to fire up MS Failover Manager. On and if I want to actually format a LUN, well then I’ll need to get into Disk Management.

I’m sure you get my point….

We could talk about virtual switches and distributed virtual switches (or the lack there of), or things like NIC teaming and how it is implemented, but I don’t want to write a book.

The take away

OK so normally the number one reason I hear that people are using HyperV is because it comes with Windows … Its free. But is it really free? I could argue that it takes less time to implement VMware than it does HyperV… and isn’t it known that “Time is Money”… it certainly is if you are paying someone to set it up.

I could also argue that 3+ hours of downtime trying to resolve a CSV issue where VM’s are not accessible is definitely a loss of productivity and in turn a loss of money.

Lastly SCVMM is not free and while it is not required, if you want to compare apples to apples you will want it…

So in the end VMware may be an additional line item on a bill of materials, but in the end it may be the best damn investment you will make for your virtual environment.

If you are not convinced yet here are some other fun stories about the HyperV

  • <- Failover manager required AD, if you virtualize all your Domain Controllers Failover Manger wont start.
  • <- Expanding a datastore in vmware is as simple as right clickign the lun to extend, selecting expand, clicking next like 3 times…. check out this link for the process for a CSV 🙂
  • do a google search for HyperV live migration using multiple NICs…. you wont find much, so if you buy that server with 128 or 256 GB of ram and need to take it out of production for maintenance… better have 10Gbps networks in place or grab dinner and a movie… (1Gbps network can theoretically move 125MB/sec or 7.5GB/minute so to move 256GB that would be about 34 minutes… so if you want to do a rolling outage multiply the number of hosts you have by 34 minutes to move all that ram….  just one more way VMware costs you less)
  • Look up how to do nic teaming in HyperV… Then look up how to do it in VMware

The bottom line

If you want a solution that is straight forward and easy to use then HyperV is probably not the way to go. While it may not be a big line item on a bill of materials, what it saves you there it is certainly to cost you in time and effort (not to mention frustration).

Veeam NFR Keys for VMware and Microsoft Certified Professionals

If you are a VMware vExpert, VMware Certified Professional (VCP), VMware Certified Instructor (VCI) or VMware User Group (VMUG) member, you can get a FREE 2-socket NFR license for Veeam Backup & Replication v6 for your home or work lab.
Register here.
Additionally, If you are a Microsoft Certified Professional (MCP) or a Most Valuable Professional (MVP), you can get a FREE 2-socket NFR license for Veeam Backup & Replication v6 for your home or work lab.
Register here.

The licenses are good for 12 months. Using my MCP status, I am going to use my licenses to set up a proof-of-concept for my peers and management for our Hyper-V environment. Today, we are still treating VMs as physical servers when it comes to backup so I hope to show the value-add of having Veeam for faster, more-efficient backups and very quick recoveries.

Hyper-V 3 for Me

Microsoft’s Hyper-V virtualization platform may be a 12 year old hanging out with a bunch of high school seniors, but this pre-teen is mature for his age.  Could Hyper-V be the Doogie Howser of virtual infrastructures?

I am humbled to be invited to share my Hyper-V experiences, opinions, lessons learned, etc. on Justin’s IT Blog.  I am a former co-worker of Justin’s and faithful follower of his blog even though I am not a VMWare administrator;  It’s always nice to know what the other guys are doing.  I have 14 years of experience in the IS/IT field and am currently working in the healthcare industry.  Our Hyper-V environment consists of a single highly-available failover cluster with 120 VMs over seven hosts and makes up approximately one-third of our entire server footprint.

Just like Dr. Howser being able to treat patients as effectively as an older doctor, Microsoft’s Hyper-V is able to meet enterprise-class requirements for stability, performance, and high availability.  However, Doogie, even being a genius, had two character flaws that created each episode’s comedic and learned life lessons.  One such flaw was with lack of wisdom (knowledge gained through experiences).  Whereas matured virtualization hypervisors have been honed by years of use by thousands of customers, Hyper-V hasn’t had that much time to find what works and what doesn’t.  Yes, they’ve paid attention to the lessons learned by Citrix and VMWare but that knowledge cannot equal working through the bugs and challenges of keeping an innovative edge over your competition.  Doogie’s other flaw is that he was a teenager in an adult’s world.  He was torn between socializing with his adolescent friends while still trying to fulfill his adult responsibilities.  Needless to say, conflicts always ensued.  The same comparison can be made with Hyper-V as it is desperately trying to achieve the same level of product maturity and necessary features expected in any medical doctor, yet you still sometimes think “What am I doing putting this much trust and importance into such a young man?”   This brings us to the intended point of this article.  With the release of Windows Server 8, Hyper-V will turn 3 and will bring with it its first mustache.  The difference is this ‘stache is not thin and patchy, it is a full on Burt Reynolds crumb catcher! Our little hypervisor is growing up so fast.

To start off, v.3 will support multiple simultaneous Live Migrations (or VMotions in VMware speak) which VMware shops have enjoyed for many years.  Microsoft is also removing the shared storage requirement to perform a Live Migration which gives more flexibility on the medium in which the VMs are moved (wireless, anyone? No? Good choice!) and the target, even if the target does not have access to the cluster shared volume (VMWare translation: Vdisk).  This is achieved because Hyper-V’s Quick Storage Migration feature is evolving into Live Storage Migration which allows you to move the VM’s disk at the same time as Live Migrating the VM.

In addition to the improvement with uninterrupted migrations of VMs, Hyper-V has learned to do virtual switching.  This was another major feature VMware offered that Microsoft couldn’t match.  With Cisco’s announcement of Hyper-V support with its Nexus 1000V virtual switch, Hyper-V will now be able to support the same inter-VM traffic visibility, shaping, and bandwidth provisioning as ESXi.

To round out the list, Microsoft has increased the scalability of Hyper-V by supporting hosts up to 160 processors (cores and hyperthreads) and 2TB of RAM.  Each VM will support up to 32 vCPUs and 512GB RAM which is a significant improvement over the current 4vCPUs and 8GB RAM.  Also due to an upgrade to the VHDX file format, you will be able to provision up to a 16TB disk.  These scalability improvements along with virtual switching pave the way supporting more enterprise, large-scale apps.

Those squeamish about trusting their health to young doctors (especially when they are the same age as their child) may just want to take another look at Hyper-V when Windows Server 8 is released.  In almost every instance, there is a very significant cost difference between Hyper-V and ESXi environments in both initial acquisition and yearly maintenance.  I believe VMware will have to make some tough pricing changes in the near future or risk losing even more market share to the boy genius.

HP NIC Teaming on 2008 Server Core

As part of an upcoming HyperV project I was tasked with installing 2008 R2 Server Core with the HyperV role on a HP DL360 G7 server.

So I started out by loading the Windows OS just like normal… with the HP Smart Start Boot CD, the only difference was that I selected the Datacenter “Core” install instead of the normal GUI install. Then I sat back and waited, after it was done I logged in and installed the HyperV role as well as joined the machine to the domain. I was able to remotely connect to the server and create a virtual machine to install SCVMM on (which wasn’t too hateful).

But them I got to thinking… I can only assign one NIC to a virtual network ? What the heck !?

Obviously with the GUI installed I would just Team the NIC’s with the HP tool and be on my way… but how the heck was I going to team the NICs when I had no GUI to run the tool from?

Well after some checking around other sites I read that the HP Network Config Utility was installable on Server Core and could be launched, could it be? Plus because I used the Smart Start CD there was no need to go out and download the software and install it because it was already there, I just needed to run the following command:

Then after a short pause I was presented with:

After that it was pretty easy to setup a team with NICs 3 and 4 on the server! After doing that the NIC Team was available to assign to a Virtual Network.

The two problems with this method of teaming the NICs:

  1. There will be finger pointing when it doesn’t work (Microsoft <—> HP) and we all know how well HP NIC teaming software works these days.
  2. The team shows up as an HP Virtual Adapter #1 in the adapter list… there is no way to tell which nics are in the team, this will make management a pain once you have several teams in the same box.

HyperV Posts Coming Soon

Seriously I’m sure your probably like “WTF is he thinking”.

Let me explain.

Obviously if you’re reading my blog you know I’m a VMware guy… but I am determined to keep an open mind and not blackball HyperV  too badly. While I still feel VMware is going to be the leading virtualization platform for the foreseeable future, I also think that anyone working in the reseller environment needs to know a second hypervisor so when VMware doesn’t fit the customer you don’t have to walk away from the deal.

So why HyperV?

Well XenServer would have been a more fitting choice for me since I like Linux, but the amount of development that Citrix is putting into XenServer just doesn’t seem that great anymore. Its like Citrix knows that it’s not win the hypervisor war and instead have focused resources more on XenDesktop as their new flagship product.

Plus the fact that Veeam Backup and Replication will support HyperV in v6 helps a lot… I can keep a familiar backup product and still protect VM’s just as well as on VMware. For me I think HyperV is really going to come in to play in its next release when they can support host to host replication on local storage… combine that with a GUI for the very small business owner and you have a very attractive (almost) HA package.

Stay tuned.