CentOS / RHEL 7 Azure Tips

Possible problems you will face

In my experience, moving a VMware or HyperV virtual machine to Azure can become quite the headache compared to non-RHEL based distributions. When I first started helping customers troubleshoot CentOS 7 we were running into issues with the root filesystem not mounting, NIC IP address assignment, and installing the proper HyperV drivers into the initramfs. Luckily, we were able to work through all the issues and now by following the guidance below, it is pretty easy to get an on-premises CentOS VM into Azure.

Assumptions

This tips and tricks page will assume that you are using Zerto to migrate an on-premises VMware VM into Azure. It is entirely possible that these same steps will apply if you are using a different tool, but I have not tested any other tool and have no intention of doing so.

If you are only looking to use Azure as a Disaster Recovery platform, and plan to continue running your VM’s in VMware or HyperV that is perfectly fine. These steps simply prepare a VM to run in Azure, if that never happens, the modification will remain harmless while running on VMware. It is MUCH easier to do this before a failover than it is after.

I also assume that if you have found this page, then you have probably seen this Microsoft article: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/create-upload-centos

That article is great if you are trying to build your own custom CentOS “golden image” that you can then create templates from in Azure. It is also a HORRIBLE guide for migrating a production VM to Azure, why?, because several of the steps will absolutely thrash a production VM. Most of the steps in this guide were taken from that page, but the steps that were harmful or unneeded were left out.

Tested Versions of Linux

Tested means, the steps in this article were performed on a machine, and then that machine was failed over to Azure (from VMware) with Zerto.

Working:

RedHat Enterprise Linux: 7.0, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8

The Tips

On the production virtual machine, the one running in VMware that you are planning to failover or move to Azure, you will want to do the following steps.

Modify Grub Options

These steps will enable a serial console so if something goes wrong you can leverage the Microsoft Serial Console for troubleshooting. In my opinion, you should NOT skip this step.

User VI or nano to edit /etc/default/grub. Look for the line that starts with “GRUB_CMDLINE_LINUX”

Remove the following options

rhgb quiet crashkernel=auto

These options told grub to use the graphical bootloader and to use verbose kernel output to the console. Microsoft says they are not used for public cloud VMs. Crashkernel consumes about 128MB of ram, it can be left according to Microsoft, but they recommend removing it to save RAM.

Now add these options

rootdelay=300 console=ttyS0 console=tty0 earlyprintk=ttyS0

These options tell grub to wait for 300 seconds for the root volume and turns on both the local console and serial console for boot output messages.

Save and exit your text editor.

Now run this command to update Grub.

grub2-mkconfig -o /boot/grub2/grub.cfg

Adding HyperV drivers to the initramfs

For Linux to work properly in HyperV as well as Azure you need three modules: hv_bus, hv_netvsc, and hv_storvsc.

To install these into a VM that is running on VMware we need to modify the dracut config file and then run dracut to update the initramfs.

Using VI or nano edit /etc/dracut.conf; add the following line into the file then save the file.

add_drivers+=" hv_vmbus hv_netvsc hv_storvsc nvme ena xen_blkfront xen_netfront mptbase mptscsih mptspi "

It is very important to preserve the spaces as well as using the proper ” ” marks. If you use the Azure configuration document the double quotes that they use are interpreted improperly and will cause the drivers to not be loaded.

Now run dracut with the following command.

dracut -force -v

Here is a HUGE initramfs side note/tip too

The dracut command above will rebuild initramfs for the currently running kernel version. HOWEVER! If you ran yum -y update above and it installed a new kernel, and you havent rebooted, dracut will build for the WRONG kernel. (It will build for the currently running kernel… not the new kernel that will be used on the next reboot)

You have two options.

1.) reboot the machine before running the dracut command

2.) tell dracut which version of the kernel to build for

If rebooting is not an option my recommendation is to use cat your grub file to see what version of the kernel are installed.

cat /boot/grub2/grub.cfg | grep initrd16

then run

uname -a

Your output will be something like this:

Output showing kernel versions in grub2 as well as the current running kernel

If the kernel version from the uname command (3.10.0-862.9.1.e17.x86_64 in my case) is also listed as the top line from the first command, dracut will build the initramfs for the proper kernel.

If they are not the same then you can force dracut to build for the default kernel by using this command (modify as needed for your kernel)

dracut -f /boot/initramfs-3.10.0-862.el7.x86_64.img 3.10.0-862.el7.x86_64

If you don’t build for the default kernel, your failover test won’t work as you won’t have the Hyper-V drivers in the default kernel.

Check to see if Hyper-V modules are loaded

If you are able to reboot the VM after running the dracut command to update initramfs, you should then check to see if the Hyper-V modules have been loaded.

lsinitrd | grep hv

You should see something like the following:

All three HyperV kernel modules are now loaded

If those commands worked you should be good to go for failover or migration. I recommend performing a test failover into Azure just to double check, and I would also recommend running a test failover after each kernel upgrade just to make sure all is still working well.

Configuring Networking

Now that we have the drivers loaded into the initramfs, we can focus on network configuration.

Azure (and AWS for that matter) require VMs to get their IP address via a DHCP request. So if you want a static address, you can do that via Zerto’s VPG settings, but the VM will still need to think it is getting a DHCP leased address.

Most VMware VMs will use ensXXX for VMware NICs, you can check what your’s is by typing ifconfig (as root or sudo).

VMware Nic on RHEL 7.4

Azure nics will show up as ethX, most commonly eth0.

So you can either assume that it will always be eth0, or you can modify some network rules to make it something more reliable like eth99 (or whatever you would like).

To do that edit /etc/udev/rules.d/70-persistent-net.rules
(you may need to create this file if it doesnt exist)

Add this line at the bottom of the file:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="hv_netvsc", ATTR{type}=="1", KERNEL=="eth*", NAME="eth99"

What it should look like now (assuming you only had one rule to start):

Mapping Azure nic to eth99

Now we can create an ifcfg-eth99 file and set it up for DHCP.

Creating an ifcfg-eth file for the Azure nic

Create a new file at /etc/sysconfig/network-scripts/ifcfg-eth99

Then add the following to the file:

NAME="eth99"
DEVICE="eth99"
BOOTPROTO=dhcp
ONBOOT=yes

It should look like this:

eth99 config

Enable ssh if it isnt already

In Azure you only have access to a serial console, and via the network. So if you don’t already have the ssh server turned on, I would highly recommend doing so.

service sshd start

chkconfig sshd on

Enabling a Serial Console

RHEL 7 has an example getty file that we can copy over and enable. To do that run the following commands.

cp /usr/lib/systemd/system/[email protected] /etc/systemd/system/[email protected]

Next Symlink it to the systemd “wants” directory

ln -s /etc/systemd/system/[email protected] /etc/systemd/system/getty.target.wants/

Next we will start, and enable the new console. Note, on VMware this won’t do anything, but when you failover to Azure you will be able to use this to login if networking isn’t working.

systemctl daemon-reload
systemctl start [email protected]
systemctl enable [email protected]

Testing in Azure

Once you have your machine prepped, create a VPG to Azure, and then kick off a test failover.

After the VM is listed in your Virtual Machine inventory click it and go to Boot Diagnostics then the Serial Log tab

Azure Serial Log output, available since we turned on the serial console

You can scroll through the serial output, and look for any issues.

Next Steps

You will probably notice that I did not install the WAAgent into the VM, it is technically not needed to get the VM up and running in Azure. Therefore in my opinion, it can be installed later after you have migrated or failed over. For a Zerto customer who only plans to use Azure while their production site is offline, there may be no reason to install it at all.

Comments are enabled, feedback is appreciated. Let me know if your mileage varies.

5 Responses to "CentOS / RHEL 7 Azure Tips"

  1. sorry I put the question in another article it should be here

    hi great article I was doing some failover tets today and we had the CentOs issue not booting

    one question is it OK If I install the VM Agent before the failover os it’s all always there?

    thanks

Leave a Reply