Creating a proper Zerto Failover Test Network is crucial for testing your replicated applications with Zerto Virtual Replication. The reason it is so crucial is because in order to test things without affecting production, or making Active Directory really grumpy, we need to make sure that things are isolated. The problem with isolation however, is that it makes it really hard to have people who know the applications test them.
So the idea behind this post is to present suggestions on how to design a Failover Test Network so that you can maximize your ability to do DR testing with actual application owners while staying safe and not causing problems on your network. Each of the phases listed below will build upon the phase before it, so you can stop at whatever point you are comfortable with or proceed to the next phase for more advanced testing ability.
Phase 1 – Simple but isolated
The most simple method of creating a Failover Test Network is to create a VMware portgroup on each of your VMware hosts (at the recovery site) that is attached to some sort of isolated switch. I say some sort of isolated switch because we could have a dedicated switch with dedicated NIC ports on each host, but this doesn’t scale very well and eventually would be hard to manage. The better way to do it is to use a VLAN that can ride on the NIC ports already connected to your hosts.
It would look something like this:
What I have here is my normal VMware vSwitch with two network ports in trunk mode back to the switch. My management / production VLAN is VLAN 10 and I have also created a portgroup for VLAN 11 for my Failover Test Network. VLAN 11 exists on my switches but is isolated (meaning no SVI and no routing in or out of it).
So what you need to do is:
- Create a VLAN on your switches that attach to your VMware hosts, allow that VLAN to ride on the trunk ports to your ESX hosts
- Create a portgroup in VMware, name it something that indicates it is a Failover Test network, tag it with the proper VLAN number that you used on your switches.
At this point we can use “FOT Network” inside of Zerto and failover VM’s into that port group. This is the most basic Failover Test Network, it works well if your testing requirements are minimal.
The reason that this only works for minimal testing is because in order to test the applications inside of this FOT network you have to use the console of the VM in order to access it. So anyone who does not have access to the vSphere console won’t have a way to test things.
Phase 2 – Less Isolated
Now that we have a VLAN on our switches and have a working (but isolated) Failover Test network, let’s move on to getting access into that network for the people who would need to test the applications that we failover.
The easiest way to do this is to create a terminal server with two network cards. In VMware this is simple, just use whatever OS you would like (server 2012 is what I’ll work with) and then place one NIC in your normal VMware management port group with a static IP address and put the other NIC in the Failover Test Network with another static IP address.
As you can see the FOT-TermServer is in both the VM Network and the FOT Network. It has the following Network settings:
VM Network IP: 192.168.10.200/24
VM Network Gateway: 192.168.10.1
FOT Network IP: 172.16.1.200/24
FOT Network Gateway: NONE!
It should also be noted that this VM is not joined to any domain. This helps to ensure that it doesn’t get confused when we have a domain controller in the FOT Network as well as our production domain controller on the VM Network.
Operation inside of Zerto remains the same, we simply do a test failover of our VPG’s and they are assigned to the FOT Network. Once there we can allow our testers (the application owners) to RDP into the FOT-TermServer and interact with the VM’s that are running in the test network. Depending on your application settings some people find it easier to create a static hosts file on the terminal server so that people can use the familiar DNS names that they are used to on the real network.
At this point we can install any tools or client applications that might be needed in order to test things on the FOT-TermServer… think of it as your application owners desktop… anything you would need to do your job gets put on this desktop.
Phase 3 – Access from the outside world
The last thing we can add to our Failover Test Network is internet access. Most applications will not need Internet access but in some cases you may want to download application updates, or maybe your application needs access to a resource that is on the internet. There are a couple different ways to accomplish this, but essentially what we want to do is provide a path to the internet while blocking any access to the corporate network. You can accomplish this through some creative firewall rules if you have a network team, or if you have an extra IP address from your ISP a virtual firewall is probably the safer bet.
What I have done is create a virtual firewall who’s LAN interface is attached to the FOT Network (with a static IP address of 172.16.1.1 /24), and who’s WAN interface is attached to a portgroup that gets a public IP address (in my case 188.8.131.52). As you can see below, the FOT-Firewall can now become the default gateway for any VM in the FOT Network, and it will provide NAT out to the internet as if the FOT Network was a completely independent company. Now I can download updates, access resources on the Internet, and even forward ports to internal services for test access from the internet.
Additional Tips for FOT Network Success
- Active Directory in your FOT Network – Most people will require Active Directory for their testing purposes simply because so much stuff relies on it these days. Normally you would not want to fail over a domain controller for production use because AD is grumpy to say the least, so you would want to build a domain controller at your DR site and have it fully operational and natively replicating with other AD servers (outside of Zerto… IE use native AD replication). However you can’t simply put that domain controller into your FOT Network as you would confuse it, so the best way to get Active Directory into your FOT network is to build a VPG and protect one of your production domain controllers and fail it over when you test your other VPGs. Don’t panic if you have already used all of your Zerto licenses and don’t have an extra one to use on an AD server as there is a grace period where you can protect an extra machine on top of what you are licensed for in Zerto. However if you plan to test often, purchasing an extra license to protect an AD server is probably the best idea.
- Failover tests can’t run forever – Keep this in mind when doing your testing. Two things can happen if you leave a failover test run too long, the first is that your journal will grow and grow and not be able to remove the old data until the test is complete. This is due to the fact that it might be in use by the test. Second – your VM can generate so much “scratch” data that eventually there is no more room to write scratch data, this will eventually cause the VM to lock up. Remember that both of these take a really long time…but irreguardless you don’t want to leave a Failover test run forever. If you need a test long-term you are better off to do a clone of the VM and then it will be outside of Zerto completely, but could still be attached tot he FOT Network manually.
Disclaimer – I work for Zerto, however this is a personal blog. Zerto was not given prior notice of this post or the chance to review its content. Therefore Zerto is not responsible for the content or views expressed in this post. Any and all questions should be posted as comments to this post and will be answered by the author as soon as possible.
Pingback: Tech Field Day 11 - the presenters part 1 #TFD11
Thanks for this. I’m in the process of setting this exact thing up. We envision using this method to both verify our failover capability, and to do routine upgrade testing for protected VMs. I do have a question though: Do you use Zerto to assign new IP addresses (safer but more inconvenient), or do you bring them into the test network with their IP addresses intact?
Yes, thank you for this great resource. We are setting up this test now before our Zerto goes live. To clarify something about the non routed FOT network, when you say ‘switches’, are you referring only to the VC switches or the top of rack switches that the uplink sets connect to in order to make a vlan tunnel?
I’ve seen customers do it either way. It just depends on how you want to use the FOT network, but there is really no wrong answer.
I was referring to your physical switches. Although, your VMware port groups will need to be tagged as well if you are doing trunking from ESX to the switch.
Basically you will want the VLAN’s on your TOR switches as well otherwise a VM that is recovered to host 1 wont be able to talk to a recovered VM on host 2.