Recently while working with a customer we had to get a little creative with an installation and I thought I would share the experience.
Here is a little background info…
Datacenter and Hardware Layout:
- Customer has 2 datacenter: one on the east coast, one in the midwest
- Customer has vSphere clusters at both datacenters
- Customer has purchased one new Data Domain 2500 for each datacenter
- Customer has purchased Veeam licensing for each datacenter
- Both vSphere Clusters run some production workloads
- Replication pipe between sites is 1Gbps with triple redundancy
- Cross site replication of backups between datacenters
- Ability to quickly restore a backup from datacenter A at datacenter B (and vise versa)
- Ability to do restores locally at either site without connectivity to the other datacenter
- Must use DD Boost to increase backup speed
Given this situation we had a few options:
- Install a “master” Veeam server at one location or the other and then create several proxies at each location
- Install a “master” Veeam server at each location as well as some proxies
There are advantages and disadvantages to each setup. And we actually tried both. The only real advantage to having a single veeam server is that all jobs were in one pane of glass and all restores could be started from one area. The disadvantages were much more obvious.
What we found is that because the file level restore happen on the “master” Veeam server it took incredibly long to mount backup images from the east coast Data Domain to our midwest Veeam Server. We were getting about 50Mbps on the wire while the mount operation happened… but from the local data domain we were getting almost 300Mbps.
Now for full VM restores I don’t think it would be quite as bad because we can specify a proxy at the remote site to do the full vmdk restore in hotadd mode… that will keep all the data local and it.
Because we had Data Domain replication in place I decided to see if there would be a way to mount the replicated copy in read only mode to the local Veeam “master” server. The idea is pretty simple… Use a Veeam “master” server at each site for local backup and restore as well as ALWAYS have restore’s pulling data from their local site’s data domain regardless of where the VM was originally backed up.
So here is how it works:
East Coast Datacenter:
Local Veeam “master” server does backups via DDBoost to its local Data Domain in an MTree called “veeam-east”. That MTree then uses native Data Domain replication to the Mid-West Data Domain to an MTree called “veeam-east-replica”.
Local Veeam “master” server does backups via DDBoost to its local Data Domain in an MTree called “veeam-midwest”. Then that MTree is cross replicated to the east coast DD MTree called “veeam-midwest-replica”.
So that takes care of backing up locally, and getting offsite replication… but the cool part is how we can do restores of data from the opposite datacenter…
The replicated MTree’s are placed into a read only mode by default. However! the Data Domain box WILL still let you share that read only folder via CIFS or NFS. Sooo I thought why not just share out the replicated MTree’s via CIFS and have the Veeam “master” mount that share. Sure enough this worked.
So the east coast Veeam “master” server has three backup repositories:
- the default backup repository
- one mounted via DDBoost integration called “Veeam-east”
- one mounted via CIFS called “Veeam-Midwest-Replica”
The midwest Veeam “master” has the mirror of this:
- a default repo
- one DDBoost integrated share called “Veeam-midwest”
- one CIFS mounted repo called “Veeam-east-replica”.
There is only one caveat to this setup. Because Veeam is not actively writing backup data to the CIFS read only share, it does not automatically import new backups that have been replicated. So you do have to preform a manual “Re-Scan” of the CIFS repo so that it pulls in the latest restore points. (However this can be automated with this powershell script http://helpcenter.veeam.com/backup/80/powershell/sync-vbrbackuprepository.html).
So here is what things look like: (click for pdf version)
Basically what I have tried to do here is show the backup data path. You will notice a line with arrows at each end between the local Veeam master server and its DDBoost repository, meaning primary backup and restore data flows between them, then a directional arrow between the primary MTree and the replicated MTree showing the direction of DD replication. Lastly is a dashed directional line showing that “if” a restore of data needs to be done at the opposite site, the local Veeam server has a read only copy from the replicated DD MTree.
I should mention that the only reason that you would want to use the Read only copy to do a restore is if your primary site if offline and you need your data back. I didn’t envision this to be a replacement for datacenter migrations… however with some more testing it may actually be a decent option.
After seeing this particular customers requirements I do see some weakness in Veeam for a multisite deployment. But I’m not sure how many other backup products would really handle it any better. My suggestions would be to somehow use Veeam Enterprise Manager as a “vCenter Linked Mode” type controller. Combine that with DDBoost multisite replication (meaning that because of DDBoost Veeam could control the replication from one site to another at the DD level) and you could then create a multisite backup catalog…. then ideally all Veeam servers would know where all copies of the data are and be able to restore from them without any special hack like the one above.
Other advantages of this would be that restores during a disaster could be faster provided that all backup catalog data was replicated like active directory data to the other Veeam Master servers …
Great write up… Out of curiosity, why the customer didn’t think of Streached ESXi Cluster..
very nice article. I have one question:
Did you ever try an instant recovery of a vm backed up to a DataDomain using vPower NFS?
And one thing:
We had some representatives from Veeam a couple of weeks ago in our company. They showed us what´s going to come with Veeam 9 at the end of this year (should be available at the end of Q3). Just a few things coming:
Veeam will be multi site configurable. That means, that the remote proxies can handle jobs and even logins individually at the site without the master server. Even multi tenancy support.
Veeam (should) support more DDBOOST features like BOOST replication and virtual synthetic fulls.
Of course it´s not yet offical and considering my experience in the past with breaking news from Veeam and getting really disappointed by the reality later… Well let´s see what will be implemented in the v9.0 in a few months.
Brilliant write up Justin! I liked the one specially on Veeam having a multisite backup catalog. Never crossed my mind. Thats a really good thing to have!
Yes, It can be slow… you have to make sure your DD infrastructure is up to the task.
Thanks for sharing the updates.
Pingback: Reflections from the First Distribution Lab | First Distribution - Helping our customers to help their customers
Excellent article on the integration part Justin. We have almost a similar requirement currently with veeam 9 & data domain dd-boost replication. I am going to try this out.
You might want to check with the Veeam Team but I think they have some extra integration with DDBoost in V9 that might not require this anymore….
As at Veeam v9.0 update 1, there still isn’t Veeam managed ddboost replication between data domain storage units. Maybe it might make it into v9.5 ?
Whilst there are some advantages to having Veeam managing offsite copies, the DataDomain MTree replication is much more efficient over WAN links.
This is old, but accessing a DDboost MTree via CIFS is not supported and can cause corruption.
Can we acess a boost Mtree via CIFS share is that supported? How will you achieve the same thing with Veeam 9 if accessing boost Mtree via CIFS is not supported?
That is probably a question for Tim Smith… I haven’t installed Veeam + DD in a long time… check out his blog at http://tsmith.co or shoot him a message on twitter @tsmith_co