Are you using HP StoreOnce VSA? I am!

One of the guys I’ve known for a long time is an engineer / IT guy / jack of all trades for a fairly small SMB (in terms of IT needs). They only have a handful of VM’s and a dataset size of about 500GB of production data. So when I get asked to give advise one of the biggest problems I have is that I’m thinking WAY to big, for example could I really tell him to buy a Data Domain or a StoreOnce (physical appliance) that holds several TB of backups physically, and logically scales to hundred of TB’s? Probably not… even if it were in budget it would still be a huge waste!

So when asked how we could do some offsite backups while keeping a budget in mind, I remembered that HP was allowing production use on their 1TB free HP StoreOnce VSA! Having worked for an EMC/Cisco/VMware reseller for a long time I haven’t had a chance to install StoreOnce in any capacity, so this would be my first encounter with real data on StoreOnce. (I had deployed it in my lab a couple of times but I had never used replication or any of the features more than a week at a time, plus lab data gives no really measure of dedupe capability.)

Before StoreOnce

Before StoreOnce all backup data was stored on a P4000 2 node SAN along with all of the production data. Production data was taking up about 500GB and Veeam backup data was taking up about 800GB for 3 weeks of retention. Aside from the P4000, he would also copy the latest backups to an external USB drive so that he had some sort of DR plan.

At one point Rick over at Veeam let me play around with a Veeam Cloud Provider license and we tried that out with this SMB’s data by replicating to my colo. It works pretty well, no issue with the technology, but there really just wasn’t enough bandwidth to push the nightly change data to my colo without also running into production hours. (Nightly change from Veeam to disk is about 12GB, and the company’s upload rate is about 2Mbps).

After StoreOnce

So let me start by saying this is a work in progress, but honestly I’m really excited about StoreOnce so I didn’t want to wait a month or more before writing this post… So it will get updated as I have more data.

The first thing I did was deploy a StoreOnce VSA on their VMware cluster, the licensed it… which was a bit of a pain in the ass… So one recommendation I would make is add a spot for licensing to the GUI. Don’t get me wrong I’m not afraid of the command line, but for the normal SMB customer who would be the target for a self install VSA… yeah GUI would be better. After that I added a 1TB VMDK to the VM and powered it on. Initial startup and install takes about 10 minutes, but its all hands off, basically you just need to sit and wait for the login prompt.

Veeam V8 has support for StoreOnce as a Dedupe appliance, although right now it doesn’t really take advantage of Catalyst but I’ve heard it will in V9. I then created a clone of all the backup jobs and repointed them at the StoreOnce Backup Repository and left them to do their thing. Lastly before I logged out for the night I also started a copy of all the backup retention to the StoreOnce VSA… about 3 weeks of backups…. 800GB of raw disk space.

The next morning I checked the StoreOnce interface to see how much dedupe had been achieved and I was impressed to say the least!

day 2

4.34:1

Thats pretty Impressive considering Veeam already did it’s dedupe and compression on the data before it landed on the StoreOnce!

After a couple more days of backups we are still at a 4.3:1 dedupe rate only having added about 3GB of unique data.

day 3 dedupe

 

So here is what I have seen so far in terms of StoreOnce’s ability to compress and dedupe on top of Veeam:

Day 2: Veeam sent 12,746 MB of data to StoreOnce; StoreOnce “data on disk” size increased by 1GB (a 12:1 savings)

Day 3: Veeam sent 12,792MB of data to StoreOnce; StoreOnce “data on disk” size increased by 2GB (a 6:1 savings)

Day 4: Veeam sent 12,160MB of data to StoreOnce; StoreOnce “data on disk” size increased by 1GB (a 12:1 savings)

Pretty impressive!

Replication

So I mentioned that offsite backups were the goal here… something that didn’t require user intervention was really the big thing. So to make this happen I created a VPN from the company’s Fortigate to a VPN endpoint on my colo gear and then deployed a StoreOnce VSA there, the same way I deployed one on site.

Configuring replication was pretty easy after the VPN was up.

Source appliance was on the 192.168.3.x subnet and the DR appliance was on the 192.168.13.x subnet. StoreOnce has a really easy wizard for replication. I simply went into the replication area, clicked the share I wanted to replication and started the wizard. I had to enter the IP/Hostname of the DR appliance and then create a share to replciate to… which was all handled by the wizard.

Because of the 2Mbps WAN connection, and my being to lazy to drive an hour away to seed the data, I simply set a 1Mbps cap and configured StoreOnce to only upload with 2 “slots”. (each slot wants 512Kbps minimum). Inserting traffic graph just to add color 🙂 … I guess you could say that the throttle works as advertised too I guess.

traffic

I estimate that it will probably take about 3 weeks to get in sync, I guess I should have done a seed, but honestly I’m more interested to see how it can handle a slow connection.

On a side note

While setting up this pair for my friend I also though, “shouldn’t I also be doing offsite backup of my data”? Lately I have been slacking… if the colo I use were to “go away” my blog would be in trouble. But setting up a StoreOnce VSA pair and replicating back to my home lab didn’t take long at all. The Veeam backup of my blog is about 10GB after it lands on StoreOnce. The first night (on my 3Mbps download crap connection) it took StoreOnce about 8-9 hours. On night two it only took about 2 hours, but as you can see it wasn’t maxing out my connection. BTW night two had 616MB of data send to StoreOnce but I honestly don’t even see a bump in the “on disk” storage LOL… That’s awesome!

home graph

More to come as it continues to chug away…but in the mean time would love to hear your StoreOnce stories if you are using it.

Update: 9/21/2015

It’s been about 3 weeks since I implemented StoreOnce VSA so I thought I would share how it has been doing so far.

Capacity

Below is a spreadsheet I’ve been keeping relating Veeam backup file sizes to disk growth on the StoreOnce appliance. I’m keeping track of these simply to show how much disk and bandwidth savings can be expected compared to storing and replicating Veeam files on their own.

21 days

As you can see we are up to a 9.5:1 dedupe rate and have only consumed about 25% of the free StoreOnce VSA’s capacity. At this point we have over 5 weeks of backups on disk, and based on the rate of growth I would say a full year of backups would be pretty conceivable. However I will most likely roll to a G-F-S hierarchy once I hit 60 daily’s.

Backup Job Length

While the StoreOnce VSA’s are doing their initial backups I have also been letting the old backup jobs run, which go straight to disk. Backup times average about 1 minute per job longer on incremental backup days for StoreOnce jobs, full backup days are a little harder to judge because I’m using synthetic full backups to normal disk, and I’m using Active Full Backups each week on the StoreOnce jobs.

Restore Times

Deduplication and compression are both CPU and memory intensive tasks, and therefore most of the time when you add either or both to the mix the time it takes to do those processes increases processing time. Basically the thought is that storage is expensive and CPU and Memory are “cheaper”. With that said most of the time when doing restores from dedupe appliances we will see longer restore times than if we were pulling straight from raw disk.

With that said these test results were pretty surprising to say the least…

Test Parameters

  • Same virtual machine
  • Data Size: 60GB
  • Test 1 was with backup files located on the Veeam’s “d” drive… which is an RDM from a 2 node Hp P4000 iSCSI array.
  • Test 2 was with backup files located inside of StoreOnce which is backended by a VMDK on the same HP P4000 iSCSI array.

The time to restore from the Raw RDM inside of Veeam was 11 minutes and 12 seconds (for just the VMDK).

sage no storeonce test restore

The time to restore from the StoreOnce VSA backups was 10 minutes and 23 seconds (for just the VMDK).

sage storeonce test restore

So in this case a restore of a real VM actually took 49 seconds LESS!

Obviously to be 100% sure I would need to test some more VM’s and repeat over multiple days…But honestly I wasn’t looking for exact numbers… Just knowing that it is pretty much the same is good enough for me!

Share This Post

14 Responses to "Are you using HP StoreOnce VSA? I am!"

  1. Thanks for starting this review. It would be great if you can cover how Veeam’s Surebackup jobs perform against the StoreOnce repository and restore performance. Both critical components for us.

  2. Doing those tests with StoreOnce VSA would not be an accurate test in any sense… as the underlying hardware could be a million different combinations. IF you want to see that, I would reach out to your HP rep and ask for a try and buy box.

  3. Justin,

    Let me first say, I am a huge fan of your site and I appreciate the time and effort behind your very thorough post. I just finished an HPE class on StoreOnce and used your other blog (great note about the virtual machine gotcha) post to setup my own VTL with Veeam 8 in my home lab. Everything is working flawlessly and I would like to take it to the next level with a “colo” setup using a VTL on the other side. With regard to replication, does VTL offer any type of WAN acceleration, or are you utilizing any other product to optimize the bandwidth across the WAN? I know Veeam offers WAN acceleration OOTB, but I thought that what was only if you replicating to “cloud/paid” backend services. On another note, what tool/app/program are you using to measure your bandwidth utilization (pretty graphs)? It looks similar to MRTG, but I don’t know too many people that still use that program anymore.

    Thanks
    MG

  4. Hey Michael,

    I dont use any wan acceleration other than the built in reductions gained from replicating from StoreOnce to StoreOnce… this basically means that the data is deduped before its sent…. i dont use Veeam to replication over the WAN because its session based dedupe not global dedupe.

    I utilize pfSense firewall’s pretty heavily in my lab and at my colo… it allows me to backup the config easier and keep different workloads seperate and it allows me to keep my ASA config very simple (just for managment). So the graphs that I was showing were RRDTool graphs generated by pfSense.

  5. Justin,

    Thanks for the quick turn. I must say, you definitely have my hamster running of his wheel with regard to standing up a colo with StorOnce VSA. I think I will start brainstorming the data flow so I can avoid any issues when I start building VM’s on the other side.

    I don’t any experience with pfSense firewall, but I know a couple of my buddies swear by it. I am currently using Sophos UTM Home Edition as my firewall and I must say, it’s pretty robust when it comes to features. I use a Cisco 1941 to terminate VPN’s and serve as core/distro router for my home network. Might have to spin up a pfSense VM to compare the two.

    Thanks Again!
    –MG

  6. I have the storeonce vsa downloaded and installed but i am in a fight with HP about the license key.
    They keep sending me away and hence i have a 60 day instant-on license but no matter what i say to them or show in the form of printscreen they keep claiming that i allready have a license
    How did you solve that for the 1 TB free license key ??

  7. Hi Justin – great post. What versions of the HP VSA and Veeam were you using?

    I have the latest Veeam v9 U1 and HP VSA (on Hyper-V) but keep encountering issues whereby Veeam will has errors when trying to catalog the media. Is there any gotchas that I need to look out for?

  8. Hey Don,

    The only thing I have ran into is when Im trying to keep really high restore point counts… when i get close to 120 or so Veeam file level recovery times out when trying to do restores.

  9. Hi Justin, this is a really good post… but i’m worried about something… we have acquired a StoreOnce VSA and our “specialist” is requesting that we indicate the ports to be opened for replication. I checked the manuals and an lot of information online, but I couldn’t find any document specifying the ports used for replication. In addition he asked me, whether it supports nat or not.
    can you give any advice about this?

  10. As far as I know NAT is not a problem, but Im wondering how he plans to set it up. Generally the traffic isnt encrypted so replication traffic should be tunneled through a VPN or private line to a remote site… Most of the time I seee customer just having routers between sites and not NAT… sometimes i run into it though… anyhow, your best bet would be to give HP a call on the ports… or open everything on the firewall and use a traffic sniffer to see what ports its using.

Leave a Reply