DPM Backup of Cluster Shared Volumes Starwind VSS Provider

[jeffhamm]

We are in the process of evaluating the following combination:

- 2 node StarWind HA node, with all targets as HA targets
- Windows Server 2008 R2 Hyper-V and Cluster Shared Volumes
- Microsoft Data Protection Manger 2010

One issue we are trying to address prior to implementing a production system is an interaction between DPM 2010 and Cluster Shared Volumes on the StarWind SAN. By default, anytime DPM does a backup of the VMs, it will put all nodes of the Hyper-V cluster residing on a CSV volume into what is called "Redirected I/O Mode". This results in all Storage traffic being bottled-necked through one node in the cluster during the duration of the entire backup, and having that traffic not go through the iSCSI network but instead across the cluster network.

Microsoft recommends the use of a "Hardware VSS Provider" from the SAN vendor, which should facilitate a "hardware snapshot", which can then be used to do backup, instead of causing all the Hyper-V nodes to use the "Redirected I/O Mode" for the duration of the entire backup.

Here are some questions I have based on reading a little bit through some of the StarWind documentation:

- Does StarWind provide a "VSS Hardware Provider" that can be installed on the Hyper-V hosts?

- If so, once it is installed can DPM 2010 "automatically" create and remove these hardware snapshots on the back-end StarWind SAN for backup operations? We will be doing probably 20 backups of production data daily (just changed bits, not full backups), on almost an hourly basis, so having to manually go into the StarWind console to manually create these snapshots is not a viable option for us.

- When creating an HA target in StarWind, can it be used for snapshots, or am I limited to using the "CDP / Snapshot" target type if I need hardware snapshots? Ideally we will be able to create HA targets that are also capable of snapshots.

Thanks,
Jeff

[Max (staff)]

Unfortunately there is no hardware Provider available yet, however the good news are that we are now working on a new StarWind version which will be able to combine HA and snapshots. This option is really awaited by our clients so we're pushing forward to release it in a few month.

[jeffhamm]

I understand from your post that my backups from DPM will cause the Hyper-V Nodes accessing the same CSV to go in re-directed mode, and there is not much that can be done about it until the next version releases.

Do you know if once the DPM backup completes it will successfully release the reservation so that traffic can again flow directly to the StarWind SAN and not over re-directed mode? I have seen some other posts with folks having issues with the the the Hyper-V nodes going into re-directed mode connecting to the Starwind SAN, and wondered if you have any guidance of how to correct it if a Hyper-V node gets "stuck" in this mode.

Thanks,
Jeff

[anton (staff)]

Unfortunately there's no way to do what you want w/o proper hardware VSS provider (we do use software VSS now and that's the point). Good news - what you want will be possible really soon, hardware VSS development is under way.

P.S. I'm not big fan of what MS does actually in this particular case: during backup system gets really unprotected, vulnerable as I/O is re-directed to one node only. My $0.02 here...

[jeffhamm]

2 Other questions:

1) If I have a total of 5 CSV targets, and only one is being accessed for a vss-based backup, is it safe to assume that the other 4 will not be put into re-directed mode?

2) I also assume this issue affects only Hyper-V/Host based backups of the actual VHD files residing on the CSV, not any VSS-based backups run from inside the virtual machine itself?

Thanks,
Jeff

[anton (staff)]

1) No. Unfortunately there's no guarantee ((

2) Absolutely! Hypervisor holds writes itself and StarWind VSS agent (software) installed as CDP service does the rest.

2 Other questions:

1) If I have a total of 5 CSV targets, and only one is being accessed for a vss-based backup, is it safe to assume that the other 4 will not be put into re-directed mode?

2) I also assume this issue affects only Hyper-V/Host based backups of the actual VHD files residing on the CSV, not any VSS-based backups run from inside the virtual machine itself?

Thanks,
Jeff

[codex]

Has there been any update on development of the Hardware VSS provider for Starwind ?

Just wondering as we are seeing a greater demand for this feature.

[anton (staff)]

Both updated Software VSS and brand new Hardware VSS providers are ready and should be included into V5.8 release scheduled for mid-September. You may wish to participate in our Beta program to get hands over preliminary Hardware VSS builds.

Has there been any update on development of the Hardware VSS provider for Starwind ?

Just wondering as we are seeing a greater demand for this feature.

[Aitor_Ibarra]

My experience of DPM is the version prior to 2010 (2007 IIRC), but I've got a lot experience of Starwind, and Hyper-V clustering with CSV storage, and in my experience, you are generally better off with guest based backup rather than host based backups, unless you are happy losing some of the advantages that CSV and SAN give you.

With DPM or any other host method, even with a hardware VSS provider, you are going to have to avoid mixing the storage of different VMs, or groups of VMs, on the same strorage, as all will be affected by the backup. If that's how you organise your storage (e.g. client A's VMs use this set of CSV's and iSCSI targets) then it shouldn't be an issue. But if you organise storage by performance tier then host based backup becomes a real pain for everyone else not in the backup. Also, if just one of the VMs has trouble getting ready for a host initiated VSS backup, the entire backup will fail. Maybe that's what you want, but somethimes isolating the problem is very difficult.

If you backup within the guest (to another iSCSI target perhaps) then this is not an issue, and every VM can have the backup schedule that suits it best. I have better backup performance and reliability from using free WSB in each VM than I have from DPM on the host.

Hopefully the other problems I had got fixed in DPM 2010 - it used to eat disk space, was incredibly CPU and disk intensive, and always seemed to have a new reason for why the backup had failed! DPM seemed to be intent on sending as little data as possible over the network, which was a nice aim, but meant an incredible amount of disk i/o at both ends with attendant cpu activity while it worked out what it had to send. Even if the delta was tiny it would really stress storage at both ends.

If you use DPM, or any other multi-server package, then usually guest based backup works out more expensive than host based, but generally I think you end up with less headaches.

I am not 100% against host based backup - it can work very well when there are only a few VMs and you aren't clustering Hyper-V - I've been using such a solution to do backups every 30 mins of an offic server for about 2.5 years.

[Constantin (staff)]

I agree with Aittor. Basically backup on VM level or application inside VM, gives you additional level of flexibility to restore your data, including file level restore.

[anton (staff)]

It's a way for a very small shops. Big ones really have to do host-based backups to 1) offload disk I/O as only one CSV snapshot is taken for multiple VMs 2) price "per host" is much cheaper then "per VM" * number of VMs and 3) it's easier to configure one place backup then to run thru hundreds of VMs for every single change in backup scenario.

[codex]

I agree with Anton on this.

In a small setup, the impact of using serialized backups via software VSS provider is not really an major issue, as soon as you scale up this becomes a major factor in the decision to purchase one piece of hardware/software vs another.

The nodes will only remain in redireced I/O for the duration of the hardware snapshot process, which is more than likely going to take less time than the guest based backup of each system using software vss provider in the O/S as the backups will be peformed in parallel rather than serial so the nodes are going to spend less time in redirected i/o overall.

Guest based backups as Constantin says are useful as you can backup workloads running within the VM's - i.e. SQL, Exchange etc so you will want some agents within specific guests running workloads so you can restore the workload rather than the entire machine (why nuke from orbit when a surgical strike is enough?), restoration of generic files from a host based backup can be accomplished with Item Level Restore so again a mix of both host based and guest based will be required I would guess in most environments.

Ultimately I'm happy as we have had a great experience with Starwind, the issues we have come across have 99% been configuration issues (usually end user tweaking something they shouldn't be or turning something off they shouldn't!) and if the hardware vss provider does what it says on the tin, then that just strengthens the product even more.

Is there any timeline for when the 5.8 beta will be available ?

Codex

[Constantin (staff)]

I think that beta will be available at middle of September.

[hixont]

Unfortunately host based backups do not work in all scenarios. Backing up a Sharepoint server farm comes to mind and represents my current top priority. None of my existing host based backup solutions can manage brick level backup/restore of the SharePoint data in a distributed environment. I do not want to have to restore entire systems just to export out the file level data to the production SharePoint farm. DPM 2010 will backup distributed SharePoint environments.

[anton (staff)]

There's no "win-win" scenario. Another example is - SQL backup and Exchange backup. People don't need snapshots for them, they want to recover per-transaction and per-mail object.

Unfortunately host based backups do not work in all scenarios. Backing up a Sharepoint server farm comes to mind and represents my current top priority. None of my existing host based backup solutions can manage brick level backup/restore of the SharePoint data in a distributed environment. I do not want to have to restore a entire systems just to export out the file level data to the production SharePoint farm. DPM 2010 will backup distributed SharePoint environments.