Starwind HA demo

[matty-ct]

Hi all,

I've been demoing Starwind HA in the "lab" in order to be better able to pitch it to my clients. I ran into a problem today while simulating an iSCSI network failure. Here's what I have:

2 dual quad core servers running W2K8 Enterprise R2

cluster node1 runs starwind ha (primary)
cluster node2 runs starwind ha partner (secondary)

test Hyper-V R2 VM running on Node2 with primary starwind storage on node1

Now, the cluster is setup well with CSV, etc. I created a test VM in Hyper-V R2 and had great fun using Live Migration to fail back and forth. I was very proud of myself. Next, I wanted to see what would happen if the iSCSI network had an issue. So, on node1 (primary ha) I disabled the NIC for iSCSI communication. The VM survived it seems. However when I went to syncronize the HA images, something went wrong. I did a full sync on the quorum img which took no time and came back up in cluster manager. The storage img is 600gb (really only 2gb of actual data) and looks like it will take a few hours to come back up as the sync is only 5% through after 20 minutes or so. In cluster manager the volume is marked as failed upon resync.

Questions:

1.) Should sync take 4-5 hours?
2.) Should sync (resync) kill the cluster storage availability?
3.) Did I do something wrong in my config or simulation?

I am using 4 Intel 1GB server NICs in each server. One for iSCSI only, one for VM-LAN, one for VM-DMZ, and one for server nodes. Advice needed. I'll freely admit I'm an iSCSI noob. I would think that the point of HA is to eliminate iSCSI downtime either on failure or rebuild. I assume I did something wrong.

Thanks for any pointers or advice,

Matt

[Aitor_Ibarra]

Hi Matt,

you don't say which version of Starwind you are using. I'm on the latest beta of 5.5(and using it in what I will call semi-production until it's released) - so our experiences may be different.

A full sync copies the entire img, doesn't matter if it's mostly empty, and 600GB over 1GbitE should take almost 2 hours at 100MB/sec, if your drives are capable of that. I recently had to sync 2TB on drives that could manage about 50MB/sec sustained, and yes, it took about 11 hours.

Fast sync is much better, just copies the changes, but won't be possible on targets with write cache, or when BOTH starwind nodes are out of sync.

Whether you lose the target or not (from the initiator point of view) hinges on whether Starwind thinks at least one of the nodes is synchronised. If both are not synchronised, then you have to force a full sync and the target will be offline until sync is complete. This can happen if there is a comms failure between the nodes - putting the sync network on a private network helps and putting the heartbeat on the same network as the initiators or ideally a third seperate network helps avoid this.

hope this helps,

Aitor

[matty-ct]

Thanks for the reply. I'm demoing 5.4. RAID 10 primary, RAID5 on secondary. Drives are 1tb 7200rpm SATA enterprise models.Therefore, I should try to make the sync network as fast as possible between the two Starwind partners. I haven't tinkered with the TCP/IP or Ethernet settings yet. I suppose that teaming two 1GB adapters would be a big improvement, or consider dropping the cash for 10GB nics, or using smaller targets. I just discovered that one of the NICs wasn't configured for Jumbo frames. Great, not looking forward to adjusting settings and resyncing again...

Here's how the network is configured in my lab:

Netgear GS724T switch, with cluster node LAN addresses and VM traffic

3Com 5500G Switch for heartbeat/iscsi traffic only.

Before I can pitch the solution I need to figure out a way to significantly speed up resync in case of disconnection.

One more question, if network interruption is anticipated, is there anything which can be done to prevent the 3-6 hour downtime required for resyncing?

[Aitor_Ibarra]

Take a look at the 5.5 beta. It can auto-resync any HA target, most of the time. If you use no cache, or Write Through cache, then it will auto-resync using fast sync, where just the changes since the outage get copied over. Write Back needs a full sync which takes longer. In my tests it's been pretty resilient against comms failures (this is actually harder to protect against, because both servers would be running ok, and one would have to stop to prevent data corruption), but stuff like total server failure is well protected against. I ran several tests where each node was rebooted once an hour for a number of days, with constant i/o activity to the target, and it was fine.

If you anticipate network interruption just shut down the node that will be cut off. When you have your network back, bring it back up again.

[anton (staff)]

Tiny update: we've added ability to do fast sync with Write-Back Cache as well. So more reasons to get hands over V5.5 Beta