loss of connectivity and data corruption

[boris]

Good day!

My config: 2 nodes running WS2008 R2 SP1 and SW HA 4Tb (ver. 6.0.4768).

I did a little test on HA device: disabled heartbeat channels first and sync channels second - both nodes told they are synchronized and partner is not. Then did some writes to each node (using its target), then brought network back. Autosync happened and after a period of time I got corrupted file system on both nodes.
I think actually no sync was done, cause I seen different set of files on nodes when switching between targets.

Questions are:
1. If both nodes stated they are synchronized, how the sync source could be chosen (and was there a synchronization actually)?
2. Why did autosync took place in such a situation - thats obviuos if both nodes think they are synced and partner is not it is 99% probability of data corruption. IMHO immediate cease of all operations on both nodes is the best choise in this scenario. At least as an option that I could set for device.
3. How do I disable autosync for device (can't find an option nor when creating device neither for created one). I would prefere to do sync manually - inconvinient, but prevents data loss.

Some real-world situations when this is critical:
1. Updating NIC drivers causes momentary loss of connectivity (maybe not all the NICs, but intel cards are for sure). If, by accident, starwind service not stopped before update - just forgot to, inexpirienced administrator, etc - situation I described above is reproduced. Though a small period of time, that can be enough to damage critical data.
2. I have second node installed in our partner's server room - this is by design: second storage node must be in a separate room (corporate policy). That room is governed by other organization and I can't garanty no one will disconnect my server (e.g. during network maintance). That's a really small, but real chance.

Best regards!

[anton (staff)]

First of all you've misconfigured your installation... For safety reasons you need to run heartbeat channel over the network you route your client iSCSI traffic.
So if there's no network - no heartbeat but clients cannot spoil anything as there's no incoming requests from them. Of course you can connect to the target
in a loopback and ruin the data but it's definitely not something could happen in the real life

Few question so far:

1) Do you want to "force sync" the targets?

2) Do you want us to make something like "M" token assigned to one of the nodes so if we'll have potential brain split issue we'll stop the cluster (or continue working if "M" node is alive?)

3) You cannot now but I think we'll have to add this option.

Your feedback is appreciated. You have a great chance to get exactly the solution you want

[boris]

Thanks for reply!

Ok, the whole story is: we bought a licence two years ago to run it in 2-nodes cluster with SAN config. Sales told this is possible and it's actually is, except the fact that in split brain scenario VMs on each node still able to connect to targets on that node.
There was no "native SAN for Hyper-V" that time - that is something that fits our layout best - and no downgrade is available.
I first rannt into split brain issue two years ago during first attempt to deploy (with SW 4th version as far as I can remind) and two years licence was lying on the shelf. Not only because this issue - there was other reasons to delay deployment.
Now tried over with ver.6, but no luck.

So what I actually looking for is how "native SAN for Hyper-V" protects against split brain and reproduce same config with HA edition if possible.
As timely solution, I'm thinking about L3 switch, which will staticaly route some IPs that not assigned to nodes themselfs back to nodes and direct "client" iSCSI traffic to that IPs - so that all connections will go thru wires and if switch fails, nodes can't connect to themselfs thru hyper-v virtual switch.

And yes - detect potential split brain issue and stop the cluster is a good choise IMHO, as an option at least. Think about same situation with upgrading NIC drivers or NIC drivers fault and restart. Additional reliability is always a good thing.

[anton (staff)]

At this moment I'm not talking about Native SAN or whatever, I'm about running StarWind on a dedicated hardware. The whole setup for NS is a bit more
complicated as you cannot isolate initiators - they connect in a loopback. The way to go is - use multiple sync channels (at least two) and multiple h/b channels
(sync channels can be used for h/b). If you need to be 200% sure no SB is possible and you cannot guarantee you run redundant sync & h/b channels -
use active-passive cluster for Hyper-V and failover mode for Native SAN but performance will suffer from this of course...

Configurations and optimal network settings for NS and core product are DIFFERENT.

For NIC drivers (or any other kind of an upgrade) the scenario is:

1) Put down StarWind
2) Upgrade your software / hardware
3) Reboot
4) Re-sync StarWind with alive nodes
5) Continue working

Thanks for reply!

Ok, the whole story is: we bought a licence two years ago to run it in 2-nodes cluster with SAN config. Sales told this is possible and it's actually is, except the fact that in split brain scenario VMs on each node still able to connect to targets on that node.
There was no "native SAN for Hyper-V" that time - that is something that fits our layout best - and no downgrade is available.
I first rannt into split brain issue two years ago during first attempt to deploy (with SW 4th version as far as I can remind) and two years licence was lying on the shelf. Not only because this issue - there was other reasons to delay deployment.
Now tried over with ver.6, but no luck.

So what I actually looking for is how "native SAN for Hyper-V" protects against split brain and reproduce same config with HA edition if possible.
As timely solution, I'm thinking about L3 switch, which will staticaly route some IPs that not assigned to nodes themselfs back to nodes and direct "client" iSCSI traffic to that IPs - so that all connections will go thru wires and if switch fails, nodes can't connect to themselfs thru hyper-v virtual switch.

And yes - detect potential split brain issue and stop the cluster is a good choise IMHO, as an option at least. Think about same situation with upgrading NIC drivers or NIC drivers fault and restart. Additional reliability is always a good thing.