Not Synchronized

[datastream187]

Hi

I have setup a 2 node cluster with version 8 but i am having a problem with the nodes keep going out of sync. For example i started to install a new virtual machine with hyper-v and normally this would take minutes when you are installing via ISO but it took over an hour and every few minutes the event log is showing "not synchronized" then it goes back into sync and this keeps happening.
I have put a section of the log at http://pastebin.com/cDjwd1ev as obviously its quite large
My setup consists of 2x HP DL380 G5's with 2x sync NIC's and 2x ISCSI NIC's There is no switch in between the sync,heartbeat and ISCSI network cards they are just directly linked.

Any help would be appreciated.

Thanks

[Anatoly (staff)]

May I ask you to provide me with next information:
· StarWind logs from all problematic SAN boxes
· Windows Application and System logs (in *.csv format) from all problematic SAN boxes
· Detailed network diagram of SAN system
· Description of the actions that were performed before/at the time of the issue
· Approximate time frames when the issue happened
 
I`d appreciate if you`ll separate the logs from different servers into the different folders

Thank you

[datastream187]

Hi Antony

Sorry its been a while getting back to you, i though that as i had reinstalled starwinds and the HA devices a few times while i was learning it would be better to reinstall from scratch and see if i got the same results, im afraid its exactly the same
I have included the event logs and the starwinds logs in this post, i dont have a detailed diagram, but my setup is just 2xDL380 G5's with 2 extra NIC's in each one. The 2xSync and the 2xISCSI connections are straight from machine to machine with no switch, the heartbeat and vEthernet goes into a switch.
I have turned on 9k Jumbo frames and turned off virtual machine queuing. Other than that its a straght forward setup.
The only thing i have been doing at the times the ISCSI goes offline and unsyn's is either installing a new VM or doing a quick migration of a VM between hosts.

Please let me know if you need any more information.

Data

[Anatoly (staff)]

i dont have a detailed diagram, but my setup is just 2xDL380 G5's with 2 extra NIC's in each one. The 2xSync and the 2xISCSI connections are straight from machine to machine with no switch, the heartbeat and vEthernet goes into a switch.

Well, we still need it. Brief Visio diagram with all the IPs, components and their purpose should be enough.

[datastream187]

Hi

Here is the network Diagram you asked for.

Thanks

[Anatoly (staff)]

What is the throughput of each of your sync channels and Vethernet channels? Try disabling jumbo frames and let me know about the result. Look forward to hearing from you

What about any third-party apps, that can use iscsi/network interfaces, because I have seen mysterious connection drops mentioned in logs?

Try to change SYNC adapter (use broadcom for it).

Also provide me please with benchmarking results of all of your network interfaces. Hardware stability test results (network, local disks, cpu, etc) are highly appriciated.

Just to be sure, create a new one HA storage and using Hyper-V, instead of Failover Clustering, install VM there and give us a feedback about how much it takes. Also I`ll apreciate if you could provide us results about network load and sync drops.

[datastream187]

Hi Anton

Sorry i did not get back to you, i decided the rip the whole thing down again and rebuild, this time i used the "failover only" ISCSI MPIO and not "round robin" which i had been doing in version 6 (full paid for version, and installed in client sites) which worked fine and was the correct way to configure that version.
I setup the test network before the documentation for version 8 had been released so i obviously just used the same that worked with version 6.
Long story short when i configured it with the settings in the "hyper converged hyper-v guide" everything started working correctly. ( until the problems in my other post "storage degradation" )
Thanks very much for the help with this problem.

DataStream

[Anatoly (staff)]

Our pleasure! Let me know if you need anything else please!