Sync Performance

[Caillin]

I'm not sure if I should throw all my different question in one thread, but as there wasn't a definitive thread that I could find regarding my issue, I thought I'd throw it up as a separate one.

We've got our new portential production starwind environment built and configured and currently doing benchmarking and testing. What I'm finding though, is that I can't get the sync channels to use more than 15% of the available bandwidth when doing a full sync. I've tried changing the sync prioritization the next notch up to full, but it doesn't seem to make a difference.

I've performed full network testing of the 1GBE iSCSI channels and the 10GBE sync channels with iperf and a ram disk for all NICs on all nodes, and they can all saturate the links up to around 99%, so I know the network side of things is working perfectly.

My current configuration is:

3x Dell 720xd
2x 10GBE Broadcom NICs per server. Each 10GBE NIC direct connected to the other two nodes and dedicated to sync traffic only.
1x Quad Port 1GBE Broadcom NIC per server. Two of these NICs are direct connected to the other two nodes for iSCSI traffic only.

Are there any other settings or tuning parameters I need to change to get it to utilise more of the sync channel? By my calculations, going by HA best practices, I'd need to get over 60% utilisation on the 10GBE sync channels to have more than 2x the potential 1GBE iSCSI client traffic. I found a post where a starwind employee mentioned that you should see up to 50% usage on a sync channel which I'm definitely not.

Cheers.

[Caillin]

Just an added note. This was a single sync request for one of our 4x1TB Raid0 devices in Starwind. I have another 3 of these 4TB arrays on each node. I manually started the sync on the other 3 nodes, and I'm seeing the Sync channel usage go up to around 430mb per 10GBE NIC, so around 40% utilisation.

So the question is, is there a way to make a single sync operation use more of the 10GBE connection?

[Anatoly (staff)]

Hi!

Thanks for staying with us:)

May I ask you if you have benchmarked the RAID arrays that are used to store the SW virtual disk files?
Also, have you tried to run the ATTO or IOmeter tests against the Basic Image file through the SyncChannel data link?

If not, I`ll ask you to run the tests that I~ve mentioned and share the results with us.

[Caillin]

Yes, I have run those Anatoly.

Please note that the IOPS for the 4k Random Read was after 20 mins, allowing the Cachecade hot tables to populate enough for getting a majority SSD hits, hence 77k IOPS off 4 x 1TB 7200RPM drives.
Also note that the last IOMeter test was using a cold, cool, warm, hot worker set with the smallest set small enough to fit in the Perc 710P onboard cache, hence the 1.8 Million figure.

[Caillin]

Also another performance related question.

When I do a 4k random write test in IOMETER across all 3 nodes, I get about 27k IOPS @ 106MB/s. Is this around the expected write performance for the Starwind memory based write cache? This test only utilised around 30% of the two 1GBE iSCSI NICs, where I was expecting it to pretty much max out each link, as technically it should be writing to system RAM on each of the 3 nodes. I'm guessing it's likely the latency and timing around the Starwind service waiting for the ACKs from all 3 nodes before comfirming the write to the initiator is responsible for the ceiling on the throughput? I notice when I run this test too, that each node sits at 38% CPU usage.... no mean feat considering each node has 2x Xeon E5-2680 CPU's (16 Cores/32 Logical processors @ 2.7GHZ per node).

Obviously doing 2 workers at 64 queue depth on 4k random writes is absolutely worse case scenario and in no way indicative of what to expect in production, but I'm just trying to make sure I have everything running optimally before proceeding with the CSV testing.

[Anatoly (staff)]

To be honest I don`t really trust ATTO when trying to troubleshoot the performance. It is a good tool for quick tests, but not for serious production environments. I would recomend to do your tests over with the IOmeter tool. Basically I need:
You to performed next tests:
· Local storage on node1
· Local storage on node2
· RAM disk that is running on node1 through the SyncChannel (node2 should be client machine)
· RAM disk that is running on node2 through the SyncChannel (node1 should be client machine)
· Basic Image target that is running on node1 through the SyncChannel (node2 should be client machine)
· Basic Image target that is running on node2 through the SyncChannel (node1 should be client machine)
· HA device
 
Every test should have following configurations:
· 32K 100% read sequential, # of Outstanding IOPS= 32 (or 64)
· 32K 100% write sequential, # of Outstanding IOPS= 32 (or 64)
· 32K 100% read random, # of Outstanding IOPS= 32 (or 64)
· 32K 100% write random, # of Outstanding IOPS= 32 (or 64)
· 64K 100% read sequential, # of Outstanding IOPS= 32 (or 64)
· 64K 100% write sequential, # of Outstanding IOPS= 32 (or 64)
· 64K 100% read random, # of Outstanding IOPS= 32 (or 64)
· 64K 100% write random, # of Outstanding IOPS= 32 (or 64)

I think it will be better if you will share the outputs in *.csv format with us after you will finish all teh tests.

The detailed network diagram with all the bandidths included.
Also, I think that [url]this [/url=http://www.starwindsoftware.com/starwin ... ice-manual]document will be really helpfull for you.

Thank you

[Caillin]

I've done similar tests as per the Benchmarks Best Practices document, but I'll run them again to be configured using the exact workers you specified above.

While I'm doing this, can I get a bit of insight into how the Starwind Write-Back cache is used and what sort of performance should be expected. I've built what I had hoped to be the ideal hardware environment for Starwind, and I was expecting pretty low disk write latency for our virtual environment by virtue of having a 10GB write-back cache on each of the 3 nodes. Having created and installed a new guest VM on our fresh 2012 cluster using all 3 nodes, I was seeing write latency with spikes up to over 200ms (really laggy), with some sustained 50ms+ times and subsequent lag inside the VM when navigating windows. From my understanding, the write latency should only be as bad as it takes the three nodes to acknowledge the disk write into memory and report back to the intiator. Anatoly, I believe in the past in the write-cache vs controller cache thread you mentioned that SW cache will always be faster than controller cache, but I'm seeing much worse write latency using SW cache than writing direct to spindle.

I'll get the thorough benchmark results up as soon as possible, and it may be worth getting a remote sessions happening just to make sure I have everything correctly configured.

[Caillin]

I'm disconnecting and removing all the targets from the 3 nodes, but every time I remove a target, the StarWind Management Console stops responding, then crashes, and then the Starwind service stops.

I had this happen once before during testing, but it's happened three times now as I've tried to remove each of the 4 targets. I've had to restart the server each time.

Logs attached here:
https://www.dropbox.com/s/8pe1ohu2irbb5 ... 0Target.7z

[anton (staff)]

Nice I've kicked devs to take a look ASAP. Stay tuned!

[Caillin]

Thanks Anton. I'll also have the IOMeter results ready in a few hours when I get to work. Takes a while to get through that many tests, but really the only way to see any bottlenecks.

[Caillin]

Here is a link to the full bench-marking results as requested above. https://www.dropbox.com/s/wj3zo3ajd0ezq ... sults.xlsx

I'm still looking through all the results, but there are some definite bottlenecks going on when comparing iSCSI over single and multiple nodes, and the local benchmarks.

Also, it seems as though even with the loopback connection installed in 2012, when you do 2 or 3 node HA benchmarks, it under-utilises the local disk on that node, where I would expect it to perform much better than the throughput to the other nodes, given they are restricted to 1GBE to each node.

There may be some kind of configuration problem here, but I've gone over every technical document over a dozen times and I'm sure everyting is set up correctly.

[Alex (staff)]

Caillin,
I've checked the StarWind log, and couldn't find the moment when console crashed or service stopped.
Could you please point the time when this occurred? Also please add StarWind service logs for the moment when you see that StarWind service has stopped.
Log file in the package start from 9/2 9:25:06 and have latest record at 9/2 9:43:12.
StarWind service starts new log file on each new run, so may be you need too include several latest log files.

If some critical error occurs, StarWind service writes .mdmp file with debugging information.
Do you have such files in c:\Program Files\StarWind Software\StarWind direcotry?

When Management Console crashes, it shows message box with error message and path do own .mdmp file.
Do you have any .mdmp files in c:\Users\<user name>\AppData\Roaming\StarWind\5\Console directory?


Regarding high CPU usage - have you checked, which process consumes the most of process time? Is it StarWindService.exe?

[Caillin]

I'm guessing there has to be something wrong with my configuration. Attached are two disk latency perfmon captures of installing a new Server 2012 VM from initial boot, to fully patching up to date from our WSUS server.

First test is read/write latency on local disk:

Local Disk

HyperVHost1LocalDisk2012Install.png (72.57 KiB) Viewed 19727 times

Second test is read/write latency over iSCSI using Starwind on the same array, but from the seond Node:

Through Starwind iSCSI with 10GB Write-back Cache

HyperVHost1fromHost22012InstallSingleNode.png (139.66 KiB) Viewed 19727 times

There has to be something wrong with the way I have Starwind configured, as I can't reconcile the terrible write latency of the SW test with it's 10GB of write-back cache vs uncached 4x1TB 7.2k spindle local disk.

Is there any way I can organise a support session with you guys to confirm my config?

[anton (staff)]

Does not look very informative (to say at least).

Sure (about remote session). Just drop a message to support@starwindsoftware.com with a link to this thread.

[Caillin]

I'm trying to find a proper way to gauge the cache effectiveness in real terms with a real guest VM workload. The definitive measurement for guest VM responsiveness and disk IO issues is disk latency. The differences above between the local node and the remote node can definitely be felt when navigating the guest VM.

Maybe I don't understand how the Starwind write-back cache works (not that there is any documentation to explain), but I wouldn't expect more than a few ms write latency on any writes up to the size of the write-back cache amount, as they should be confirmed to the initiator once they are confirmed written to memory on the node.