Replication never goes over 1Gbit

transparent · Mon Jun 02, 2014 9:24 pm

We have 2 x 2012 R2 Server running v8 (although this issue existing with v6 as well). There is a direct connection between the two systems using IPoIB for sync channel.

When using iperf I can get 8.79Gbit/sec across the link. However when Starwind is performing a fully sync it will get to 1Gbit (around 115MB/sec we see written) and never go above that. It's almost as if its hitting some sort of limit?

Are there settings/changes/tuning I need in order to get about 1Gbit for a full sync across hosts?

Thanks,
Andrew

Tue Jun 03, 2014 7:27 am

Did you play with S/W sync priority set? What latency do you get with your connections? Any chance to see also NTtcp numbers?

transparent · Tue Jun 03, 2014 9:07 pm

Yes, I've moved the slider all the way to the left (Faster Sync). The latency between hosts is <1ms using ping test over an extended period. Please see attached screenshots of ntttcp test (was able to get 1130 MB/sec). We are syncing a flat image file and no load on server during sync. What strikes me as odd is that it can hit 1Gbps no problem and pins it there. If it was dipping/rising or was lower then I'd suspect disk, but because it always goes to exactly 1Gbps led me to believe that something was throttling it.

Wed Jun 04, 2014 3:15 pm

±50% CPU usage is annoying... Also IPoIB is not fast. OK, I'll ask guys to do a remote session with you to see what's wrong with your config.

Wed Jun 04, 2014 3:43 pm

What about the underlaying storage? I mean the location on which StarWind images are stored. Is it capable to show better numbers than 1Gb/s (128MB/s) locally?

Wed Jun 04, 2014 3:54 pm

That`s weird. JFYI in our test lab the Sync channel hit 100% utilization of 10Gigs interfaces.
Can I ask you to create the RAM-based HA and run the speed test again?

transparent · Fri Jun 06, 2014 7:18 pm

We have a 6 disk (SATA) RAID 10, and disk test show it's capable of ~340MB/sec Read and ~290MB/sec write (sequential, and I'm assuming an image replication would be mostly sequential? I'm guessing).

That's a good idea with the RAM drive to take disk subsystem out of the equation. I'll give that a try over the weekend and report back.

Andrew

Fri Jun 06, 2014 9:26 pm

Not really... Sync channel is the same as your write pattern (multiplexed b/c of a multiple data paths). So if you have mostly random writes then your sync traffic is also going to be "pulsating".

transparent · Tue Jun 10, 2014 1:36 am

Yes, I would assume that would be the case with live or 'real-time' I/O that are ongoing while the cluster is 'in-sync'. Is that the case though while doing a full sync with no client I/O active, and the replication priority set to 'faster sync'? I would assume in that case it would be starting at the beginning of the image file (I'm using flat images, not LSFS) and going to the end sequentially?

Fri Jun 13, 2014 12:55 pm

That's a good idea with the RAM drive to take disk subsystem out of the equation. I'll give that a try over the weekend and report back.

Can I ask you if you have any results for us to share?

Is that the case though while doing a full sync with no client I/O active, and the replication priority set to 'faster sync'?
I would assume in that case it would be starting at the beginning of the image file (I'm using flat images, not LSFS) and going to the end sequentially?

Do you meant synchronization as the recovery from one of the nodes failure? It`ll be seq writes on the recipient if so.

barrysmoke · Sat Jun 21, 2014 9:23 pm

Having a similar problem with replication times, so I went to create a ram ha device, and replication manager is grayed out.
also, can't post a screenshot to show you...
Could not upload attachment to ./files/4009_6957aff394d20cba3134b92befdaf3af.

Sun Jun 22, 2014 8:15 pm

Do you have some numbers / config to share? Please send what you wanted to attach to support@starwindsoftware.com and I'll ask web team to check forum attachments. Thanks!

barrysmoke wrote:Having a similar problem with replication times, so I went to create a ram ha device, and replication manager is grayed out.
also, can't post a screenshot to show you...
Could not upload attachment to ./files/4009_6957aff394d20cba3134b92befdaf3af.

Nikolay (web team) · Mon Jun 23, 2014 7:31 am

barrysmoke wrote:Having a similar problem with replication times, so I went to create a ram ha device, and replication manager is grayed out.
also, can't post a screenshot to show you...
Could not upload attachment to ./files/4009_6957aff394d20cba3134b92befdaf3af.

Hi barrysmoke,
Please try to upload your file again.
Thanks

Mon Jun 23, 2014 9:46 pm

...which means "there was an issue with forum and we've fixed that so attachments are working now"

barrysmoke · Tue Jun 24, 2014 11:28 pm

Alex, and Bohdan are helping troubleshoot the issues over the next couple of days.
I was able to determine a 5TB thick sync'd in 2 hours, and ran at 5.9GB/s on the 10GB sync & heartbeat nic.
might be an issue with thin/lsfs

the screenshot was just of the ha option greyed out for ramdisk. Alex said you can't put a ramdisk into an ha sync...