slow read/write speed

[leon3147]

i have 2 converged hyperv cluster in HA mode.
i test the hardware raid speed with this command :

fio --bs=4M --direct=1 --rw=read --iodepth=64 --name=c:\test
fio --bs=4M --direct=1 --rw=write --iodepth=64 --name=c:\test
fio --bs=4k --direct=1 --rw=randread --iodepth=64 --name=c:\test
fio --bs=4k --direct=1 --rw=randwrite --iodepth=64 --name=c:\test

results are : 726 mo/s - 646 mo/s - 2312 io/s - 1028 io/s

i do the same test but with the iscsi disk like this :

fio --bs=4M --direct=1 --rw=read --iodepth=64 --name=c:\ClusterStorage\Volume1\test
fio --bs=4M --direct=1 --rw=write --iodepth=64 --name=c:\ClusterStorage\Volume1\test
fio --bs=4k --direct=1 --rw=randread --iodepth=64 --name=c:\ClusterStorage\Volume1\test
fio --bs=4k --direct=1 --rw=randwrite --iodepth=64 --name=c:\ClusterStorage\Volume1\test

result are : 282 mo/s- 159 mo/s - 1613 io/s- 868 io/s

so the strawind LUN is very slow (282 mo/s-159 mo/s) but the IO/s is ok !
jumbo frame is ok and the MPIO Policy is "fail over only" and active path is localhost 127.0.0.1.
cpu is 3% and ram 4% use.

so what is wrong ?
any idea are welcome !

[leon3147]

test with a ram disk all speed is ok.
test with a new target + device without replication, read an write speed is very slow again

so the probleme is not with replication process.

what could be wrong with local link 127.0.0.1 ?
StarWind have juste write bit using iscsi command with local link !
so why it doent use my backend raid full speed ?

are StarWind buggy ?!!

[Michael (staff)]

Hello Leon3147,
Could you please provide us with underlying storage settings, i.e. type and number of the disks, RAID type and its settings (Disk cache, Read\write policies, Stripe Size)?

[leon3147]

hello,

it's a perc h700 with 512Mo cache : raid 50 with 12 disk sata 7200 "WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU"
i make 2 VD in this big raid 50.
one 200 go for the hyperv 2012r2 system and one 14to for store StarWind files.


Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name :Virtual Disk0
RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
Size : 200.0 GB
Sector Size : 512
Parity Size : 100.0 GB
State : Optimal
Strip Size : 64 KB
Number Of Drives per span:3
Span Depth : 4
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU

Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU

Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disk's Default
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: Yes
Cache Cade Type : Read Only


Virtual Drive: 1 (Target Id: 1)
Name :Virtual Disk1
RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
Size : 14.355 TB
Sector Size : 512
Parity Size : 7.177 TB
State : Optimal
Strip Size : 64 KB
Number Of Drives per span:3
Span Depth : 4
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU

Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU

Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disk's Default
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: Yes
Cache Cade Type : Read Only


i make a test target with 5go write-back 128Mo, no replica in StarWind, connect with 127.0.0.1 iscsi link.
don't understand where the problem is !

[Michael (staff)]

Hello Leon3147,
Thank you for the provided information.
Please set RAID settings with the following parameters:
Disk cache policy: default;
Read policy: read ahead;
Write policy: write back;
Stripe size: 64KB;
Also, disable Cache Cade on the RAIDs and check storage performance one more time with DISKSPD utility: https://gallery.technet.microsoft.com/D ... e-6cd2f223
Please keep us updated about the results.

[leon3147]

Hello Michael,

thanks for your help.

i test your raid setting but no change with Fio benchmark.

i test with DISKSPD utility and that's change all !
now read and write speed is ok with the device without replication, on 127.0.0.1 iscsi link.

but when i test with the device with HA replication i have very strange result.

in the first hyperv node read is ok but write speed and IO is very slow (76 MB/s).
in the second hyperv node read and write is ok !

i test my intel dual port x520 10G synchro network card with iperf but all is ok. (7go/s)

do you have see that before?

[Michael (staff)]

Hello Leon3147,
Please check MPIO settings in the iSCSI Initiator on the first node - click on the Target->Devices-> MPIO. The target should be connected via two paths: by loopback (127.0.0.1) and by iSCSI address from the Partner server.
You can find a step by step guide for configuring StarWind Virtual SAN in hyper-converged scenario for Hyper-V here:
https://www.starwindsoftware.com/techni ... manual.pdf

[leon3147]

hi Michael

i use this step-by-step guide to create my hyperv StarWind hyperconverged system.
i check a new time but all is correct.
my 2 CSV is connected 3 times. one by loopback 127.0.0.1 + one first ip iscsi Partner + one second ip scsi ip Partner.
only one path is active ,and is 127.0.0.1.

read speed is very good (1.2Go/s) but write speed is very slow (47Mo/s)

if i check " first ip iscsi Partner" as active and unckeck "127.0.0.1" read and write is ok.
i m gonna be mad with this !

[John (staff)]

Hello Leon3147

It would be great to know how many iSCSI connections you have for one HA device on cluster node at iSCSI iniciator?

For understanding your case i would ask you to provide inforamtion about NiCs and networking.

my 2 CSV is connected 3 times. one by loopback 127.0.0.1 + one first ip iscsi Partner + one second ip scsi ip Partner.
only one path is active ,and is 127.0.0.1.
i m gonna be mad with this !

[leon3147]

see attachements,
i use this network schema.

hyperv3 is the first node with write speed problem.
on hyperv4 read write speed are ok.

i have one intel dual port x520 10G on each hyperv node.
i connect cross over fibre channel like this :
hyperv3 G1 <-->hyperv4 G1
hyperv3 G2 <-->hyperv4 G2

i make 2 Vlans on.
one for iscsi and one for synchro (see network schema)

i put managementOS QOS Weight on it: 50 for iscsi + 50 for synchro (with MinimumBandwidthWeight command)

attachements are the iscsi hyperv3 configuration.
one with 127.0.0.1 link and another with 2 links to the Partner node.

[leon3147]

i try to change 127.0.0.1 iscsi active link with my local ip adresse.
write speed grow from 70 Mo/s to 270 mo/s !

270 is not the full speed, hyperv4 write at 450 Mo/s but it 's 4x greater than with 127.0.0.1.

So it's definitively a probleme with 127.0.0.1 Accelerator !
i think it's same issue than here : https://forums.starwindsoftware.com/vie ... f=5&t=4436

i use hyperv2012r2. is there any stuff with QOS make by hyperv in his 127.0.0.1 link ?

[Michael (staff)]

Hello Leon3147,
Thank you for your investigations.
Please check that you have uncommented the line «<!--<iScsiDiscoveryListInterfaces value=»1»/> -->» in StarWind configuration file.
The default path is C:\Program Files\StarWind Software\StarWind\StarWind.cfg. The line should look as follows: <iScsiDiscoveryListInterfaces value=»1»/>
Also, please check StarWind log for events like "Srv: Flb driver is already running." or "Srv: Flb has been activated.". In case if you see them in the log, it means that Loopback Accelerator has been installed.

[leon3147]

hello

yes i double check this line : «<!--<iScsiDiscoveryListInterfaces value=»1»/> -->»
all is ok and i have this in my log :
10/27 15:52:55.819 62e64 conf: Variable 'iScsiDiscoveryListInterfaces' is set to '1'.

and yes i have this in my log :
10/27 16:22:55.383 5c0 Srv: Flb driver has been started.
10/27 16:22:55.383 5c0 Srv: Flb has been activated.

can i disable loopback Accelerator to make some test ?

[Michael (staff)]

Loopback Accelerator is a part of the software, thus it can be uninstalled only with StarWind. It can be installed later by running StarWind installer and checking corresponding checkbox.
Meanwhile, have you restarted StarWind service after you have edited the configuration file?

[leon3147]

i edited the file when i install StarWind one month ago so yes i alerady restart service and server too.
did you plan to recreate this issue in your labs and patch it ?