ESXi iSCSI initiator WRITE speed

[rchisholm]

Not using any teaming on the iSCSI. I have the sync channels as 2 separate 10GbE crossover connections also. Just tried the teaming during some testing to see what would happen. It really wasn't pretty. Only 1% of the performance it has now. I'm going to start testing jumbo frames Monday morning, and do some tests utilizing both sync channels, as well as the much faster RAID's. After that I'm going to start working with the new Xen servers. Should be very nice once I'm done tweaking. It already smokes our old SAN.

What are StarWind's recommended settings for the newest version of Xen? I know you have done recent research into it and would like to start my testing with the best known settings.

Thanks

You should skip using NIC teaming for routing iSCSI traffic. For clients use properly configured MPIO and for sync channel please stick with failover (V5.8 should have custom MPIO for sync channel, no this feature in V5.7 sorry...)

[CyberNBD]

I used a completely seperate subnet for each iSCSI channel. including seperate vSwitches and kernel interfaces.

Did you setup each of your iSCSI NIC's in ESXi in separate virtual switches? When I had a single virtual switch and the standard 1000 IOPS I got about 6 MB/s. Going to separate virtual switches for each iSCSI NIC gave me about 60 MB/s with sufficient load. Then I did the IOPS change which brought it over 600 MB/s. So, with the changes combined, it increased the write speed 100X.

I'm going to continue testing with different IOPS settings and frame sizes. I will also be testing with Xen next week. I still need to do some testing with the fast RAID's also.

[Max (staff)]

What are StarWind's recommended settings for the newest version of Xen? I know you have done recent research into it and would like to start my testing with the best known settings.

For 5.6 FP1

Goto StarWind installation folder, open starwind.cfg file and uncomment string below:

Code: Select all

<!--<iScsiDiscoveryListInterfaces value="1"/>-->

Reboot will be required to changes applied

StarWind Servers Registry (HKEY_LOCAL_MACHINE/System/ControlSet001/Control/Class/{4D36E97B-E325-11CE-BFC1-08002BE10318}/0000/Parameters):
MaxBurstLength - 0x00040000(262144)
MaxRecvDataSegmentLength 0x00040000(262144)
MaxTransferLength 0x0080000 (524288)

StarWind Management console:
Max Burst Length 262144
Max Receive Data Segment Length 1048576
->Restart

Xen servers which will be using HA:
put the server in maintenance mode and enable the Multipathing->Exit the maintenance mode
edit etc/iscsi/iscsid.conf ->
maxburstlength - 262144
maxreceivedatasegment - 131072
Added node.con[0].tcp.window_size = 262144

edit etc/multipath.conf - add these lines into the uncommented defaults section:
user_friendly_names no
polling_interval 10
path_grouping_policy group_by_prio

After the files have been edited-> service open-iscsi restart
In order to connect the targets:
1. iscsiadm -m discovery -t st -p SAN1IP:3260
2. iscsiadm -m discovery -t st -p SAN2IP:3260
3. iscsiadm -m node -T SAN1 target IQN -p SAN1IP:3260 -l
4. iscsiadm -m node -T SAN2 target IQN -p SAN2IP:3260 -l
after this go to the etc/iscsi/nodes/SAN#-IQN/SAN# IP/ -> in the default edit the login and startup sequences, lower the tcp.window.size to 262144 and save the changes (this must be performed for both targets logged on)
When done - service open-iscsi restart
Now add the SR using the GUI like a regular iSCSI SR (specify SAN# IP and the appropriate target)
The SR will be visible as multipathed.♠

[rchisholm]

Thanks Max.

Next question. I was doing some iPerf tests on the servers and can get 9.46 Gbits/sec on each of the iSCSI NIC's that are going through the switches. However, the same model NIC's, with the same settings, each NIC is only getting 4.17 Gbits/sec for the Sync channels that are directly attached to each other with the same iPerf test. The NIC's are HP NC522SFP Dual Port 10GbE. I'm using HP Direct Attach SFP+ cables between them and the HP Procurve 6600 switches, and between the NIC's that are directly attached. Any ideas?

[Max (staff)]

There can be multiple reasons for this: starting from the PSU voltage lack and PCI bus limitations and ending with the drivers

[rchisholm]

With some more testing and a larger window size for iPerf, 3 of the connections on the dual port 10GbE NIC's consistantly run at 98.5% network utilization. 1 of the connections, which is a sync channel, runs between 70-80%. I'm going to do some more testing. Maybe a cable issue.

[anton (staff)]

Could be! Please check number of collisions and retransmitted packets with your NIC and/or switch statistics.

With some more testing and a larger window size for iPerf, 3 of the connections on the dual port 10GbE NIC's consistantly run at 98.5% network utilization. 1 of the connections, which is a sync channel, runs between 70-80%. I'm going to do some more testing. Maybe a cable issue.

[rchisholm]

After further testing, sometimes each of the sync channels want to run at 70-80%, and sometimes they run at 98.5%. There are 0 collisions and 0 errors. They are directly connected, so no switches are involved.

I've also found that even though I can push the write speeds up to very high speeds using a utility, real world won't come close. I had to abandon round robin because no matter how I tweaked it, it's writes were way too slow to be usable. I've found that the best real world performance is coming from setting the path selection to VMW_PSP_FIXED_AP. Is StarWind going to come out with it's own multipathing for ESXi like some of the other SAN solutions as part of the upcoming performance enhancements with HA?

Could be! Please check number of collisions and retransmitted packets with your NIC and/or switch statistics.

With some more testing and a larger window size for iPerf, 3 of the connections on the dual port 10GbE NIC's consistantly run at 98.5% network utilization. 1 of the connections, which is a sync channel, runs between 70-80%. I'm going to do some more testing. Maybe a cable issue.

[anton (staff)]

How many IOPS had you configured for RR before link switch should be performed?

Not sure what you mean under "own multipathing for ESXi some of other SAN solutions...". I though MPIO is one and only. Do you happen to have any samples? PM if you don't mind. Thanks!

After further testing, sometimes each of the sync channels want to run at 70-80%, and sometimes they run at 98.5%. There are 0 collisions and 0 errors. They are directly connected, so no switches are involved.

I've also found that even though I can push the write speeds up to very high speeds using a utility, real world won't come close. I had to abandon round robin because no matter how I tweaked it, it's writes were way too slow to be usable. I've found that the best real world performance is coming from setting the path selection to VMW_PSP_FIXED_AP. Is StarWind going to come out with it's own multipathing for ESXi like some of the other SAN solutions as part of the upcoming performance enhancements with HA?

Could be! Please check number of collisions and retransmitted packets with your NIC and/or switch statistics.

With some more testing and a larger window size for iPerf, 3 of the connections on the dual port 10GbE NIC's consistantly run at 98.5% network utilization. 1 of the connections, which is a sync channel, runs between 70-80%. I'm going to do some more testing. Maybe a cable issue.

[michal]

On the ESXi side, did you properly set up the vswitches and vmkernels for multipathing? If you don't, all traffic will go over the first NIC with the same subnet, never touching the second nic, thus negating any possible boost providable by MPIO.

Read this:
http://virtualgeek.typepad.com/virtual_ ... phere.html

[anton (staff)]

...also traffic nature could make the whole thing looking either completely black or white. Tuned setup may shine in tests but fail to deliver reasonable results for production. So what exactly does OP do to test performance?

[georgep]

He meant like DELL EQ has it`s own RR MPIO module path policy EMS i think it`s called or something like that...that integrates with the hyper visor

[quote="anton (staff)"]How many IOPS had you configured for RR before link switch should be performed?

Not sure what you mean under "own multipathing for ESXi some of other SAN solutions...". I though MPIO is one and only. Do you happen to have any samples? PM if you don't mind. Thanks!


[quote="anton (staff)"]Could be! Please check number of collisions and retransmitted packets with your NIC and/or switch statistics.

[anton (staff)]

Not really. Hardware vendors (Google a bit to find out who exactly) prefer replacing ESX built-in multipath controller with something called MEM (Multipathing Extension Module) to boost performance.

He meant like DELL EQ has it`s own RR MPIO module path policy EMS i think it`s called or something like that...that integrates with the hyper visor

How many IOPS had you configured for RR before link switch should be performed?

Not sure what you mean under "own multipathing for ESXi some of other SAN solutions...". I though MPIO is one and only. Do you happen to have any samples? PM if you don't mind. Thanks!

Could be! Please check number of collisions and retransmitted packets with your NIC and/or switch statistics.

[georgep]

yeap its called MEM

[anton (staff)]

So you may want to experiment with a system with or w/o MEM enabled, run some tests and share your results with S/W team and forum readers. We'd appreciate it greatly. Thank you!

yeap its called MEM