Why so different rates of iSCSI read/write on Win vs vmware?

[BradI]

I am mighty impressed with the Starwind tests I started doing last week. On an old server with RAID5 I was pushing 2000 IOPS and getting 10-40MBps via iSCSI when I used Windows iscsi initiator. I connected to the lun via my Windows 7 workstation and using IOMETER, was able to get those rates.

I have a dedicated 1GB switch for storage.

However, when I created a new lun and used vSphere to connect to the Starwind ISCSI using the same path I used before, rates were cut to 1/10th of the original. I tried via vmware software and hardware initiator. Both gave a max of 300 iops (but an average of <100) and a throughput of 1-1.5MBps.

I am sure it's something in my config. Do you have any ideas on what I may have missed or at least a starting point for my troubleshooting efforts?

Thanks!

[anton (staff)]

Both system numbers suck. Anything lower then wire speed is not normal. Check this forum to start with network tuning, local I/O tests and VMware optimized IOPS settings. Topic has been discussed millions of times. If you'd still experience the issue after all we'd be happy to help however.

[BradI]

Just to follow up, I tested the network and it is working good.

What made it work great in my estimation was I took it off a VMware hardware iscsi and moved it to VMware's software initiator. Now it gets 15,500 iops and 60MB/s write speed. That seems great for this old server bought in 2004.

Any ideas on why the hardware initiator would perform so much worse as compared to the software?

[anton (staff)]

IMHO your iSCSI HBA is powered by some old 200MHz offload processor with a 2006 or so dated firmware. And your multi-core and multu-GHz CPU runs circles around it... However I'd address the same question to your HBA vendor

[georgep]

if you use MPIO RR with starwind just set RR to 1IOPS instead of default 1000IOPS and you`ll see the same exact seems you get with windows initiator.

[BradI]

Thanks, I will give it a try

[georgep]

how many nics you use with Starwind for ISCSI traffic ? Are you using ESX5 or 4 ? I can give you some CLI to set that.. I beleive the forum has some command on ESX 4

[BradI]

I am running ESXi 5 with 2 hosts and 2 nics on each host.

[georgep]

Please see this : http://www.starwindsoftware.com/forums/ ... 95-30.html

You`ll see what a huge difference made RR to IOPS1 instead of default VMware IOPS 1000.

I`ll try to get you the commands tonight on how to change the RR IOPS on ESX5

[BradI]

Thanks, but you don't have to write up scripts if you don't want to. I am in a small environment and can manually do the config.

[anton (staff)]

Good! Please change your config and let us know did it work or not. Thanks!

Thanks, but you don't have to write up scripts if you don't want to. I am in a small environment and can manually do the config.

[georgep]

These are some of very useful commands on Vsphere 5 to set MPIO RR to change every 1 iop. Now you can play and increase from 1 to 3 , 6 and 9 if you think 1 it`s too much. I left it to 1 as I got my 99% network card full utilazation on 4 nics so I did not bother to try with other numbers.

Please change the device UUID to match your LUN.

esxcli storage core device list

esxcli storage nmp psp roundrobin deviceconfig get -d eui.b4e0d749d45bcd47

esxcli storage nmp psp roundrobin deviceconfig set -I 1 -t iops -d eui.b4e0d749d45bcd47

[anton (staff)]

Thank you for your valuable input! Hope OP would manage to have everything up and running.

These are some of very useful commands on Vsphere 5 to set MPIO RR to change every 1 iop. Now you can play and increase from 1 to 3 , 6 and 9 if you think 1 it`s too much. I left it to 1 as I got my 99% network card full utilazation on 4 nics so I did not bother to try with other numbers.

Please change the device UUID to match your LUN.

esxcli storage core device list

esxcli storage nmp psp roundrobin deviceconfig get -d eui.b4e0d749d45bcd47

esxcli storage nmp psp roundrobin deviceconfig set -I 1 -t iops -d eui.b4e0d749d45bcd47