ESXi 5.x poor iSCSI performance

[RandyVF]

I have been following the posts about iSCSI performace with ESXi 5.x on the forum for a while. I have tried all of the recommendations. DelayedAck, Disk.UseDeviceReset = 0, Jumbo Frames, etc. All to no avail. I have a 10G nic in my ESXi 5.1 server and a 10G nic in the StarWind iSCSI server. When running Iperf between them I can get 600 - 700 MB / sec. However, with ESXi software iSCSI to Starwind, I cannot get over 20MB/sec and then it often falls to 2.5 MB/sec..

So to test my iSCSI Target server (Starwind 6.0), I put a 10 GB Nic into PC (running Windows 7) and installed the Starwind iSCSI initiator and can get read speeds in excess of 500 MB/Sec and Write speeds in Excess of 400 MB/Sec. I do not know where else to look on my ESXi box(es). I have not looked at esxtop as of yet though.

ATTO Disk Benchmark from windows 7 with Starwind Initiator to Starwind 6.0 iSCSI server

iSCSI-10G-SAS.JPG (76.94 KiB) Viewed 11503 times

My Starwind box is:
Dual 5420 Xeon quad core CPUs
32 GB Ram
Supermicro X7DBI+ Motherboard
2 x 1 GbE Nics on motherboard (OS use, not iSCSI)
10 GbE Nics (direct connection between servers) (for iSCSI)
SATA Raid 1 for Windows OS
SAS Raid 1 for iSCSI Disk (Perf stats are based on these disks)

ESXi server [ESXi 5.1]
Dual Intel 5345 2.33 Ghz quad core CPU
32 GB Ram
Intel S5000PSLSATA motherboard (Rackable C2004 server)
(2) 1 GbE Nics on motherboard (Management and VM networks not for iSCSI)
(1) 10 GbE Nic (direct connection between servers) (for iSCSI)
(1) 250 GB SATA II for boot and local datastore

Any suggestions or ideas?

Randy

[RandyVF]

To be clear, I wanted to reiterate that the Windows 7 PC is testing against the same iSCSI target (Starwind 6) server that the ESXi box(es) were using.

[lohelle]

Have you tried running iscsi over one of the 1gig cards?
This will rule out driver issue for the 10gig card on the ESXi host.

Can you ping the vmkernel-port of the vSphere host from the Starwind server? (to be sure that the correct VMkernel port is used if more than one)

If you install the vSphere client on the Starwind server you could also try to copy files to/from an internal datastore (if there are internal storage). Connect to the VMkernel IP (same subnet as Starwind)

[RandyVF]

Before I installed the 10 GbE Nics I had an Intel 1 Gbe Quad port nic in the slot. In fact the other ESXi host still has the Quad port 1 GbE card and performance absolutely stinks there as well.

I had dedicated VMKernel ports for iSCSI on both ESXi boxes. Both the 1GbE and 10GbE cards.

I will load the vSphere client on the Starwind iSCSI target and copy from the datastore. Although, if I am not mistaken, that will come through the management network.. I suspect that will be fast as I have copied files to / from the datastore on my desktop (same network as management network) and the speeds were what I expected.

The host with the 10GbE card is ESXi 5.1 and the host with the 1GbE quad port is ESXi 5.0U1

Randy

[Anatoly (staff)]

BTW, I`d recommend to use IO Analyzer tool for Benchmarking on ESX.
Here is the download link:
http://labs.vmware.com/flings/io-analyzer
and here is the How-To (there are a lot of them, so if this one will not fit you just google a while):
http://vmsarefreeright.wordpress.com/20 ... -analyzer/