ESXi Round Robin performace worse that fixed path

[craggy]

Thanks, I just replied to the mail now.

[Max (staff)]

Just to update - We're currently working on this case, me or one of my colleagues will post an update here

[craggy]

Sorry, it's been a while since i was on here for various reasons but I need to revisit this issue.

Basically we are still experiencing slow performance when using RR MPIO with ESXi 5.1.
When we switch to just using one link or the other we can saturate a 1gb link with 122MB/s write / 120MB/s read but once RR is enabled we can only get max 158MB/s write and 163MB/s read.

I have tried setting ESXi IOPs to 1000, 100, 10, 3, 2, and 1with best performance at 3 of around 150MB/s each way.
Then I tried setting ESXi IOPS to Bytes and set it to 8800, 8972 and 9000 with best performance at 8972 of around 163MB/s

So its really strange to see such low performance from RR when direct link can saturate a 1gbe nic to almost wirespeed.

Things i've enabled and disabled from both sides where possible are as follow:
Delayed ACK
RSS
LRO
LSO
Tcp/ip checksum offload
Tcp/ip send offload
Jumbo frames
Interrupt moderation disable (most significant performance gain here ~25%)
Receive window auto tuning
Chimney offload.

Don't think I can adjust anything else.

Any suggestions?
Thanks.

[Max (staff)]

We're going to update the ESXi Guidelines really soon - these have a fully updated set of multipathing recommendations.
One question though - do you see consistent performance with just port 3 and port 4 being used for iSCSI traffic?

[craggy]

Max,

Sorry, when you say port 3 and 4 what exactly do you mean by that?

[Max (staff)]

Max,

Sorry, when you say port 3 and 4 what exactly do you mean by that?

In your initial post you've described the configuration of the Storage box -

Storage Host:
Nic1 Broadcom Wan and MAnagement (80.9x.xx.xx subnet) vSwitch 1, Vmkernel1
Nic2 Broadcom Admin and Vmotion (10.10.1.x/24 subnet) Vswitch 2, vmkernel 2
Nic3 Intel iScsi (10.10.10.x/24 subnet) vSwitch 3, vmkernel 3
Nic4 Intel iScsi 2 (10.10.11.x subnet) vSwitch 4, Vmkernel 4

So Ports 3 & 4 are NICs 3 & 4 from the description

[craggy]

Sorry, i get you now.

Yep, if I disable port 3 or 4 and just use a single path I can saturate the link and get 120MBs / 123MBs read / write.

[Max (staff)]

Yes, I've got that.
I was wondering if it aggregates well when you only use 3 & 4 together.