ESXI 5 poor performance, even worse after IOPs change

[jmchristy]

Hello,

I'm currently experiencing an issue where my VM's are not performing as they should be. Even worse, when I make the recommended change with the iops setting to 1, my read performance plummits.

Attached are my performance tests within the VM and a diagram of my setup. I only have 1 VM running on 1 HA LUN, I disabled Delayed Ack, set RR, Jumbo Frames enabled, and bound 4 NICs to my Storage Adapter in ESXI. I ran a test before I made the change on the host setting iops to 1, and after. I was thinking I could swap out the physical switches and see if that makes a difference - I did read another poster have a problem with Nortel switches. I applied the changes he used to fix his issue, but it didn't seem to have any affect here.

My SAN's have 12 450GB 15K iSCSI drives in a RAID 10, with the 5 1GBps NICs and the 4 1GBps NICs from the host - shouldn't I easily be able to achieve 200MBps?

Thoughts?

[jmchristy]

I'd like to add an update.

I removed the Broadcom network cards from both the ESXI host vSwitches, and from the Starwind SAN - with my iops set to 1 and my read performance went up. I checked and the Broadcom NIC's on the SAN did have jumbo frames enabled, as well as on the ESXI hosts. I'm not sure why with those in the team, it causes the issue.

So now, I have 2 1GB Intel NIC's on my host's iSCSI network. Is this about the best I'm going to get performance wise? Any other suggestions?

[jmchristy]

So I got to spend some more time on this today, and it turns out I'm having a similar issue that a user posted on this forum several months ago.
http://www.starwindsoftware.com/forums/ ... 85-15.html

I swapped out my Nortel switches for an HP ProCurve, added back the Broadcom NICs that I thought caused the problem before, and it performed ALOT better. I put the Nortel's back, and kept the Broadcoms and boom....slow read again.

I came across that link above, and tried what he said fixed his issue but that didn't resolve it for me. I actually disabled flow control and auto negotiation and I was able to get some pretty fast speeds - almost double what I was getting before....except for the last couple of tests the Read performance again drops off. So this looks like it is still a Nortel/Avaya issue. When I had the Procurve in, it was at least able to sustain 150-200MB READ performance during the last few tests (2048,4096 and 8192)

I'm thinking I'm just going to bite the bullet and purchase a support contract with Avaya to have one of their techs look at it and see. At this point it certainly doesn't appear to have anything to do with Starwind SAN - just thought I'd share in case someone travels down a similar path...

[Anatoly (staff)]

Hello,

Well, we love self-supporting users:)

To be honest the only thing that I can suggest for now is to think twice if you need a support or the replacement.

[jmchristy]

Update:

I've had Avaya technical support troubleshooting this with me for almost 2 weeks now and we haven't been able to resolve the performance issues. Where we are at now, is that the tech noticed 3 of my 4 Nortel switches are hardware revision 1, and the 4th one is hardware revision 32. He's shipping out 3 new Avaya switches that are hardware revision 32 to see if that resolves it - I'm skeptical. I'd love to just buy new switches but it's not in my budget for this year....

He asked the question, is ATTO Benchmark an accurate utility to benchmark or should we try something else? I see that a lot of posts on here recommend using iperf, but how will that test all the channels? I can use that and easily saturate a single link. My issue is that when I run the ATTO benchmark test, with what seems like 4 or more links to my SAN's, the read performance plummets. It doesn't matter if it's 5 links from my primary SAN, 3 links from primary, 3 links from secondary, as long as it's more than 4 links. I have RR enabled on the LUN, and I can see when I run the tests it's hitting the correct NIC's on each SAN.

Is there a different tool I should be using to get performance results?

[anton (staff)]

OK, please keep us updated with your hardware update results & status.

ATTO is fine for round estimates. You can use also Intel I/O Meter and app dedicated workload benchmarks (Intel NAS performance toolkit, MS JetStress and SQLIO).

[jmchristy]

Exchanging the switches did not fix the performance results.

Back to the drawing board....

[Anatoly (staff)]

We`ll wait for the update from you.

But may I ask you why don`t you want to leave HP switches? As far as we understood you achieved good performance using them.

[jmchristy]

I did not want to leave the HP switch in because it was a single unmanaged switch. I liked the idea of using the Avaya 5510's which I could stack, and spread out the connections over multiple connections for failover. Plus, I can achieve greater speeds - plus I'm persistent and hate to give up!

I have been able to spend a good deal of time on this, and I came across a problem that I was able to replicate several times. This problem caused me to receive mixed results ever since day one of my performance testing, and here I thought I found something but it was identified already earlier this year! It's actually a bug in my ESXI host, that is fixed in a patch which we have not applied however there is a work around. The problem is described in the link below.

http://kb.vmware.com/selfservice/micros ... Id=2008144

I had 2 virtual switches configured in my ESXI host, with 2 VMkernels in each. The article describes the issue that if you have a link go down in this setup, the Vswitch will try to push traffic from the 2 logical vmkernels out one physical connection causing latency/dropped packets. The work around is to create a Vswitch for each vmkernel, so I created 4 vswitches, each with 1 vmkernel and now I can reboot the switch stack and the speed doesn't drop. I haven't gotten a chance to speak with VMware again since I corrected this, to see why when I put the HP Procurve in - I was still at least able to get steady speed results. That's the only head scratcher I've got left with that one...

My 2nd issue, is adding the secondary array back into the mix...where I'm also dropping on my read performance once that's added. Of course I wasn't lucky enough to have this problem kill 2 birds with 1 stone! Once I add the partner SAN to the storage adapter in ESXI, my write performance stays lightning fast but my read performance again drops. I'm going to reach out to Vmware again today to see if it's another issue with their software or if this directs me back to Avaya.

I'll also note that any of my other testing/comments about changes made to the Avaya switches should be ignored - since the culprit was this issue described above. Everything that the user paulow1978 described in the link below needs to be done to receive optimal performance. You must also keep in mind, that if you are using Avaya/Nortel 5510 48-T switches, that there is a limited qos buffer space on the 5510 switches. You have to spread out your SAN/Host connections every 12 ports, since each bank of 12 share the buffer. I have a stack of 4 switches, so I have my SAN's spread out in ports 1/13,2/13,3/13,4/13 and the servers in 1/1/,2/1,3/1,4/1, etc. etc.
http://www.starwindsoftware.com/forums/ ... 85-15.html

I'll post my outcome after working with the techs on this second issue with my secondary SAN performing sluggish although it may be awhile as I'm heading out to the McAfee security conference next week

[Anatoly (staff)]

Wow! That is the great work that you have done already and will do soon! What is interesting is that this is second time when I hear about Avaya switches, and both times it was looooooong investigation and troubleshooting.

OK,we`ll look forward to hear back from you and good luck at the conferense!

[jmchristy]

Ok, figured out what the last problem was with adding the secondary SAN to the mix.

In my setup, my ESXI server had 4 iSCSI network paths while each of my SAN's had 4 paths. So my SAN's had 8 paths total, where as my ESXI host only had 4.

When I dropped the total number of SAN paths down to 5, the performance was good but increasing the paths to 6 the read performance would go down. I added a 5th path to my ESXI server, and then I was able to get good read performance with 6 paths from my SAN but not 7. You get the drift, basically trying to suck a golf ball thru a straw. The write performance was always high (300-500) because it was sending the data to the SAN, but when reading from the SAN it dropped because there's 8 paths sending data but only 4 to receive it.

I don't think this would have translated into negative performance in a real world scenario, unless one process was calling for a ton of data from the SAN. I could be wrong

Has anyone seen this before with their Starwind setup? I mean it makes sense to me, just surprised I haven't seen more information on it about ensuring you have equal paths or close to it from host to SAN and vice versa. I've seen about having equal SYNC/iSCSI paths or bandwidth from the Starwind SAN's but nothing mentioned about coming from the virtual machine hosts themselves. This would mean at least for my setup, that I should have 8 iSCSI connections from each host.

Obviously, I'm at the borderline for 10GBe - as I think I mentioned earlier I just don't have that in the budget

[Bohdan (staff)]

Could you please clarify: on the ESXi host you are using separate vSwitches for each NIC. Right?
This case according to http://pubs.vmware.com/vsphere-51/topic ... -guide.pdf page 89
"If you use separate vSphere switches, you must connect them to different IP subnets."

Is it possible for you to reconfigure the environment and use (for example) 10.0.0.x/24, 10.0.1.x/24, 10.0.2.x/24, ... networks?
Like it is described in this manual http://www.starwindsoftware.com/images/ ... ere_v6.pdf
What will be the performance results?

[jmchristy]

Yes, I am using separate vSwitches for each network card. That was per the instructions of the VMware tech I was working with, who confirmed there is a bug with assigning multiple nics to a single Vswitch that causes the traffic to route incorrectly with an adapter reaches a link down state. It only affects my build of ESXI 5.0, and there is a patch for it I just decided to use the work around.

I will be upgrading to 5.1 soon, and when I do I will deploy my iSCSI network in an easier method probably assigning all network cards to one vSwitch and having multiple Vmkernels as opposed to having 4 vswitches, each with their own network card.

Thanks for posting links to those 2 white papers, first time I've seen those....looks like they are hot off the press!

[Bohdan (staff)]

Good. Please let us know about the results.