Looking for some peer support (5.6 performance problem)

[ggalley]

nbarsotti, if you have 10gig uplinks to a switch that does not support jumbo frames of 9k you will not be able to achieve 10gig through put.

If you have 10gig crossovers you will want to enable jumbo frames of 9k on those crossovers.

Thanks
Garrett

[nbarsotti]

Hi Anton,

1) I have 1 Windows 2008 R2 server with a dual port 10Gb NIC, and two ESXi hosts with dual port 10Gb NICs. The ESXi hosts connect to the Windows 2008 R2 server with cross-over cables (no switches). I am unaware of how to do any network throughput testing from a ESXi host. If you have any suggestion I would love to hear them.
2) Has Starwind found a way to swtich caching modes while leaving a target available? Or do I still need to delete my target and create a new one? Raid controllers have been able to swtich caching policies on the fly for years, why has it not been impleted by Starwind yet?
3) From my understanding Jumbo frames are a mandatory part of the 10Gb ethernet specification not but they are DEFINATELY NOT DEFAULT configuration for Windows 2008 R2 or ESXi 4.1u1. I don't know how to do pure network bandwidth testing from with in ESXi (open to suggestions). I have not enabled jumbo frames on my ESXi because the vSwitch that is connected to my 10Gb NICs is also connected to 1Gb nics that need to connect to my physical non-jumbo frame switch. I probably could enable jumbo frames on all the 10Gb NICs and the ESXi vSwitch without affecting my 1Gb ethernet, but my understanding was the jumbo frames was only good for 10-20% speed bumps. I'm seeing 50-60% lower speeds on my 10Gb.
4) Your are correct that my server running Starwind is different than the original thread starter. What is the same is that we both have server with VERY fast SSD based local storage, but when accessed via Starwind iSCSI over 10Gb ethernet show MUCH lower performance numbers.

[ggalley]

nbarsotti, have you checkedout iperf I believe they have both a linux and windows distro.

Thanks
Garrett

[nbarsotti]

Hi Garrett,
I have looked at iPerf but the problem is that virtually nothing can be installed or run from ESXi. iPerf can be run from the ESX service console but that is not an option with ESXi. This fall vSphere 5 is supposed to launch and there will be no ESX only ESXi, hopefully they will add some benchmarking troubleshooting tools to ESXi at that time.

Nick

[CyberNBD]

ESXi also has a service console it just isn't (or wasn't) officially supported.
For ESXi 3.x Press ALT-F1 on console, type "unsupported" (without quotes) followed by enter and you're in.
For ESXi 4.x you can enable tech support through the Troubleshooting Mode Options menu. Local Tech Support gives you console access using ALT-F1, Remote Tech Support gives you SSH access through your management IP. User/Pass are the same as console User/Pass.

Hi Garrett,
I have looked at iPerf but the problem is that virtually nothing can be installed or run from ESXi. iPerf can be run from the ESX service console but that is not an option with ESXi. This fall vSphere 5 is supposed to launch and there will be no ESX only ESXi, hopefully they will add some benchmarking troubleshooting tools to ESXi at that time.

Nick

[nbarsotti]

Hello CyberNBD, I am aware of the ESXi tech support mode and currently have SSH access into my ESXi servers. Have you ever run any application from inside that console other than built-in vmware tools? Have you used iPerf from within that console. If so I would love to know how you did that. Thank you.

[ggalley]

Ok I found my answer and figured I would post something about it.

After a month of learning what not to do and tweaking settings from BIOS to Every settings you can thing of on network cards. Learning how what I should be looking at when considering good performance.

If you are using a PCIE SSD drive running at 120000 IOPS.
Fusion IO/OCZ RevoDrive and others fall under this.

DO NOT USE STARWIND CACHING.

Atto Benchmark tool
This is the OCZ RevoDrive X2. The benchmark was run directly against the OCZ card from the local computer.

RevoInternalTest.jpg (65.13 KiB) Viewed 21352 times

This benchmark is against a basic virtual disk using starwind write-back caching 64/5000 ms.

The read performance drops off sharply.

RevoISCSIWithWritebackcaching.jpg (62.44 KiB) Viewed 21334 times

This benchmark is against a basic virtual disk using starwind with no caching.

RevoISCSIWithNoCahce.jpg (61.46 KiB) Viewed 21321 times

As for some other things I learned dealing with starwind and ISCSI.

Your syncronization channel cannot be configured with link aggregation. ISCSI can only use one channel at a time. you have to go with a 10GBe sync channel.

All ISCSI network connections running jumbo frames should have this registry key set.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\<Interface GUID>Entry: TcpAckFrequency
Value Type: REG_DWORD, number
Valid Range: 0-255
Set to 1.

You may also need to increase your PCIE Max read spead in the BIOS.

Thanks
Garrett

[anton (staff)]

Looks very impressive. Did you try to increase cache size and queue depth? 64MB is just smallest value and overlapped I/O of 4 is not close to anything real.

[anton (staff)]

1) I'm lost... Can you draw some dirty interconnecton diagram?

2) Not yet but we'll do it for sure with V6 and new Cache Manager. Sorry for it now ((

3) Yes, tuning cross-link has not much to do with other network infrastructure.

4) For SSDs you need to experiment with caching modes (WB Vs. WT Vs. No Cache). And also you need to tune cross-link making sure it does close-to-wire speed with iSCSI. Or all SSDs IOps are going to be lost (bad performance on tiny request sizes).

Hi Anton,

1) I have 1 Windows 2008 R2 server with a dual port 10Gb NIC, and two ESXi hosts with dual port 10Gb NICs. The ESXi hosts connect to the Windows 2008 R2 server with cross-over cables (no switches). I am unaware of how to do any network throughput testing from a ESXi host. If you have any suggestion I would love to hear them.
2) Has Starwind found a way to swtich caching modes while leaving a target available? Or do I still need to delete my target and create a new one? Raid controllers have been able to swtich caching policies on the fly for years, why has it not been impleted by Starwind yet?
3) From my understanding Jumbo frames are a mandatory part of the 10Gb ethernet specification not but they are DEFINATELY NOT DEFAULT configuration for Windows 2008 R2 or ESXi 4.1u1. I don't know how to do pure network bandwidth testing from with in ESXi (open to suggestions). I have not enabled jumbo frames on my ESXi because the vSwitch that is connected to my 10Gb NICs is also connected to 1Gb nics that need to connect to my physical non-jumbo frame switch. I probably could enable jumbo frames on all the 10Gb NICs and the ESXi vSwitch without affecting my 1Gb ethernet, but my understanding was the jumbo frames was only good for 10-20% speed bumps. I'm seeing 50-60% lower speeds on my 10Gb.
4) Your are correct that my server running Starwind is different than the original thread starter. What is the same is that we both have server with VERY fast SSD based local storage, but when accessed via Starwind iSCSI over 10Gb ethernet show MUCH lower performance numbers.

[ggalley]

Hello Anton,
I tried itterations of starind cache size up to 8000MB with the 5000ms setting

Even with the starwind cache up to 8000MB the reads would remain static around 200 MB/S.

I am not sure what you mean by queue depth?

overlapped I/O of 4 is not close to anything real. Anton you are correct I was trying to see what it would due under a crazy load put 50 VM's on something like this and you will start to see numbers like that.

I could be wrong but it appears when you create a RAM disk with starwind does it bypass all caching. Which totally makes sense once you think about it what exactly would you be caching.

I think these RAM disks fall into that same category.

I could be totally off base here but I think starwind cache uses a 512k to read from since that is where all my testing started to have read problems.

[anton (staff)]

1) Cache is not going to help much with reads as you need to make it "dirty" before using it.

2) "Queue Depth" setting inside ATTO Disk Benchmark.

3) Stress test scenario is a very good one! Please give it a try and let us know what happened. Numbers and pics please

4) You can cache RAM disks with StarWind. But it does make zero sense obviously.

5) No, for now StarWind uses fixed size 64KB cache line but we'll change it to variable ASAP.

Hello Anton,
I tried itterations of starind cache size up to 8000MB with the 5000ms setting

Even with the starwind cache up to 8000MB the reads would remain static around 200 MB/S.

I am not sure what you mean by queue depth?

overlapped I/O of 4 is not close to anything real. Anton you are correct I was trying to see what it would due under a crazy load put 50 VM's on something like this and you will start to see numbers like that.

I could be wrong but it appears when you create a RAM disk with starwind does it bypass all caching. Which totally makes sense once you think about it what exactly would you be caching.

I think these RAM disks fall into that same category.

I could be totally off base here but I think starwind cache uses a 512k to read from since that is where all my testing started to have read problems.

[georgep]

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\<Interface GUID>Entry: TcpAckFrequency
Value Type: REG_DWORD, number
Valid Range: 0-255
Set to 1.

where do u change this ? on starwind sans and also on WIndows initiators...vms etc ?

[anton (staff)]

You should play with this key on both target and initiator machines running Windows OS.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\<Interface GUID>Entry: TcpAckFrequency
Value Type: REG_DWORD, number
Valid Range: 0-255
Set to 1.

where do u change this ? on starwind sans and also on WIndows initiators...vms etc ?

[@ziz (staff)]

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\<Interface GUID>Entry: TcpAckFrequency
Value Type: REG_DWORD, number
Valid Range: 0-255
Set to 1.

where do u change this ? on starwind sans and also on WIndows initiators...vms etc ?

Yes, it should be changed on all windows servers involved in the configuration.

[corsari]

Hi to all of you

I've landed on this nice thread, because I was trying to understand one thing, please excuse me if that will be a bit off topic.

For some reason, I need to direct connect to each other two IBM servers with a 10 gigabit direct/crossover connection.

I've never used before the 10Gb NICs.
What I'm going to use are two IBM servers with this NIC.
On the same page, I can see they are available the following transceiver options
IBM 10 GbE SW SFP+ Transceiver
Brocade 10Gb SFP+ SR Optical Transceiver
QLogic 10Gb SFP+ SR Optical Transceiver
IBM/BNT® 10Gb SFP+ SR Optical Transceiver

Additionally, always in that page, they are listed the following cables
1 m IBM Passive DAC-SFP+ Cable
3 m IBM Passive DAC-SFP+ Cable

considering that the two servers will stay on the same desk, I suppose that the first transceiver option could be good for my case: 10 GbE SW SFP+ Transceiver, and, to stay comfortable, I may use the 3 m IBM Passive DAC-SFP+ Cable

But what about the peer-to-peer / crossover connection?
Sorry for my ignorance, but I don't understand if, should I search for a crossed Passive DAC-SFP+ Cable.

Generally speaking, are the 10Gb NICs "auto-crossing"? As said above, I need to connect two servers directly, peer-to-peer with a 10Gb connection.
Or should I search for a Passive DAC-SFP+ Crossed Cable?

Thank you for any answer/tip/help

Robert