Poor speed so far on new servers...

[adns]

Unfortunately when I search for "speed" the forum's search throws it out as too common of a term so forgive me is this has been discussed somewhere!

I setup a new HP DL180 with dual quadcore XEON 5520, 12 1TB SATA drives in RAID 10, Windows Server 2008 (installed on it's own RAID 1), Intel Gigabit NICs and Starwind Enterprise with an Image File Device. I'm testing with an HP DL320 running VMware VSphere. Setup is a breeze, kudos to Starwind for that! I orginally had my NICs teamed on both ends with jumbo frames enabled connected to a Cisco 3750 switch and I enabled Round Robin MPIO in VSphere. Well the performance was very poor and even after applying the registry tweaks outlined here I was getting nowhere fast. So I tore down the NIC teaming and just did a crossover cable between the servers and I'm still getting the same speeds.

So I've come to the conclusion that either my testing methods are faulty or I've setup something just plain wrong! My primary testing has been with HD Tune Pro 3.5. I've used this tool for many years to help me determine the theoretical capacity of my servers and I've never seen speeds this slow, but I've also never used iSCSI before! When I run a benchmark test I get about 30.5 MB/sec and my iSCSI NIC never goes above 25% utilized. See the screenshots.





Any suggestions on what I should try to do to get better performance? Or is HD Tune not a true indicator of performance? Here is the benchmark test run from a weak SATA drive on a desktop computer, and I typically get about 2-3 times these speeds on a direct attached SATA RAID 10 on my servers.

[Aitor_Ibarra]

I've got a few suggestions...

1) This is probably not the issue if they are recent nics, but what interface are your Intel NICs running on? It's just your speed is suspiciously close to what a gigabit PCI 33Mhz card would do (as opposed to PCIe). If a gigabit ethernet connection is via a slow interface, like old style PCI or USB2, it won't reach gigabit speeds...

2) To rule out starwind being the issue, can you try a simple fileshare and network copy? If this goes fast then the issue is probably with Starwind (although somehow it's specific to you!)

3) Have you tried testing against a Starwind RAM disk? If this goes faster then the problem may be with the drives or RAID card or something.

4) Another stupid suggestion - check the cable, gigabit needs all four pairs of wires to be working reliably...

5) Another thing you can try is to set up the MS initiator on the starwind box so that it points to itself, bypassing the NIC. Say you have windows on c:\, starwind offering up an img target on d:\, use ms initiator to mount that target as e:\ and then do a copy from c:\ to e:\

[adns]

1) This is probably not the issue if they are recent nics, but what interface are your Intel NICs running on? It's just your speed is suspiciously close to what a gigabit PCI 33Mhz card would do (as opposed to PCIe). If a gigabit ethernet connection is via a slow interface, like old style PCI or USB2, it won't reach gigabit speeds...

These are 2 port Intel Pro 1000PT PCI-e cards.

2) To rule out starwind being the issue, can you try a simple fileshare and network copy? If this goes fast then the issue is probably with Starwind (although somehow it's specific to you!)

When I do a normal Windows filesharing I am pulling about 65%-75% utilization on the NIC.

3) Have you tried testing against a Starwind RAM disk? If this goes faster then the problem may be with the drives or RAID card or something.

I created a RAM disk and the test was better: 50MB/sec (as expected the access time was stellar!) The NIC utilization went up to 42% which is a hopeful sign. Does that mean my disk is the culprit? I ran HD Tune Pro locally on that volume I get 90MB/sec (which is actually slower than what I was expecting but not terrible).

4) Another stupid suggestion - check the cable, gigabit needs all four pairs of wires to be working reliably...

I've tried a few cables and I'm not seeing any layer 1 or layer 2 errors on the switch. I'm assuming this is irelevent now though since I've confirmed that I can pull decent speeds via Windows filesharing.

5) Another thing you can try is to set up the MS initiator on the starwind box so that it points to itself, bypassing the NIC. Say you have windows on c:\, starwind offering up an img target on d:\, use ms initiator to mount that target as e:\ and then do a copy from c:\ to e:\

When I go through the Microsoft iSCSI initiator right back to the StarWind target on the same server I get 50MB/sec.

So here is a summary

Locally I get ---------------------------------------------------------- 90MB/sec
Using the MS iSCSI initiator through StarWind locally I get ----- 50MB/sec
Using the RAM Drive from another client I get ------------------- 50MB/sec
Using an image file device from another client I get ------------ 30MB/sec

So there is definitely a performance hit going through StarWind. It doesn't seem to be related to the NIC as I was able to finally break the 25% utilization barrier I was experiencing by using a RAM drive. Any other suggestions?

[Aitor_Ibarra]

Not really, you've got me stumped. You should be able to do more than 50MB/sec with a ram drive though - the fact that's not up at 100MB/sec suggests there's something wrong, somewhere, that's not to do with the hard drives.

HDTune is doing sequential i/o, right? Not that it should make much of a difference with a RAM drive.

If you were using 10GbE I would suggest turning off the Windows firewall, which I had to do to get the best performance, but at gigabit I doubt it will have much of an effect unless your cpus are very slow.

[adns]

I've done some additional testing and a few things that I thought were noteworthy:

When I run two simultaneous hard drive benchmarks from 2 separate VMs, both tests run slower rather than my NIC utilization increasing. I can't get past 25% utilization (except with a RAM drive).

I installed the Microsoft iSCSI target 3.2 and tested on that target and I'm getting roughly the same throughput results as the StarWind target.

[Robert (staff)]

I would recommend trying all the variants one by one - test using one NIC only, use different kinds of storage controllers and storage devices if possible.

Thanks

[mjtir]

Unfortunately I'm facing the same issue with a 3ware sata raid 0 8 sata disks , win 2008 , dual intel pro , tyan amd motherboard.

I do not dare to recommend the product as an ISCSI solution to my company for core implemantation.

[Robert (staff)]

Fair enough.

Would you mind submitting us a support request at http://www.starwindsoftware.com/support so we could take a closer look at the problem?

thanks

[adns]

I have submitted a ticket and I will post our findings on the matter here. Thank you!

[mjtir]

I must admid i had the same issues also with other software vendors , but since starwind is the top provider (and my choice as well ) it would be nice to find a solution.

[Robert (staff)]

These types of issues are always related to different things. Anything may be the reason, so everytime you experience this type of issues, please submit your requests at http://www.starwindsoftware.com/support and we will do our best to find a solution.

Thanks.

[adns]

I spoke to someone at Starwind about my setup and the problems I've been experiencing and they said they would test things in their lab and get back to me. But having an impending project on the horizon I decided to keep tweaking my setup while I wait for Starwind to get back to me.

So I tore the whole thing down and rebuilt my lab from scratch using Windows Server 2008 R2 this time. Then I started experimenting with the stripe size of the RAID 10 and the allocation size of the NTFS partition on my storage volume. After many tests I found that I was consistently getting good results with a 64k stripe size and 64k allocation size. So my local performance is now improved by about 15% - 20%. I then installed Starwind and served up a new image file device and connected to it via VMware but there was no improvement in my iSCSI speeds. I tweaked this for a full day and just couldn't get any better performace out of it so I decided to test using the exact same setup but with NFS instead of iSCSI and I got roughly 2 times the performance. During the test I noticed that my NIC was only about 45% utilized so I still have plenty of opportunity to try and increase this speed!



So at this point I'm not sure if it's VMware's iSCSI initiator or Starwind's target that's the problem. Tomorrow I plan on testing with a Windows iSCSI initiator to see what results I get.

I've considered buying two 10Gbe nics and connecting the Starwind box to a VMware host via crossover to see if that effects performance at all, but that's a decent chunk of money to throw at a lab just to test this one piece... But as my patience wanes I may just do it!

[mrgimper]

I must admit I can't seem to break the ~30MB/sec & 25% NIC utilization either.

I'm not using anything special. An HP ML115 G5 as an ESXi 4 host, and another running 2008 x64 Server with StarWind installed. Running HDTune Pro 3.50 in an x64 2008 Server VM against a 20GB RDM iSCSI Target yeilds similar results to yours. I'm not striping on the Starwind server, just a single disk.

[Robert (staff)]

Have you tried using Spotlight by Quest Software to monitor? Just curious what it would show.

Thanks

[adns]

I have it installed but I'm not familiar with the program. I assume I start a local connection and then leave it running while I run my HD tests and then look at the Spotlight results. Should I install this on the iSCSI client or host or both? Can you give me some direction on what key indicators I should be looking for in regards to bottlenecks?