EXTREME Home Lab with Starwind

[anton (staff)]

Yes no idea what happened so far... Please zip StarWind logs and send them to us so we could check target-to-port binding. Thank you!

P.S. Could you please check Windows Event Log as well? Very much appreciated!

same target , no chaps no acl... basic stuff just to test performance...

Weird eh?

[georgep]

HI guys,


So I have these 4 ET quad nics. I first had the problem with them not being able to connect to ISCSI target from my starwind SAN. But other nics could connect just fine. So I tested them all and this "Loopbackup test" was failing. Couldn`t beleive it. So I took those 4 nics and tested in a different Asus motherboards. Guess what ? the test passed on all. So now I put one on a completely different ASUS MB and it passed of course and also no problems with ISCSI MPIO things....


So it seems to be a problem only under this server specific MB Asus z8pe-d18 boards. I do have 3 of these and tested under 2 of them with Windows with latest drivers for Intel ET but they all failed that loopback test and also couldn`t connect to my Starwind target.....


So the question is where does the problem really resides to ? Asus sZ8-18 server board ...it`s an expensive board.. and I do have 3 of these... or would it be to the intel ET cards themself ?.... Weird.....


In the end I have 3 servers with that board and want to make them work with Intel Quad ET to run ESXi 4.1 Update 1 for ISCSI traffic with MPIO and jumbo frames... Hopefully the bios update of the MB will help and maybe ESXi would have better luck with the Intel ETs....

Also the Asus z8pe-d18 has 2 Intel on board nics which work just great with ISCSI and my Starwind setup.....


Any answers would be appreciated.


Will update the post for everyone...

[anton (staff)]

This is "normal". We have a set of Intel 10 GbE NICs and they do full wire speed with TCP if inserted into "el cheapo" i5 mainboard. But the same cards do only around 6 Gbps if we use them with Intel-branded server boards powered by Xeon CPUs. So... Keep trying

P.S. Yes, cards are in mainboard compatibility list.

HI guys,


So I have these 4 ET quad nics. I first had the problem with them not being able to connect to ISCSI target from my starwind SAN. But other nics could connect just fine. So I tested them all and this "Loopbackup test" was failing. Couldn`t beleive it. So I took those 4 nics and tested in a different Asus motherboards. Guess what ? the test passed on all. So now I put one on a completely different ASUS MB and it passed of course and also no problems with ISCSI MPIO things....


So it seems to be a problem only under this server specific MB Asus z8pe-d18 boards. I do have 3 of these and tested under 2 of them with Windows with latest drivers for Intel ET but they all failed that loopback test and also couldn`t connect to my Starwind target.....


So the question is where does the problem really resides to ? Asus sZ8-18 server board ...it`s an expensive board.. and I do have 3 of these... or would it be to the intel ET cards themself ?.... Weird.....


In the end I have 3 servers with that board and want to make them work with Intel Quad ET to run ESXi 4.1 Update 1 for ISCSI traffic with MPIO and jumbo frames... Hopefully the bios update of the MB will help and maybe ESXi would have better luck with the Intel ETs....

Also the Asus z8pe-d18 has 2 Intel on board nics which work just great with ISCSI and my Starwind setup.....


Any answers would be appreciated.


Will update the post for everyone...

[georgep]

that makes no sense... why is that ? intel with intel stuff should work...

[anton (staff)]

I think the same. But we have what we have (c) ...

that makes no sense... why is that ? intel with intel stuff should work...

[georgep]

Will deal with that later. So finally decided to go with RAID50 instead of RAID 5 and instead of RAID10 on my 2 sans. I run couple of hard core test with sqlio on read and write both seq and random with 8, 32 and 64k. From the tests looks like RAID50 will be the best bet. I can post the results here if u guys want.

Anton also for ISCSI optimization should that Windows things be done on stardiwnd servers as well or just on the Windows initiators servers ?

[anton (staff)]

Yes, we want results (assuming you have hardware controller write back cache enabled and StarWind distributed HA cache enabled as well).

Sure. Both sides.

Will deal with that later. So finally decided to go with RAID50 instead of RAID 5 and instead of RAID10 on my 2 sans. I run couple of hard core test with sqlio on read and write both seq and random with 8, 32 and 64k. From the tests looks like RAID50 will be the best bet. I can post the results here if u guys want.

Anton also for ISCSI optimization should that Windows things be done on stardiwnd servers as well or just on the Windows initiators servers ?

[georgep]

Just got Starwind 5.7 installed yesterday on both SANs and did some testing. I love the performance monitor tab. Good enough for the begging of a hopefully hard core monitor section. So now I just tested from SAN2 5 regular targets performance. The weird things I discovered using MPIO RR with 4 nics from SAN1 to SAN2 targets is:
Even with no caching enable i get 230MB for seq read. With Cache with 256 , 512 and 1024 no big difference what so ever. With caching I should see the RAM use on that SAN2 go up no ? Didn`t see that.... and it was write back mode with 5000ms.
When I created a 3GB ramdisk iscsi target I saw double performance than regular but i could not pass more than 320MB per sec with 4 GB nics ET for ISCSI with MPIO.

What TCP/IP optimizations should I start doing on those SANS to achieve better performance ?

[anton (staff)]

1) Nice to hear about performance monitor. Please let us know what kind of values you'd like to see reported and we'd be happy to add them soon.

2) The whole idea behind cache is to represent just touched data faster then it (data) could be actually read from the medium. If you did not put anything into cache (and you did not as it's just a test with a random patter) it's not going to help or influence / mess with any results numbers...

3) With properly configured round robin MPIO you should expect something close to 400 MB/sec so your 320 MB/sec is not that bad at all. Start with checking how good single link performs (each of them). All should do 100MB/sec + and all should have the same / close performance numbers.

Just got Starwind 5.7 installed yesterday on both SANs and did some testing. I love the performance monitor tab. Good enough for the begging of a hopefully hard core monitor section. So now I just tested from SAN2 5 regular targets performance. The weird things I discovered using MPIO RR with 4 nics from SAN1 to SAN2 targets is:
Even with no caching enable i get 230MB for seq read. With Cache with 256 , 512 and 1024 no big difference what so ever. With caching I should see the RAM use on that SAN2 go up no ? Didn`t see that.... and it was write back mode with 5000ms.
When I created a 3GB ramdisk iscsi target I saw double performance than regular but i could not pass more than 320MB per sec with 4 GB nics ET for ISCSI with MPIO.

What TCP/IP optimizations should I start doing on those SANS to achieve better performance ?

[georgep]

Hey anton thanks for replying. I use at work EQ and love their monitoring interface. But again it`s a very good start for Starwind to have this tool. There`s a lot of things you guys can work on like emailing reports based on each target or whatever we need. That would be a good one.
Also if possible to break down the Total number of IOPs on read and write instead of displaying the total number of both. Exporting graphs into PDFs.. etc I`m sure many other guys can help with what we would like to see.

I am still having issues with SAN2 i guess it`s the RAID controller that the issue or maybe the backpanel of that case. Will change the whole backpanel to another one and if still problems will change the controller. I want to get this running ASAP... Invested over 25k into this lab and it`s still not running.. Not very happy.

Will post some pics for you guys here to see what I am talking about. Also I need a lot of help with understanding starwind. I am still very confused with my configuration. So I would really appreciate if you could let me know if that`s how it should be.

So right now I have this:
SAN1 - 15 HDDs WD green 2TB in RAID 50 plus one hot spare.
ISCSI1 192.168.1.111/24 connected to Dell 6448
ISCSI2 192.168.2.111/24 connected to Dell 6448
ISCSI3 192.168.3.111/24 connected to Dell 6448
ISCSI4 192.168.4.111/24 connected to Dell 6448
Heart Beat 192.168.200.1/24 connected to Dell 6448
SYNC CH 192.168.100.1/24 connected straight to SAN2


SAN2 - 15 HDDs Hitachi 2TB 7200rpm in RAID 50 plus one hot spare.
ISCSI1 192.168.1.112/24 connected to Dell 6448
ISCSI2 192.168.2.112/24 connected to Dell 6448
ISCSI3 192.168.3.112/24 connected to Dell 6448
ISCSI4 192.168.4.112/24 connected to Dell 6448
Heart Beat 192.168.200.2/24 connected to Dell 6448
SYNC CH 192.168.100.2/24 connected straight to SAN1

ESX1
ISCSI1 192.168.1.10/24 connected to Dell 6448
ISCSI2 192.168.2.10/24 connected to Dell 6448
ISCSI3 192.168.3.10/24 connected to Dell 6448
ISCSI4 192.168.4.10/24 connected to Dell 6448
ESX2
ISCSI1 192.168.1.20/24 connected to Dell 6448
ISCSI2 192.168.2.20/24 connected to Dell 6448
ISCSI3 192.168.3.20/24 connected to Dell 6448
ISCSI4 192.168.4.20/24 connected to Dell 6448
ESX3
ISCSI1 192.168.1.30/24 connected to Dell 6448
ISCSI2 192.168.2.30/24 connected to Dell 6448
ISCSI3 192.168.3.30/24 connected to Dell 6448
ISCSI4 192.168.4.30/24 connected to Dell 6448

Questions:
1. Under Starwind server Configuration- Network I should only list the NICs that are for ISCSI right ?
2. Let`s say i create this target thats HA. How is my host knowing that there`s a replica of that lun somewhere else ? Do I have to add on the ESX a vswitch with 4 nics and somehow it will see both target target 1 and target 2 that`s the replica ? How does it work really
3. Is it better to use with Starwind Physical Partitions rather than img in terms of performance. Should I fill them with 0 when created which I guess will make them thick ? Better perm ?

[georgep]

SAN1 RAID50 Local Speed test using

1. Crystal Disk Mark for all read/write seq and random creating a file of 4GB. See attachment

san1.JPG (47.59 KiB) Viewed 18689 times

2. SQLIO for all read/write seq and random 64k block creating a file of 5GB. Running multiple tests taking the lowest/average numbers
READ: Seq:
IOPS 8297
Average Latency: 3
MBs: 520

Random:
IOPS: 1500.35
Average Latency: 20
MBs: 95

WRITE: Seq:
IOPS: 4530
Average Latency: 6
MBs: 285

Random:
IOPS: 520
Average Latency: 60
MBs:32

SAN2 RAID50 Local Speed test using
SQLIO for all read/write seq and random 64k block creating a file of 5GB. Running multiple tests taking the lowest/average numbers
READ: Seq:
IOPS 10857
Average Latency: 2
MBs: 678

Random:
IOPS: 1840.23
Average Latency: 16
MBs: 115

WRITE: Seq:
IOPS: 4454
Average Latency: 6
MBs: 278

Random:
IOPS: 700
Average Latency: 45
MBs:43

[georgep]

From SAN1 ISCSI Initiator test to SAN2 RAM DISK target 2GB

I love SQLIO for testing and maximizing my throughput over the network. My tests from now on will be only SQLIO....that`s it.

No Windows TCP optimization, no jumbo frames.
4 NICS MPIO RR from SAN1 2k8 R2
Crystal Disk Mark
READ SEQ: 319MB/s
WRITE SEQ 259MB/s

ramdisk4nics.JPG (180.7 KiB) Viewed 18674 times

ramdisk4nics-2.JPG (197.56 KiB) Viewed 18674 times

SQLIO 64k testing different story got my nics all the way up.

ramdisk4nics-sqlio-1.JPG (188.7 KiB) Viewed 18669 times

-----------Results:
Thu 07/21/2011 16:32:53.77
********************** SEQUENTIAL 64kbs READ ********************
sqlio v1.5.SG
using system counter for latency timings, 2864716 counts per second
parameter file used: SQLIO_test_param.txt
file E:\testfile.dat with 4 threads (0-3) using mask 0x0 (0)
4 threads reading for 900 secs from file E:\testfile.dat
using 64KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
size of file E:\testfile.dat needs to be: 2097152000 bytes
current file size: 0 bytes
need to expand by: 2097152000 bytes
expanding E:\testfile.dat ... done.
using specified size: 2000 MB for file: E:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 6656.40
MBs/sec: 416.02
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 4
Max_Latency(ms): 99
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 2 57 12 3 1 1 0 0 0 0 0 1 2 6 9 5 1 0 0 0 0 0 0 0 0
Thu 07/21/2011 16:51:07.16
********************** RANDOM 64kbs READ *************************
sqlio v1.5.SG
using system counter for latency timings, 2864716 counts per second
parameter file used: SQLIO_test_param.txt
file E:\testfile.dat with 4 threads (0-3) using mask 0x0 (0)
4 threads reading for 900 secs from file E:\testfile.dat
using 64KB random IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 2000 MB for file: E:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 6654.54
MBs/sec: 415.90
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 4
Max_Latency(ms): 77
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 2 57 12 3 1 1 0 0 0 0 0 1 2 6 9 5 1 0 0 0 0 0 0 0 0
Thu 07/21/2011 17:07:07.11
*********************** RANDOM 64kbs WRITE ***********************
sqlio v1.5.SG
using system counter for latency timings, 2864716 counts per second
parameter file used: SQLIO_test_param.txt
file E:\testfile.dat with 4 threads (0-3) using mask 0x0 (0)
4 threads writing for 900 secs to file E:\testfile.dat
using 64KB random IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 2000 MB for file: E:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 7233.92
MBs/sec: 452.12
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 3
Max_Latency(ms): 202
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 3 22 60 12 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Thu 07/21/2011 17:25:07.19
*********************** SEQUENTIAL 64kbs WRITE *********************
sqlio v1.5.SG
using system counter for latency timings, 2864716 counts per second
parameter file used: SQLIO_test_param.txt
file E:\testfile.dat with 4 threads (0-3) using mask 0x0 (0)
4 threads writing for 900 secs to file E:\testfile.dat
using 64KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 2000 MB for file: E:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 7233.91
MBs/sec: 452.11
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 3
Max_Latency(ms): 204
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 2 26 53 17 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

[georgep]

What do you guys think ? 4 links with MPIO RR on a ramdisk 450MB.. that`s crazy.

Well. Now I am testing from SAN1 a target that`s created on SAN2 with all MPIO configured properly I get poor results on SEQ READ.

From SAN1 to SAN2 target.Basically from a windows host.
SQLIO for all read/write seq and random 64k block creating a file of 5GB.

READ: Seq:
IOPS:2268.20
Average Latency: 13
MBs:141.76

Random:
IOPS: 1696.80
Average Latency: 18
MBs:106.05

WRITE: Seq:
IOPS: 4257.23
Average Latency: 7
MBs: 266.07

Random:
IOPS: 699.84
Average Latency: 45
MBs:43.74


______________________________________
From ESXi 4.1 Update 1 with MPIO configured properly RR. Target on SAN2 with 64mb cache. Windows 2008 R2 VM. Results:
READ: Seq:
IOPS
Average Latency:
MBs:

Random:
IOPS:
Average Latency:
MBs:

WRITE: Seq:
IOPS:
Average Latency:
MBs:

Random:
IOPS:
Average Latency:
MBs:
______________________________________
From ESXi 4.1 Update 1 with MPIO configured properly RR. RAMDISK Target on SAN2 of 5GB. Windows 2008 R2 VM. Results:

READ: Seq:
IOPS
Average Latency:
MBs:

Random:
IOPS:
Average Latency:
MBs:

WRITE: Seq:
IOPS:
Average Latency:
MBs:

Random:
IOPS:
Average Latency:
MBs:

__________________________________________
Disabled tcpackfrequency=1 to all ISCSI nics on both SANs. Run 2 tests from SAN1 again against SAN2 target with these amazing results:
It looks like SEQ READ got increased by 50MBs but random read is extremely slow as well as write both seq and random
READ: Seq:
IOPS 3066.68
Average Latency: 9
MBs: 191.66

Random:
IOPS: 153.88
Average Latency: 207
MBs:9.61

WRITE: Seq:
IOPS: 2953.46
Average Latency: 10
MBs: 184.59

Random:
IOPS: 679.96
Average Latency: 46
MBs:42.49

-------------------------------------------
So now I just run from another WIndows initiator box without changing TcpAckFrequency. But the SAN2 still has TcpAckFrequency disabled.
THESE RESULTS LOOK MUCH BETTER. THIS KINDA MEANS THAT ITS ONLY BETTER TO DISABLE IT ON THE TARGET BUT NOT THE INITIATOR

READ: Seq:
IOPS 2948.73
Average Latency: 10
MBs:184.29

Random:
IOPS: 1776.38
Average Latency: 17
MBs: 111.02

WRITE: Seq:
IOPS:4251.98
Average Latency: 7
MBs: 265.74

Random:
IOPS: 698.79
Average Latency:45
MBs:43.67

[georgep]

how much should starwind be able to pump on ISCSI with MPIO from SAN2 targets(without HA) ? knowing you guys saw the results of the local speeds ? .

You can see clearly that the write speed on local is 278MB/s seq and on iSCSI traget is 266MB/s seq. That`s perfect
But you can see clearly the issue with seq READS. on local 678MB/s and on iSCSI target only 141.76 MBs.

Please help guys. What do you think it`s the issue ?

[georgep]

A few pics of the lab.