EXTREME Home Lab with Starwind

Tue Jul 26, 2011 7:48 am

how much should starwind be able to pump on ISCSI with MPIO from SAN2 targets(without HA) ? knowing you guys saw the results of the local speeds ? .
It depends on the network speed. For 2 gig (2x1Gbps NICs MPIO) it should be around 220 megs.

Have you updated all drivers on your NICs (I assume that you are using Intel, correct?)? Is there any switch in topology?

georgep · Tue Jul 26, 2011 1:33 pm

Yes there is a switch. I posted already the results from SAN1 to SAN2 ramdisk which maximized the network 450MB/s over ISCSI. But it`s the the same for the targets with image drives on that local SAN2....

Why is my READ suffering so much ?

Tue Jul 26, 2011 3:47 pm

Are you sure that switch cannot be a problem? I mean the cache can be fulled and cause the performance issues... ?

georgep · Tue Jul 26, 2011 3:56 pm

I dont think so because when I run same test same size 5GB on a target that`s a ramdisk it`s killing my network using all 4 Gb nics to 98% and giving me results of 450MB.. i posted them up there... So it must be something else... I am doing lots of testing and trying to figure it out. Because TCPACKFREQUENCY disabled to 1 does more damage to RANDOM READ AND WRITE also dicresses performance on SEQ WRITE... but it does indeed increase performance on SEQ READ by 50MB/s in my case

Wed Jul 27, 2011 8:43 am

Do you have equivalent settings applied to both initiator and target sides of iSCSI stack?

georgep wrote:I dont think so because when I run same test same size 5GB on a target that`s a ramdisk it`s killing my network using all 4 Gb nics to 98% and giving me results of 450MB.. i posted them up there... So it must be something else... I am doing lots of testing and trying to figure it out. Because TCPACKFREQUENCY disabled to 1 does more damage to RANDOM READ AND WRITE also dicresses performance on SEQ WRITE... but it does indeed increase performance on SEQ READ by 50MB/s in my case

georgep · Wed Jul 27, 2011 3:06 pm

Hey anton,

Yes I applied the same settings to both. Then only to target , then only to initiator. It looks like when the target has the TCPACKFrequency to 1 then it`s the speed boost on SEQ READ by 50MB extra from 140MB let`s say to 190MB but some weird problems sometimes happend with RANDOM READ goes down from 100 to 8MB...
It`s weird sometimes it does go down sometimes it doenst... and I run over 10 test and saw the behaviour. Next will try jumbo frames with no TCP setting see what happends and then will play more with TCP settings again...

Wed Jul 27, 2011 4:10 pm

Are you sure that this is not hardware related issue? I'm doubting because you`ve mentioned "jumpy behaviour".

Mon Aug 08, 2011 9:34 pm

So what was the end of the story? Did you manage to have stable results with StarWind or any other software target? Please let us know as we're interested

georgep wrote:Hey anton,

Yes I applied the same settings to both. Then only to target , then only to initiator. It looks like when the target has the TCPACKFrequency to 1 then it`s the speed boost on SEQ READ by 50MB extra from 140MB let`s say to 190MB but some weird problems sometimes happend with RANDOM READ goes down from 100 to 8MB...
It`s weird sometimes it does go down sometimes it doenst... and I run over 10 test and saw the behaviour. Next will try jumbo frames with no TCP setting see what happends and then will play more with TCP settings again...

georgep · Wed Aug 10, 2011 6:30 pm

Hi everyone,

I took a small break with new updates for my SAN2 which is the fastest one.
Upgrade from 6GB RAM TO 24GB RAM DDR3( lots of ram for Starwind cache)
Upgrade to the raid controller from Adaptec with 256MB cache to Areca top of the line ARC-1880dix-24 with 1 GB cache by default upgradeable to 4GB later on....

So the new test results are:

SAN2 RAID5 -15 Hitachi 7200 rpm drives Local Speed test using
SQLIO for all read/write seq and random 64k block creating a file of 5GB. Running multiple tests taking the lowest/average numbers
READ: Seq:
IOPS 24242.66
Average Latency: 0
MBs: 1515.16

Random:
IOPS: 2808.35
Average Latency: 10
MBs: 175

WRITE: Seq:
IOPS: 21324.70
Average Latency: 1
MBs: 1332.79

Random:
IOPS: 507.71
Average Latency: 62
MBs:31

--------------------------------
SAN2 RAID6 -16 Hitachi 7200 rpm drives Local Speed test using
SQLIO for all read/write seq and random 64k block creating a file of 5GB. Running multiple tests taking the lowest/average numbers
READ: Seq:
IOPS 26128.85
Average Latency: 0
MBs: 1633.05

Random:
IOPS: 2922.78
Average Latency: 10
MBs: 182.67

WRITE: Seq:
IOPS: 20887.63.
Average Latency: 1
MBs: 1305.47

Random:
IOPS: 536.34
Average Latency: 59
MBs:33.52

Fri Aug 12, 2011 3:10 pm

Looks good! Can we have "redirected" numbers used thru StarWind iSCSI mapping?

georgep · Wed Aug 24, 2011 12:57 am

so decided to go with RAID6 all 16 drives as per local speed it`s fast and also has 2 parity drives. Now I did a quick test from a Windows initiator to a target with no cache on starwind and indeed I maximized all quad nics for both read and write sequential. I am very happy with this new top of the line controller. Now I also did a quick test from ESXi 4.1 Windows VM and results are not that great. I am playing more with optimization and will post results soon.

Stay tunned I did some changes and from ESXi 4.1 VM I get huge performance improvements... MPIO RR set to 1 instead of 1000 IOPS huge difference.

georgep · Wed Aug 24, 2011 10:26 pm

Ok guys it`s all good now. My SAN2 is achiving the results I wanted. I manage to 3xtimes performance from Windows VM Guest only by changing the MPIO RR from 1000 IOPS to 1 IOPS.

Results: From Windows ISCSI Initiator to SAN2 Running RAID6 with 16 drives. No optimization what so ever.
READ: Seq:
IOPS 7203.74
Average Latency: 3
MBs: 450.23

Random:
IOPS: 2561.96
Average Latency: 11
MBs: 160.12

WRITE: Seq:
IOPS: 7233.25
Average Latency: 3
MBs: 452.07

Random:
IOPS: 521.95
Average Latency: 60
MBs:32.62

----------------------
Results: From ESXi 4.1 Host VM to SAN2 Running RAID6 with 16 drives. No optimization what so ever.
READ: Seq:
IOPS 2116.33
Average Latency: 14
MBs: 132.27

Random:
IOPS: 1539.87
Average Latency: 20
MBs: 96.24

WRITE: Seq:
IOPS: 2171.76
Average Latency: 14
MBs: 135.73

Random:
IOPS: 344.95
Average Latency: 92
MBs:21.55

----------------------
Results: From ESXi 4.1 Host VM to SAN2 Running RAID6 with 16 drives. With MPIO RR IOPS set to 1 instead of default 1000
READ: Seq:
IOPS 7149.73
Average Latency: 3
MBs: 446.85

Random:
IOPS: 1766.51
Average Latency: 17
MBs: 110.40

WRITE: Seq:
IOPS: 7154.84
Average Latency: 3
MBs: 447.17

Random:
IOPS: 394.63
Average Latency: 80
MBs:24.66

As you guys can see it made a huge difference just by changing that. I might play more to increase the numbers a lil bit more for random and stuff. Also need to specify VMtools were installed and also the ESXi host had disabled ACK...

Wed Aug 24, 2011 11:59 pm

Indeed numbers are all fine now. Thank you very much for keeping us updated. Good job!