i/o read/write caching

Thu Apr 24, 2008 7:55 am

No. iSCSI should be faster compared to the SMB. SMB does extensive caching however so if you'll use the tool allowing to check "absolute" performance at current moment you'll see SMB does copy VERY fast (b/c it goes to the local cache) and and after this "overboost" copy slows down. For iSCSI (and any SAN) copy process goes very stable on ALL the way file is copied. We've checked this with the huge 10GB+ files some time ago.

BartHermans wrote:Anton,

We've done the tweaking of the TCP stack as you described for a better network performance. We've done some measuring : a 1GB file copy to a normal share on the StarWind server and a 1GB file copy to an ISCSI disk (image file) on the StarWind server. The file copy to the share is twice as fast compared to the iSCSI copy. The Gbit NIC is going to 45% while copying to the share and 20% using the iSCSI copy.

The iPerf utililty gives us a speed between 800 and 900 Mbits/sec.

Do you have any remarks about our measurments ? Are they normal ?
Today we're going to put the first StarWind Server in our "acceptation labo" to be used with ESX 3.5

Regards.
Bart

BartHermans · Thu Apr 24, 2008 8:14 am

Anton,
We're aware of the extensive SMB caching, but also with a new file (not copied yet) the time that the copy is twice as fast on a share on the same server. But maybe I'll have to do the test again to be sure.

The copy to the iSCSI disk is infact stable on ALL the way file is copied.

PS.: our iSCSI disk image is about 1TB big. Is there a difference between a big or a small disk image in speed ?

Regards.
Bart.

Thu Apr 24, 2008 7:36 pm

No. There's no difference between the volume sizes. But there's a huge difference between the file sizes you copy. Small ones are cached well (most of the content goes to the local RAM). Big ones cannot be cached effectively (disk cache size will be full soon). Use huge files or truncate RAM size with the switch on the client machine.

Another question. I guess you use raw images (no IBV thing, no unallocated volumes). Right?

BartHermans wrote:Anton,
We're aware of the extensive SMB caching, but also with a new file (not copied yet) the time that the copy is twice as fast on a share on the same server. But maybe I'll have to do the test again to be sure.

The copy to the iSCSI disk is infact stable on ALL the way file is copied.

PS.: our iSCSI disk image is about 1TB big. Is there a difference between a big or a small disk image in speed ?

Regards.
Bart.

drtellurian · Fri Apr 25, 2008 8:07 am

Anton,

You bring up this point. Is there better (or worse) performance with one type of image vs. another? We are using 10GB NICs with 9000 byte MTU to the dedicated iSCSI server with IBV volumes in testing and the speed doesn't seem as fast as it should be. This is a dedicated box with no other software and SAS hardware RAID6 with 14 active spindles. Any suggestions would be appreciated. Our files being copied to the iSCSI target for backup purposes are also 10s to 100s of GB each. We would like to use the 32GB of RAM in the server for write caching if possible. Thanks!

-Robert

BartHermans · Fri Apr 25, 2008 1:06 pm

Anton,
Our first performance tests were bad, but we've found the bottleneck. We used a RAID5 set with 3 500GB disks. After tuning the NIC's with the tool TCPOptimizer (measuring with iPerf) and changing the RAID5 to a RAID0 we can finally talking about performance.
The bottleneck was NOT StarWind but the RAID set performance.

The microsoft utility for measuring the (iscsi) disk speed 'sqlio' :
sqlio -LS -kW -s20 -b64 -fsequential -Fparam.txt (=write test on F disk)
sqlio -LS -kR -s20 -b64 -fsequential -Fparam.txt (=read test on F disk)

RAID5 : Write 15MB/s and Read 112MB/s
RAID0 : Write 80MB/s and Read 80MB/s

Next week we will optimize ESX3.5 and try to finetune the disk and NIC interface to get the best out of it. So our future design will be a RAID10 for performance reasons.

Fine weekend.
Bart.

Sat Apr 26, 2008 9:56 am

You'll get wire speed with the GbE but with 10 GbE you'll get 250-400 MB/sec maximum. We've tested multiple initiators and targets on the different machines against different NICs - result is pretty much the same. So there will be NO such a benefit with GbE -> 10 GbE like it was with 100 Mb -> GbE migration. This is number one

IBV (renamed to the CDP in the next release) does a lot of the hoursework to keep track of the snapshots, individual writes etc. So it will not be as fast as raw image or partition-based iSCSI target. This is number two...

drtellurian wrote:Anton,

You bring up this point. Is there better (or worse) performance with one type of image vs. another? We are using 10GB NICs with 9000 byte MTU to the dedicated iSCSI server with IBV volumes in testing and the speed doesn't seem as fast as it should be. This is a dedicated box with no other software and SAS hardware RAID6 with 14 active spindles. Any suggestions would be appreciated. Our files being copied to the iSCSI target for backup purposes are also 10s to 100s of GB each. We would like to use the 32GB of RAM in the server for write caching if possible. Thanks!

-Robert

Sat Apr 26, 2008 9:58 am

It's very bad idea to use disk-based location for performance tuning. You need to start with the RAM-disk based iSCSI target and tune network before you go with the real-world configurations.

For 112MB/sec - sounds like GbE limit...

BartHermans wrote:Anton,
Our first performance tests were bad, but we've found the bottleneck. We used a RAID5 set with 3 500GB disks. After tuning the NIC's with the tool TCPOptimizer (measuring with iPerf) and changing the RAID5 to a RAID0 we can finally talking about performance.
The bottleneck was NOT StarWind but the RAID set performance.

The microsoft utility for measuring the (iscsi) disk speed 'sqlio' :
sqlio -LS -kW -s20 -b64 -fsequential -Fparam.txt (=write test on F disk)
sqlio -LS -kR -s20 -b64 -fsequential -Fparam.txt (=read test on F disk)

RAID5 : Write 15MB/s and Read 112MB/s
RAID0 : Write 80MB/s and Read 80MB/s

Next week we will optimize ESX3.5 and try to finetune the disk and NIC interface to get the best out of it. So our future design will be a RAID10 for performance reasons.

Fine weekend.
Bart.

MattBoB · Thu May 08, 2008 1:08 pm

Going back to my original question, if I tick the "Use file system buffering" option in the latest version 3.5.3 what does it do exactly (read, write caching?), and is there any value in using it for general applications?

I'm still running 3.5.0 at the moment as it seems to be running OK and I don't use IBV so I'm not sure the latest version offers anyhting new for me?

Thu May 08, 2008 8:49 pm

Nothing. This option is DISABLED (unless you'll apply some extra switch).

V4.0 will have own caching and FS caching dropped completely.

MattBoB wrote:Going back to my original question, if I tick the "Use file system buffering" option in the latest version 3.5.3 what does it do exactly (read, write caching?), and is there any value in using it for general applications?

I'm still running 3.5.0 at the moment as it seems to be running OK and I don't use IBV so I'm not sure the latest version offers anyhting new for me?

MattBoB · Tue May 13, 2008 1:19 pm

What is the expected time frame for version 4.0 release?

aaron (staff) · Wed May 14, 2008 9:06 am

Mid-summer 2008 I think.

MattBoB wrote:What is the expected time frame for version 4.0 release?

BartHermans · Sat May 24, 2008 2:47 pm

Hoi Anton,

We bought us a StarWind version and now we're using it with ESX 3.5 (client). It works well as I already mentioned.

I've found battery backuped RAM drive (PCI slot) for PC's

http://www.anandtech.com/tradeshows/sho ... i=2431&p=5

The physical limitation is 4 memory modules and it's acting as a SATA disk.

Just for your information.

Regards.
Bart.

Sat May 24, 2008 7:48 pm

Thank you for keeping us updated

Nice solution but I think for most of the people UPS-backuped PC with A LOT of RAM should work just great.