Recommendations for a Large SAN for Virtualization

Wed Dec 15, 2010 6:58 pm

Start with doing huge file copy in a parallel processed to / from storage. No SAN installed. Tomorrow I'll ask guys to provide some dd scripts for you.

oxyi wrote:Sure, but I need to know if there is tool that I can use to stress test the IO and what's the setting for it.

oxyi · Wed Dec 15, 2010 10:37 pm

I created a 100Gb file on the one of the Areca 1680 raid card. 32.5TB Raid 60.
And I created the same file on C, and I did copy to each location at the same time.

687 MB/sec from C to the Raid

104 MB/ sec from Raid to C.

Not sure what I should be at look at , I can run IOmeter, if you can tell me the parameters.

Thu Dec 16, 2010 12:33 am

I/O meter does not really stress the system, dd app (running multiple copies) is. We'll provide you with a sample scripts tomorrow morrning (actually in a couple of hours from now). That should make the trick...

oxyi wrote:I created a 100Gb file on the one of the Areca 1680 raid card. 32.5TB Raid 60.
And I created the same file on C, and I did copy to each location at the same time.

687 MB/sec from C to the Raid

104 MB/ sec from Raid to C.

Not sure what I should be at look at , I can run IOmeter, if you can tell me the parameters.

Thu Dec 16, 2010 8:20 am

Attached

oxyi · Thu Dec 16, 2010 9:05 pm

Thanks for the script!

How about some instruction

and what I should look out for, thanks a lot !!

Fri Dec 17, 2010 8:13 am

Edit bat files specifying the right drive letter. Also if it is necessary you can change the other dd parameters like block size and count (http://www.chrysocome.net/dd).

The scripts work in the following way:
- show the time (t1)
- perform operation(s) with the dd: reads or writes some amount of data (D)
- show the time (t2)

To calculate the performance you need P=D/(t2-t1).

For example
echo. | time
dd --progress if=/dev/random of=E:\test.dat bs=256k count=38000
echo. | time

Means: show the time before and after dd operation, show progress of the dd operation, input file - generate random sequence, output file - E:\test.dat, block size - 256k, count=38000 - number of blocks to copy. 256k*38000=9728000k=9500M. Therefore dd writes 9500 megabytes. Divide this number by the (t2-t1) in seconds and you'll get the performance (MB/s).

oxyi · Fri Dec 17, 2010 2:47 pm

thank for the detailed explanation!! I will give it a try

Tue Dec 21, 2010 10:19 am

oxyi wrote:thank for the detailed explanation!! I will give it a try

Oxyi, please keep us updated on the progress.
Thanx

oxyi · Wed Jan 19, 2011 9:08 pm

Sorry, I am back.

Just did the example you set.

write big:
I have 221.29 MB/sec for the big files.

read big:
I have 1187.5 Mb/sec for reading that big file.

oxyi · Wed Jan 19, 2011 9:58 pm

does the result explains anything ?

Thu Jan 20, 2011 8:47 am

From those I can tell that the caching has not been enabled, the write speeds should definately be higher.

Thu Jan 20, 2011 8:52 am

Hi oxyi,

could you please provide us with more details:
1) What is the size of big file?
2) What are your testing steps: write big file, reboot, read big file?
3) Do you have any RAID configured and what is the stripe size?

Let me explain: 1),2) as far as you have Areca 1680 4Gb cache version and if the file to copy is not so huge, some part of it may be stored in the controller's cache and impact the further read test. So the best way to perform the tests is: write big file, reboot, read big file. 3) the difference between read and write performance is significant, it may be caused either by the reason I mentioned above or by RAID configuration. For example here is the description for RAID5:

http://en.wikipedia.org/wiki/Standard_RAID_levels
"RAID 5 implementations suffer from poor performance when faced with a workload which includes many writes which are smaller than the capacity of a single stripe. This is because parity must be updated on each write, requiring read-modify-write sequences for both the data block and the parity block. More complex implementations may include a non-volatile write back cache to reduce the performance impact of incremental parity updates."

BTW, have you also performed the network testing with NTTTCP or iPerf? What are the results?

Thu Jan 20, 2011 9:04 am

...also volume should be dismounted / mounted to ensure disk cache is clean and is not going to impact results.

oxyi · Thu Jan 20, 2011 3:40 pm

okay,

1) The size of the big file is 9500mb.
2) Testing step. I just wrote the big file and read it afterward.
3. Yes it is on Raid 60.

Raid Level Raid 6
Stripe Size 64KBytes
Block Size 512Bytes
Member Disks 11
Cache Mode Write Back
Tagged Queuing Enabled

I am not sure if I am doing it correctly, because I ran the the batch file on the server directly, do you guys want me to do it on a separate computer with iSCSI target mounting the volume ?

Also regarding to the caching, are you referring to the Write-Cache policy in the Device manager? If so, "Enable write caching on the device" is checked.

And no, I have not done any network testing yet. If you need me to do it, please let me know what need to be done.

Thanks

Thu Jan 20, 2011 5:10 pm

1. The clean test design is the following:
->Write test
->Array cache drop (reboot)
->Read test
->File size should be increased to ~30 Gb this will show more certain results of cache work

2.You definately need to test the network, download and install NTTTCP testing tool on all the machines which will be in the iSCSI environment. Test the network link from your iSCSI SAN to your client server.

After this - drop us the test results you've obtained (A screenshot of Ntttcp will be highly appreciated)