poor performance Hyper-V over SMB share

[genko2012]

Hi, i've deployed test Hyper-V cluster over SMB share. Everything works fine, but i couldn't explain poor performance when i'm copying large file to network share. The speed is so slow (aprox 80 Mb/s)
Picture of current config

config

cluster.jpg (132.92 KiB) Viewed 14478 times

four intel platforms are the same type, 2 for hyper-v cluster, 2 for storage cluster
each Hyper-v node has 192 Gb of RAM, 2 CPU's (6 Cores), RAID 1 HDD 7200 (for OS)
each Storage node has 16 Gb of RAM, 1 CPU's (4 Cores), RAID 1 SSD's (for OS) (build in controller) and RAID 6 (8 SSD's) Intel hardware controller PSBL080 for storage
for storage network we use 10 Gbps switches.
OS on all servers is Windows 2012.
Tests:
1. Copy 3Gb file within one storage server (from C: drive (OS RAID 1) to D: drive (Storage RAID 6) - Speed - 800 mb/s
2. Copy 3Gb file between storage servers shares ( D: drive (Storage RAID 6) of one storage server to D: drive (Storage RAID 6) of another storage server using storage network - speed - 730 Mb/s
ok, creating HA file cluster, creating \\FS-SHARE to store vhdx files..
3. copying file from any Hyper-V server to \\FS-SHARE using storage network -Speed - only 80 Mb/s
4. copying file from any storage server C: drive (OS RAID 1) to \\FS-SHARE using storage network -Speed - only 85 Mb/s

Any ideas?

just tested copy speed from hyper-v node RAM disk to \\FS-SHARE - Speed - only 80 Mb/s
copy speed from hyper-v node RAM disk to share on D: drive (Storage RAID 6) of one storage server max 450 Mb/s min 200 Mb/s using storage network

[Anatoly (staff)]

I have two questions:
-what utility have you used to benchark the speed?
-had you any chance to look at our Benchmarking Servers guide?

[genko2012]

Hi, you can see copying speed in copy menu of Windows 2012
Yes, i've read this guide, but performance tuning is not simple..
for network tests i used iometer, network speed is excellent, i could load storage network up to 20 Gbps (we use NIC teaming for each connection)
Sync network - 3 x 1Gbps in team - by using iometer i can load it up to 2,9 Gbps.
it seems network is not an issue.
Ok, i will try to use iometer to check another components of the system.
And another question.. Do you have test file config for IOMeter?
Thanks a lot.

[Anatoly (staff)]

I think it`ll be better if I`ll just provide you with the list of Iometer test that should be performed:
• Local storage on node1
• Local storage on node2
• RAM disk that is running on node1
• RAM disk that is running on node2
• Basic Image target that is running on node1
• Basic Image target that is running on node2

Every test should have following configurations:
• 32K 100% read sequential, # of Outstanding IOPS= 32 (or 64)
• 32K 100% write sequential, # of Outstanding IOPS= 32 (or 64)
• 32K 100% read random, # of Outstanding IOPS= 32 (or 64)
• 32K 100% write random, # of Outstanding IOPS= 32 (or 64)
• 64 100% read sequential, # of Outstanding IOPS= 32 (or 64)
• 64 100% write sequential, # of Outstanding IOPS= 32 (or 64)
• 64 100% read random, # of Outstanding IOPS= 32 (or 64)
• 64 100% write random, # of Outstanding IOPS= 32 (or 64)

[genko2012]

Hi, Thanks for reply
Just some questions about these tests

Every test should have following configurations:
• 32K 100% read sequential, # of Outstanding IOPS= 32 (or 64)
• 32K 100% write sequential, # of Outstanding IOPS= 32 (or 64)
• 32K 100% read random, # of Outstanding IOPS= 32 (or 64)
• 32K 100% write random, # of Outstanding IOPS= 32 (or 64)
• 64K 100% read sequential, # of Outstanding IOPS= 32 (or 64)
• 64K 100% write sequential, # of Outstanding IOPS= 32 (or 64)
• 64K 100% read random, # of Outstanding IOPS= 32 (or 64)
• 64K 100% write random, # of Outstanding IOPS= 32 (or 64)
this are requirements for one test?
i mean i need to create user's profile for test that consist of 8 Workers, each of them uses one of this access spec? because if i use only one Worker it couldn't use all access spec the same time.. Am i right? 64 100% write random, # of Outstanding IOPS= 32 (or 64) - this means 64 K?
Please correct me, if any misunderstanding, because it's too important
Thanks

[Anatoly (staff)]

I understand that this is really important, bellieve me, and that is why we are here

Yes these are the specifications for each test, and no, Number of Outstanding IOPS are not the bytes, this value is the number of parallel queues that will be sent to the storage.

I hope it was helpful.

[genko2012]

Sorry, i mean this 64 100% read sequential, # of Outstanding IOPS= 32 (or 64)
i understood about last numbers
Thanks

[Anatoly (staff)]

Oh yes, exactly! Sorry for the mistype.

I will edit my previous post to do not confuse the new guys in the community.

[genko2012]

Ok, let’s start

Each storage server has 2 drives
C: drive RAID 1 (build in RAID Controller) – 2 x 120 GB SSD Intel 520 Series
D: drive RAID 6 (Intel RSBL080 Hardware Controller) – 8 x 480 GB SSD Intel 520 Series

3 network connections total
1) 1 Gbps for management (domain access)
2) 2 x 10 Gbps in team for storage private network
3) 3 x 1 Gbps in team for Sync (cross connection directly from server to server)

I used iometer to load network
Test config intel fileserver pattern, 20 Workers total
Some screenshots
First Storage Network load

01_TEST-STORAGE-NET.PNG (46.39 KiB) Viewed 14099 times

Second Sync Network load

02_TEST-SYNC-NET.PNG (59.37 KiB) Viewed 14095 times

It seems network is ok, high speed with low level of average latency.

TESTING LOCAL STORAGE (drive C and drive D separately, I would like to see any difference between built-in RAID 1 and H/W RAID 6)
Test config
• 32K 100% read sequential, # of Outstanding IOPS= 64
• 32K 100% write sequential, # of Outstanding IOPS= 64
• 32K 100% read random, # of Outstanding IOPS= 64
• 32K 100% write random, # of Outstanding IOPS=64
• 64K 100% read sequential, # of Outstanding IOPS= 64
• 64K 100% write sequential, # of Outstanding IOPS= 64
• 64K 100% read random, # of Outstanding IOPS= 64
• 64K 100% write random, # of Outstanding IOPS= 64


STORAGE NODE 01
DRIVE C:

03_NODE-01-Drive-C.PNG (29.68 KiB) Viewed 15787 times

[genko2012]

sorry it seems i cannot add more than 3 pics in one post

STORAGE NODE 02
DRIVE C:

03_NODE-02-Drive-C.PNG (29.54 KiB) Viewed 15805 times

STORAGE NODE 01
DRIVE D:

03_NODE-01-Drive-D.PNG (29.81 KiB) Viewed 14096 times

STORAGE NODE 02
DRIVE D:

03_NODE-02-Drive-D.PNG (29.63 KiB) Viewed 14097 times

[genko2012]

Next step

Creating RAM disk on each Storage server (Size 8 Gb, formatted NTFS)
Each storage is connected to RAM disk, located on another storage server by using ISCSI (Storage Network 172.16.0.X)

STORAGE NODE 01
DRIVE E: (RAM disk on STORAGE NODE 02)

04_NODE-01-RAM02.PNG (63.8 KiB) Viewed 14091 times

STORAGE NODE 02
DRIVE E: (RAM disk on STORAGE NODE 01)

04_NODE-02-RAM01.PNG (52.87 KiB) Viewed 14082 times

[genko2012]

Next step

Creating basic image disk on each Storage server (Size 8 Gb, formatted NTFS, Write-back on, default cache size 128 MB)
Each storage is connected to basic image disk, located on another storage server on drive D: by using ISCSI (Storage Network 172.16.0.X)


STORAGE NODE 01
DRIVE F: (Image disk on STORAGE NODE 02)

05_NODE-01-Image02.PNG (52.4 KiB) Viewed 14077 times

STORAGE NODE 02
DRIVE F: (Image disk on STORAGE NODE 01)

05_NODE-021-Image01.PNG (54.15 KiB) Viewed 14072 times

[genko2012]

Finally, is it a big issue if i use one network for ISCSI and SMB traffic?
but in these tests i haven't started to use SMB yet...
Can you suggest anything?
Thanks.
And another quick test (because now i have experience to build cluster config faster.. )

Change RAID 6 to RAID 10 - when i tried to copy file to SMB share by using total commander (file cashing is off) - i got total speed around 15 Mb/s
by using windows explorer - the same as before 70-80 Mb/s
Nothing is changed..
Oh, another test (strip size of RAID)
tested RAID 6 and RAID 10
IOMeter has the same config
the best results for:
RAID 6 (Read 906 MBs, Write 926 MBs, Av. Latency 13 ms, strip size 64 k)
RAID 10 (Read 1186 MBs, Write 1222 MBs, Av. Latency 9,91 ms, strip size 512 k)
By default Intel RSBL080 Controller uses 256 k. strip size
RAID 6 (Read 886 MBs, Write 900 MBs, Av. Latency 13,3 ms, strip size 256 k)
RAID 10 (Read 1169 MBs, Write 1213 MBs, Av. Latency 9,91 ms, strip size 256 k)
Sad but true. I couldn't find a bottleneck..

by using total commander (file cashing is off) i can copy large 4Gb file from D: drive local storage to RAM disk and image disk (connected by ISCSI from another server)
av. speed 700 MB/s
and speed between RAM and Image is still fast..

[Anatoly (staff)]

As we can see from the tests (screenshots below) the Average response time of your Disk System is >90ms, so I think this is the first thing that is need to be fixed because the latency is really important, it should be not more than 10ms (actually <5ms is required).



Also, its not 100% clear how you have benchmarked your network with IOmeter. Usually when doing this we are using RAM disk, but as I can see you are using in in different test, so I`m assuming that something else was used to ensure that the network part works seamlessly. Would you be so kind to clarify what and how exactly have you used to test network part? Maybe it will be good idea to use NTttcp or iperf utilities instead of IOmeter.

Thanks

[genko2012]

Ok, i will try..
Thanks