Poor VM write performance in Hyper-v HA cluster

[Amora]

I've been researching this product for a couple weeks now due to the fact that it's one of the first hits on google when speaking of active/Active HA storage..

Im setup just like the following page running WS2012 hyper-v clustering with mirrored iscsi sans running the starwinds software:

http://highavailability.starwindsoftware.com/

IM experiencing very poor write performance when running throughput testing on a single VM with nothing else running on either the storage nodes or the hyper-v nodes. I'm using Atto and crystaldiskmark

The storage nodes have 2 iscsi nics a piece as well as 2nics dedicated to the sync channel. Both storage nodes run a 4drive RAID0 for testing purposes. All nics are 1gig nics and the Hyper-v nodes are using MPIO to connect to the storage. Given the equipment, the storage nodes can spit out 4gig/sec should it be asked to. Obviously the storage subsystem in a 4 drive raid0 is more than fast enough for testing. Both arrays do ~+400MB/sec read/writes.

The Hyper-v boxes are rather handicap though, boasting only a single 1gigabit nic for iscsi, and 1 Gbit nic a piece for VM traffic, csv/liv migration...and a management nic. So 4 nics total, traffic has been segmented off appropriately with no cluster or client traffic allowed over the iscsi channel.

Knowing this bottleneck, the theoretical max performance the vm can hope to achieve is ~100MB/s to around 125MB/s writes if your lucky and have a 100% efficient 1Gbe nic.

This theory holds true if on one of the storage nodes I shutdown both iscsi nics plus the starwinds sync channel. This forces all traffic to the other storage nodes and respective iscsi target. Again a single storage node has 2 multipathing nics so a target gets 2 data streams from 2 nics.

The problem is if I run both storagenodes full bore active/active and syncing, the write performance within the Vm dives Down to 70MB/sec. And this is with all 4 storage nics doing Round robin multipathing. If I take 1 nic from each storage node away, the performance gets even worse, I'm talking 50-60MB/second writes in this active/active SAN.

So where am I missing my mark? This is just a single VM pulling from a single lun with nothing on it. As raw as you can get. No bells and whistles, the lab was set up with simplicity in mind but the storage nodes were upgraded in both networking and storage subsystem due to the performance issues I'm seeing.
The 4drive raid0 is obviously not the issue, the network can handle a full 1gbit/s(handicapped by the switch that handled iscsi traffic, it could do 4gbit/s if I had 10gig switches laying around). My question is, is that if the active/active nature of the product is causing such a massive performance penalty due to having to simultaneously read/write and sync at the at the same time?

Again if I point the vm at a single storage node, with multipathing over dual nics, the VMs write, pin at 100MB/s, the limit of the switch and the 1gig nic dedicated to iscsi. I'm curious as to what's going on. I've watch both storage nodes iscsi multipathing nics, and it's working properly, with each nic showing equal traffic spread across all 4 iscsi nics(2 in each storage node)... Given the chance it will push 4Gbit/sec.

In this config doing active/active, raid0 and 4-Gbit nics doing MPIO, I'd expect the the VM to max out the network which it limited to 1Gbit/sec or ~ 100MB/sec as such, should not dip below 100MB/s in benchmarking.

Read performance is skewed due to csv caching and such, and those speeds are insane, it's the writes that I'm concerned about...

If this get solved we deploy and purchased the product, if not. Well, it is what it is. But I'd rather stick with this product due to having worked with it so much now.

Any help would be appreciated. By all means if something is obviously out of place here please bring it to my attention.

[anton (staff)]

Can you post here:

1) ATTO Disk Benchmark results of an underlying storage (say disk D: you host StarWind images on)
2) ATTO Disk Benchmark results of "mapped" iSCSI disk inside a VM

?

Also do you use Hyper-V vSwitch to route iSCSI traffic?

[Amora]

You might want to ignore the text document notes with in the Pics, they make sense in my head, but will probably confuse you if you try to interpret them. Just with with what I put inline here. I have to break this up into 2 posts, 3 pic limit

StorageNode2 - raid0 - Subsystem Bench

sn2-raid0-bench.PNG (86.08 KiB) Viewed 12469 times

StorageNode1/StorageNode2 - HA targets w/sync - 4 way multipath(2 nics per node)

photo.PNG (182.67 KiB) Viewed 12471 times

StorageNode1/StorageNode2 - HA targets w/sync - 2 way multipath(1 nic per node)

photo2.PNG (215.59 KiB) Viewed 12468 times

[Amora]

StorageNode1 - Standalone target - no sync - 2 way multipath(2 nics, single node)

photo4.PNG (205.96 KiB) Viewed 12490 times

StorageNode1 - Standalone target - no sync - 1 way path(1 nic, single node)

photo5.PNG (205 KiB) Viewed 12492 times

[Amora]

2) ATTO Disk Benchmark results of "mapped" iSCSI disk inside a VM

--> Do you mean create a new separate target in which to mount within the VM? Then run a bench on the target?

[Amora]

Also do you use Hyper-V vSwitch to route iSCSI traffic?

--> I'm not sure what you mean here. There is no Hyper-V Virtual switch bound to my iSCSI nics. Is there suppose to be?

[Max (staff)]

Do you mean create a new separate target in which to mount within the VM? Then run a bench on the target?

For the "Inside VM test" did you connect the iSCSI disk to the VM directly or put the VHDs on the iSCSI disk?

[Amora]

I think there's a disconnect in the thought process here:

The VM itself has no iSCSI target attached connected within it, I just benching the VM's own vhd disk. The VM's VHDX is running on the Clustered Shared Volume. The quorum and cvs1 disks are the only targets connected and are connected at the host Hyper-V server level, not from within the VM.

In other-words, the VHDX VM harddrive resides on a Clustered Shared Volume. Only 2 targets exhist.

1 quorum disk(mirrored to the other storage node)
1 CSV disk (mirrored to the other storage node)

So technically 4 targets exhist, but they are just mirrors so logically they are treated as 1 entity a piece.

[Anatoly (staff)]

dear Amora,

I`d still recommend to perform the tests with the ATTO benachmark or IOmeter tools and share the results with us after.
The reason is pretty simple - I`ve never heard that Crystal Disk is the best soft to benchmark your system, unlike those two that I`ve mentioned.

Thank you