starwind as alternative for stormagic

[barrysmoke]

I'm looking at starwind to replace stormagic svsan. both have iscsi mirroring, but starwind has more features than stormagic.
i have some questions about the optimal configuration to squeeze out performance.
with stormagic, I used vmware as the hypervisor, and passed an lsi 9271-8i through to the stormagic vm, using vmware direcpath. this allowed me to get native hardware speeds(or as close as you can get), while allowing us to use the host as a vmware host in our cluster. if the vsan virtual machine needed more resources, just reserve another cpu core, or add a ram reservation. I also ran these san vm's on an ssd mirror.

I'm looking to replicate, or improve that configuration with starwind. I'd like input on the benefits/features of switching altogether to hyperv+starwind native san for hyperv, or sticking with vmware, and running starwind from a windows vm, as mentioned above. I did see that with server 2012, it is now possible to get the free hyperv server, and configure an HA cluster, something that requires a 4k+ vmware essentials plus license under vmware.

I'm currently testing starwind under vmware, and I installed v6 on 2 windows 2k8 hosts, with a 40gig c:, and a second 60 gig virtual disk. windows 2k8 sees the second disk, but when I tried to replicate my above setup virtually, by picking disk bridging, starwind didn't see the disks. so, for testing I've picked a 65gig virtual disk for now.

so, first question:
what are the advantages/disadvantages to using a virtual disk for the entire SAN/iscsi target?
I would think when replicating in a hardware environment, using a dedicated raid controller, such as the 9271-81, with a raid6 array, would have much better performance than a file on top of the OS.
I would worry about file corruption with a single file(too many layers, SAN virtual disk, managed by starwind iscsi, in a file on top of ntfs)
I did see another post about better performance for a virtual disk, and would like to see an elaboration of that.
I also saw a new term I was not familiar with in the starwind gui/and a forum post..Asymmetric Logical Unit Assignment(ALUA)

question number 2:
our stormagic had an issue that caused data corruption twice in 2 years. I've been considering rolling my own zfs based solution, however I can find no way to mirror zfs iscsi targets, which is a requirement for HA.
With Server 2012, we now have ReFS. how does starwind work with an ReFS volume, is it compatabile? In fact, how does it work with storage spaces, etc...?

I'm sure I'll have more questions, but that's a start

Barry Smoke

[anton (staff)]

1) It's a bad idea to pass the raw disk to StarWind. Whole virtualization layer we provide does a lot of things including write coalescing, log structuring, snapshotting etc. At the same time we don't use file I/O so think about StarWind imgs as a containers only, I/O is handled by storage stack directly (no NTFS/ReFS overhead).

2) Parity RAID (especially dual parity) is a bad idea with a typical virtualization workload. Tons of small writes will initiate whole stripe updates and system would become a write pig. We fix this with LSFS but it's experimental in V8.

3) ALUA is for HA configs with a different types of storage. So you combine RAM and flash or flash and spindle inside the same cluster and want writes and reads to go to the faster node initially.

4) ReFS is junk... We do have a stronger checksums as part of LSFS but for flat image files you can use ReFS (just make sure you enable not only metadata hashing). V8 will work with storage spaces as they report hardwar 4KB blocks, V6 cannot do this.

[barrysmoke]

1) It's a bad idea to pass the raw disk to StarWind. Whole virtualization layer we provide does a lot of things including write coalescing, log structuring, snapshotting etc. At the same time we don't use file I/O so think about StarWind imgs as a containers only, I/O is handled by storage stack directly (no NTFS/ReFS overhead).

so, if I'm hearing you correctly, by using a virtual disk image, that strips out the ntfs/refs overhead. even though, the container file is logically placed inside a formatted disk/volume with ntfs/refs, by putting it into a container file, that bypasses the overhead for all read/writes...is that what I'm understanding? I had thought reading/writing to any file inside a filesystem added the filesystem's overhead, but if that is indeed the case, I can really see the flexibility in the starwind solution.

2) Parity RAID (especially dual parity) is a bad idea with a typical virtualization workload. Tons of small writes will initiate whole stripe updates and system would become a write pig. We fix this with LSFS but it's experimental in V8.

Raid6 is how we've configured a san with stormagic, and are getting really great iometer/iops based on a 24 x 1 TB( 2.5) drive config. So LSFS would improve performance to what I'm used to seeing in our stormagic config? I ran a 12 disk 2TB near line sas raid 6 in production for over a year without any issues, using 20+ vm's. only real performance issues were during an iscsi target resync(where master was online, and mirror copy was resyncronized).
I have some benchmark numbers, I'll replicate the scenario with a starwind v8 setup, and do some testing.
I'm specifically interested in how adding an ssd cache drive to an lsi raid card would affect a starwind config. would it be worth it, given the ALUA/starwind caching features?

3) ALUA is for HA configs with a different types of storage. So you combine RAM and flash or flash and spindle inside the same cluster and want writes and reads to go to the faster node initially.

Ah, so this gives you control over your config, for tweaking performance based on what you have/what is faster, such as a ramdisk, or a good ssd...I'll have to play with that.

4) ReFS is junk... We do have a stronger checksums as part of LSFS but for flat image files you can use ReFS (just make sure you enable not only metadata hashing). V8 will work with storage spaces as they report hardwar 4KB blocks, V6 cannot do this.

so that's a yes, but it's junk
does it limit performance, or just a poor implementation?
I know it doesn't do deduplication, but the data integrity is worth it to me. I had an entire SAN start experiencing corruption, probably something in the filesystem, and I never want to go through that again. so are you saying, the checksums are good enough in v8 to avoid refs, and can you state why it is junk? I watched a technology video on it by a microsoft engineer, and it sounded great.
would love to have zfs on windows, but oh well.

[barrysmoke]

any issues with say a 21TB, or larger virtual disk?
**nevermind, I saw the 10TB file creation thread, and ntfs on 2008 has a limit, but 2012 is like 256TB, so plenty of room to grow***

[anton (staff)]

1) Yes, file systems are used to pass thru mode so they are basically bypassed for everything including cache, we talk directly to storage stack and do own caching. No overhead. More flexibility as you can "see" the content.

2) It's up to you but I would go RAID10 on your place. For now.

3) Adding hardware SSD cache to LSI and going LSI in general is a bad idea. Use PCIe flash cards (FusionIO) as they can be re-provisioned to be used by SYSTEM and not by LSI only. And LSI is useless with Storage Spaces (unless you build RAID10).
Key point here - buy as much COMMODITY hardware as possible. No SAS controllers, no SAS JBODs, no SAS drives. Cheap and fast SATA drives (insanely overprovisioned) + tons of RAM and flash for cache.

4) You don't need ZFS for Windows. ZFS is just another old granny like NTFS is but maybe 10 years younger. You need TOTOALLY different file system which is a) cluster-aware (both NTFS and ZFS are *NOT*) and b) optimized for a multi-initiator access
and typical VM workload (both are *NOT* optimized for fast LUN owner switch and do slow mount/unmount and both have very redimentary logging with log disks and ZIL accordingly). What you really need is VMFS5. Or 6 + VSAN. And that's
EXACTLY what we do here @ StarWind.

[anton (staff)]

Yes, right. And LSFS we'll eventually use to become a primary storage down own GB-size spanned containers to avoid huge files size issues.

any issues with say a 21TB, or larger virtual disk?
**nevermind, I saw the 10TB file creation thread, and ntfs on 2008 has a limit, but 2012 is like 256TB, so plenty of room to grow***

[kspare]

Why would LSI's Cachecade be a bad idea? We're using it and it's working pretty good. All the vms we have attached over 10gb ethernet are working great!

[anton (staff)]

Because it's waste of money. Modern storage paradigm is to run RAID in software (ZFS, Storage Spaces etc) with a combination of a strong hashes done by file system, using flash and RAM as a system-wide resource sitting on a fast system bus (RAM) or directly on PCIe (flash) and being accessible to the whole system and not to just controller. Running cache on controller is 1) stealing resources out of the system as they cannot be used by OS any more and 2) slowing down the things as CPU is always faster and memory bus is wider then controller on-board I/O processor and PCI-e attached now cache. Batteries on controllers don't have any sense as we run clusters so every node is an effective stand-alone controller by itself + UPS for a whole system. There's no much value in the whole thing any more.

Why would LSI's Cachecade be a bad idea? We're using it and it's working pretty good. All the vms we have attached over 10gb ethernet are working great!

[barrysmoke]

If I were to try to improve upon my stormagic design(where storage is passed through to san vm with vmware direcpath for direct control)
with starwind, I have 2 options:
1)vmware environment with local datastore for san os(this could be a raid card, with 2 ssd's, or a fuzion io), windows server as virtual san os, with raid card passed through, windows mounts raid volume as d:, starwind iscsi san with replication/2 node cluster(under 16TB limit on windows 2008)

2)hyperv with a fuzion io partitioned so that main os runs from a 20 to 40 gig c:, and the remaining space used as cache for the iscsi san.

are these the most optimal ways to run starwind under the 2 environments?
if a ramdisk were used, where would I put it in the above scenarios? I know that v6 has a ramdisk option, but not sure how to use it/where in the config...when creating an ha on v6, you can only specify cache size, not where it is stored. I also see when a virtual disk is created, there is a swap disk as well....could this be placed on faster storage for increased speed/performance?

[kspare]

Because it's waste of money. Modern storage paradigm is to run RAID in software (ZFS, Storage Spaces etc) with a combination of a strong hashes done by file system, using flash and RAM as a system-wide resource sitting on a fast system bus (RAM) or directly on PCIe (flash) and being accessible to the whole system and not to just controller. Running cache on controller is 1) stealing resources out of the system as they cannot be used by OS any more and 2) slowing down the things as CPU is always faster and memory bus is wider then controller on-board I/O processor and PCI-e attached now cache. Batteries on controllers don't have any sense as we run clusters so every node is an effective stand-alone controller by itself + UPS for a whole system. There's no much value in the whole thing any more.

Why would LSI's Cachecade be a bad idea? We're using it and it's working pretty good. All the vms we have attached over 10gb ethernet are working great!

Ok, I must be missing something so bear with me.

I have 4 ssd's in my server, currently running windows 2012 and I have cachecade enabled on my controller, so I use then in a raid 10 cachecade volume.

While I am running StarWind v6, how would I better utilize the ssd's and get rid of cachecade?

[anton (staff)]

Bring second node to a StarWind cluster, re-spread SSDs between them to build RAID 0+1 (RAID0 between a pair of SSDs on every node
and RAID1 between the nodes done by StarWind) to get the same used capacity you have now with RAID10 but better x-put (you have
now two controllers instead of a one, MPIO would do the trick in doubling the bandwidth). + you'd be able to use RAM as a distributed
write back cacne (you cannot now as you're locked in a single node so spoofing w/o NVRAM is dangerous).

No need in a hardware RAID. Generally speaking we recommend a 3-way replication and RAID0 on every single node for performance.
Mirroring between nodes is what StarWind does and striping is implemented properly with Windows LVM (Storage Spaces now).

Because it's waste of money. Modern storage paradigm is to run RAID in software (ZFS, Storage Spaces etc) with a combination of a strong hashes done by file system, using flash and RAM as a system-wide resource sitting on a fast system bus (RAM) or directly on PCIe (flash) and being accessible to the whole system and not to just controller. Running cache on controller is 1) stealing resources out of the system as they cannot be used by OS any more and 2) slowing down the things as CPU is always faster and memory bus is wider then controller on-board I/O processor and PCI-e attached now cache. Batteries on controllers don't have any sense as we run clusters so every node is an effective stand-alone controller by itself + UPS for a whole system. There's no much value in the whole thing any more.

Why would LSI's Cachecade be a bad idea? We're using it and it's working pretty good. All the vms we have attached over 10gb ethernet are working great!

Ok, I must be missing something so bear with me.

I have 4 ssd's in my server, currently running windows 2012 and I have cachecade enabled on my controller, so I use then in a raid 10 cachecade volume.

While I am running StarWind v6, how would I better utilize the ssd's and get rid of cachecade?

[kspare]

That explanation doesn't make sense.

CacheCade is using the ssd's right now.

to whom do I assign the ssd's? Starwind V6 doesn't do anything with them yet, only v8 does. Because V8 is beta it's irrelevant at this point.

So who uses the ssd's?

[anton (staff)]

V6 can use flash as a primary storage only. V8 will do flash-as-a-cache.
Again, distributed (safe for write back and VM moved to another physical host will start from a "hot" rather then "cold" state). LSI you have cannot do this (expensive models can).

That explanation doesn't make sense.

CacheCade is using the ssd's right now.

to whom do I assign the ssd's? Starwind V6 doesn't do anything with them yet, only v8 does. Because V8 is beta it's irrelevant at this point.

So who uses the ssd's?

[kspare]

thats basically how cachecade works.

Take cachecade out of the picture.

I have server 2012 and Starwind V6, how do you propose I use my ssd's? the way I see it right now, cachecade is my only option. You give a very vague description of allocating them elsewhere....where is elsewhere?

Until V8 is actually out, cachecade is our best bet?

[anton (staff)]

Upgrade to R2 and use SSDs as a write back cache for StarWind images. StarWind V6 will provide RAM cache and Windows will do flash cache.
Upcoming V8 will do both + mirroring of cache between the nodes (something Windows cannot do). If you don't want to upgrade to R2 and V8 then "magic hardware" is the only bet.

thats basically how cachecade works.

Take cachecade out of the picture.

I have server 2012 and Starwind V6, how do you propose I use my ssd's? the way I see it right now, cachecade is my only option. You give a very vague description of allocating them elsewhere....where is elsewhere?

Until V8 is actually out, cachecade is our best bet?