StarWind Software

I am looking at testing V8 in my lab. Is 3 nodes the max? Any support for 4? I am asking because I use 4 servers in 2u block (C6100) and was wondering if I could everything in a block? Have Hyper-V and StarWind on each node in the block and replicate with each other.

Unlimited. With a 2-way or 3-way synchronous (+1 or +2 async) replicas between LUs.

Ok, so sync limited to 3 nodes max. Was thinking a 4 way between 4 nodes.

Sync is not limited to 3 nodes. Any node can have unlimited number of LUs and partners. It's a 2-way or 3-way replica between LUs and not actual nodes. For 4-way replica we recommend 3 sync + 1 async but that would be in a Fall update only.

can you elaborate?
I thought this was completely unlimited.
What do you mean by LU? is that an iscsi target?
I suspect this is a difference in looking at it in a hyper-v environment, and a vsphere environment.
I thought I could have an iscsi target replicated synchronously with an unlimited number of nodes for scale out performance. If the vm's stored on that target start to suffer from lag, add another synchronous node, thus increasing available nodes with multipath, which should increase target speed.

this:

anton (staff) wrote:Sync is not limited to 3 nodes. Any node can have unlimited number of LUs and partners.

and this:

anton (staff) wrote: It's a 2-way or 3-way replica between LUs and not actual nodes. For 4-way replica we recommend 3 sync + 1 async but that would be in a Fall update only.

seems contradictory.
you can have unlimited number of lu's, and partners...unlimited replication partners, right?

but then, what do you mean by 2 way or 3 way replica? and 4 way replica is 3sync + 1 async, but not available till fall....I'm so confused.
I though you could add async to anything

Please elaborate on this as soon as you can.

LU is Logical Unit. Target can have multiple LUs (at least one).

In a hyper-converged scenario VMs have better performance when run locally so if you run out of IOPS you need to throw in more nodes to offload VMs there but NOT to increase number of replicas (that's slows down the things actually). DMA is always faster then RDMA.

2-way replica is two partners, 3-way replica is 3 partners.

barrysmoke wrote:can you elaborate?
I thought this was completely unlimited.
What do you mean by LU? is that an iscsi target?
I suspect this is a difference in looking at it in a hyper-v environment, and a vsphere environment.
I thought I could have an iscsi target replicated synchronously with an unlimited number of nodes for scale out performance. If the vm's stored on that target start to suffer from lag, add another synchronous node, thus increasing available nodes with multipath, which should increase target speed.

this:

anton (staff) wrote:Sync is not limited to 3 nodes. Any node can have unlimited number of LUs and partners.

and this:

anton (staff) wrote: It's a 2-way or 3-way replica between LUs and not actual nodes. For 4-way replica we recommend 3 sync + 1 async but that would be in a Fall update only.

seems contradictory.
you can have unlimited number of lu's, and partners...unlimited replication partners, right?

but then, what do you mean by 2 way or 3 way replica? and 4 way replica is 3sync + 1 async, but not available till fall....I'm so confused.
I though you could add async to anything

Please elaborate on this as soon as you can.

I think I understand this more now. There wouldn't ever be a need to scale out to more than 3, due to diminishing returns on the overhead of replicating to more nodes.
this does not affect the scale-out architecture, just how you place replication targets on the nodes.

anton (staff) wrote:LU is Logical Unit. Target can have multiple LUs (at least one).

In a hyper-converged scenario VMs have better performance when run locally so if you run out of IOPS you need to throw in more nodes to offload VMs there but NOT to increase number of replicas (that's slows down the things actually). DMA is always faster then RDMA.

2-way replica is two partners, 3-way replica is 3 partners.

more questions:
1)Is there a best practice on LU's? So LU's are the devices you add to a starwind iSCSI target. When do you add devices to iscsi targets, versus just creating another target with a device under it?
is that strictly organizational in nature?

2)I also don't understand about target groups, and what a control device is(is that a hyper-v thing?)

3)increasing number of replicas actually slows things down...so that does make sense, in that it increases the amount of reads/writes on each host that is a replication partner. It increases network saturation per additional replication partner. from internal testing, do you have numbers to back up the diminished returns on a 2-way, 3-way, and 4-way?

4)what about limits on async replication partners?

There's a point to scale out when you run either out of CPU power or storage or both.

There's very little point in having more then 2 partners for a particular LU as

a) storage still costs money

b) taut tolerance of 99.9999% is fair enough

c) every next replica increases read but decreases write performance for LU.

barrysmoke wrote:
I think I understand this more now. There wouldn't ever be a need to scale out to more than 3, due to diminishing returns on the overhead of replicating to more nodes.
this does not affect the scale-out architecture, just how you place replication targets on the nodes.

1) Target can have multiple LUs. You can have "target = LU" for simplicity.

2) ...

3) Writes. As iSCSI is SCSI-over-IP (actually TCP) and we cannot do multicast/broadcast sends. So writes need to travel down the Ethernet fabric and obviously doubled amount of data is wired 2 times slower then a single copy. Reads are different a bit (read aggregates data from local target and partners so Ethernet / NICs are used more efficiently). So 3-way replication is faster in reads but slower in writes then 2-way replication (of course with MPIO in Round Robin).

4) For now we support 1 async partner for LU. We'll make them 2 with some update and will probably stop (there's no point to have more then one probably local on-site and one remote off-site replica of the LU).

barrysmoke wrote:

anton (staff) wrote:LU is Logical Unit. Target can have multiple LUs (at least one).

In a hyper-converged scenario VMs have better performance when run locally so if you run out of IOPS you need to throw in more nodes to offload VMs there but NOT to increase number of replicas (that's slows down the things actually). DMA is always faster then RDMA.

2-way replica is two partners, 3-way replica is 3 partners.

more questions:
1)Is there a best practice on LU's? So LU's are the devices you add to a starwind iSCSI target. When do you add devices to iscsi targets, versus just creating another target with a device under it?
is that strictly organizational in nature?

2)I also don't understand about target groups, and what a control device is(is that a hyper-v thing?)

3)increasing number of replicas actually slows things down...so that does make sense, in that it increases the amount of reads/writes on each host that is a replication partner. It increases network saturation per additional replication partner. from internal testing, do you have numbers to back up the diminished returns on a 2-way, 3-way, and 4-way?

4)what about limits on async replication partners?