V8 Max Nodes

[jariangibson]

I am looking at testing V8 in my lab. Is 3 nodes the max? Any support for 4? I am asking because I use 4 servers in 2u block (C6100) and was wondering if I could everything in a block? Have Hyper-V and StarWind on each node in the block and replicate with each other.

[anton (staff)]

Unlimited. With a 2-way or 3-way synchronous (+1 or +2 async) replicas between LUs.

[jariangibson]

Ok, so sync limited to 3 nodes max. Was thinking a 4 way between 4 nodes.

[anton (staff)]

Sync is not limited to 3 nodes. Any node can have unlimited number of LUs and partners. It's a 2-way or 3-way replica between LUs and not actual nodes. For 4-way replica we recommend 3 sync + 1 async but that would be in a Fall update only.

[barrysmoke]

can you elaborate?
I thought this was completely unlimited.
What do you mean by LU? is that an iscsi target?
I suspect this is a difference in looking at it in a hyper-v environment, and a vsphere environment.
I thought I could have an iscsi target replicated synchronously with an unlimited number of nodes for scale out performance. If the vm's stored on that target start to suffer from lag, add another synchronous node, thus increasing available nodes with multipath, which should increase target speed.

this:

Sync is not limited to 3 nodes. Any node can have unlimited number of LUs and partners.

and this:

It's a 2-way or 3-way replica between LUs and not actual nodes. For 4-way replica we recommend 3 sync + 1 async but that would be in a Fall update only.

seems contradictory.
you can have unlimited number of lu's, and partners...unlimited replication partners, right?

but then, what do you mean by 2 way or 3 way replica? and 4 way replica is 3sync + 1 async, but not available till fall....I'm so confused.
I though you could add async to anything

Please elaborate on this as soon as you can.

[anton (staff)]

LU is Logical Unit. Target can have multiple LUs (at least one).

In a hyper-converged scenario VMs have better performance when run locally so if you run out of IOPS you need to throw in more nodes to offload VMs there but NOT to increase number of replicas (that's slows down the things actually). DMA is always faster then RDMA.

2-way replica is two partners, 3-way replica is 3 partners.

can you elaborate?
I thought this was completely unlimited.
What do you mean by LU? is that an iscsi target?
I suspect this is a difference in looking at it in a hyper-v environment, and a vsphere environment.
I thought I could have an iscsi target replicated synchronously with an unlimited number of nodes for scale out performance. If the vm's stored on that target start to suffer from lag, add another synchronous node, thus increasing available nodes with multipath, which should increase target speed.

this:

Sync is not limited to 3 nodes. Any node can have unlimited number of LUs and partners.

and this:

It's a 2-way or 3-way replica between LUs and not actual nodes. For 4-way replica we recommend 3 sync + 1 async but that would be in a Fall update only.

seems contradictory.
you can have unlimited number of lu's, and partners...unlimited replication partners, right?

but then, what do you mean by 2 way or 3 way replica? and 4 way replica is 3sync + 1 async, but not available till fall....I'm so confused.
I though you could add async to anything

Please elaborate on this as soon as you can.

[barrysmoke]

Code: Select all

you offload vm's, when you run out of local resources for vm performance, such as virtual cpu's, or ram, etc..
our goal with starwind, was to build out multipath iscsi targets, to be used by any cluster node to store vm's.  

I read your post about bypassing the iSCSI stack for network traffic on a local hyper-v server, so I can see that side of the argument for local is always faster than remote...  I can see how that would add even more performance for local disk access, however in a vmware environment, where every node in the cluster is accessing the iscsi target with multipath, that abstracts the local vs remote.   We are using hyper-converged infrastructure.  If all the starwind iscsi nodes have multiple 10G links, so that every node in the vmware cluster has a 10G path to the iscsi target, whether it is on local vmware node, or remote, wouldn't the limitation be the 10G network?  

so, according to this thread, there are some hard limits on replication?
you can only do 3way replication?
We were going to take this out much further, especially in VDI scenarios.

This will switch our scale-out strategy from adding single nodes to increase performance, to adding 3 nodes each time you need to scale out, basically adding 3 node clusters.

I think I understand this more now. There wouldn't ever be a need to scale out to more than 3, due to diminishing returns on the overhead of replicating to more nodes.
this does not affect the scale-out architecture, just how you place replication targets on the nodes.

[barrysmoke]

LU is Logical Unit. Target can have multiple LUs (at least one).

In a hyper-converged scenario VMs have better performance when run locally so if you run out of IOPS you need to throw in more nodes to offload VMs there but NOT to increase number of replicas (that's slows down the things actually). DMA is always faster then RDMA.

2-way replica is two partners, 3-way replica is 3 partners.

more questions:
1)Is there a best practice on LU's? So LU's are the devices you add to a starwind iSCSI target. When do you add devices to iscsi targets, versus just creating another target with a device under it?
is that strictly organizational in nature?

2)I also don't understand about target groups, and what a control device is(is that a hyper-v thing?)

3)increasing number of replicas actually slows things down...so that does make sense, in that it increases the amount of reads/writes on each host that is a replication partner. It increases network saturation per additional replication partner. from internal testing, do you have numbers to back up the diminished returns on a 2-way, 3-way, and 4-way?

4)what about limits on async replication partners?

[anton (staff)]

There's a point to scale out when you run either out of CPU power or storage or both.

There's very little point in having more then 2 partners for a particular LU as

a) storage still costs money

b) taut tolerance of 99.9999% is fair enough

c) every next replica increases read but decreases write performance for LU.

I think I understand this more now. There wouldn't ever be a need to scale out to more than 3, due to diminishing returns on the overhead of replicating to more nodes.
this does not affect the scale-out architecture, just how you place replication targets on the nodes.

[anton (staff)]

1) Target can have multiple LUs. You can have "target = LU" for simplicity.

2) ...

3) Writes. As iSCSI is SCSI-over-IP (actually TCP) and we cannot do multicast/broadcast sends. So writes need to travel down the Ethernet fabric and obviously doubled amount of data is wired 2 times slower then a single copy. Reads are different a bit (read aggregates data from local target and partners so Ethernet / NICs are used more efficiently). So 3-way replication is faster in reads but slower in writes then 2-way replication (of course with MPIO in Round Robin).

4) For now we support 1 async partner for LU. We'll make them 2 with some update and will probably stop (there's no point to have more then one probably local on-site and one remote off-site replica of the LU).

LU is Logical Unit. Target can have multiple LUs (at least one).

In a hyper-converged scenario VMs have better performance when run locally so if you run out of IOPS you need to throw in more nodes to offload VMs there but NOT to increase number of replicas (that's slows down the things actually). DMA is always faster then RDMA.

2-way replica is two partners, 3-way replica is 3 partners.

more questions:
1)Is there a best practice on LU's? So LU's are the devices you add to a starwind iSCSI target. When do you add devices to iscsi targets, versus just creating another target with a device under it?
is that strictly organizational in nature?

2)I also don't understand about target groups, and what a control device is(is that a hyper-v thing?)

3)increasing number of replicas actually slows things down...so that does make sense, in that it increases the amount of reads/writes on each host that is a replication partner. It increases network saturation per additional replication partner. from internal testing, do you have numbers to back up the diminished returns on a 2-way, 3-way, and 4-way?

4)what about limits on async replication partners?