Updating procedure 5.7HA to 5.8HA

[Paul W]

We’re currently running a Hyper-V cluster (two nodes) based upon HA Starwind storage (version 5.7). Before we upgrade to 5.8, I want to make sure I get all of the details right.
We have Hyper-V node1, Hyper-V node2, Storage node1 and Storage node2. We have HA as well as a few Basic images, in total a pretty straightforward setup I’d guess.
I’m especially interested in how much downtime to allow for. We have some huge luns (>10Tb) so we need to allow for some serious re-syncing time; during re-syncing our Hyper-V cluster gets a serious performance hit, so we need to schedule this as carefully as possible.
This is my understanding of the complete procedure and its consequences for our infrastructure; I have included a few extra steps for my piece of mind (backup/rollback);

1. Make backup-image of server of storage node1
2. Run starwind5.8 installation on running storage node1, reboot the server (?)
3. The Hyper-V cluster is now getting is storage from storage node 2
4. Make backup-image of storage node2
5. Shut down the Hyper-V VM’s and bring down the Hyper-V cluster, as we are about to lose storage connectivity.
6. Run starwind5.8 installation on running storage node2, reboot the server (?)
7. Starwind now starts re-syncing, and accepts connections
8. Start Hyper-V cluster, wait X hours for sync to be ready for best performance.

Back-out scenario; in case something goes wrong during upgrade; re-apply backup-image to go back to 5.7

Impact:
All in all our systems need to go 100% for about 30 minutes (guessing) while I’m upgrading the second storage node and will have a performance hit for a few hours after the upgrade while the HA storage is re-syncing.
My main question: is the above scenario as I understand it correct?

[Anatoly (staff)]

Dear Paul,

This is my understanding of the complete procedure and its consequences for our infrastructure; I have included a few extra steps for my piece of mind (backup/rollback);

1. Make backup-image of server of storage node1
2. Run starwind5.8 installation on running storage node1, reboot the server (?)
3. The Hyper-V cluster is now getting is storage from storage node 2
4. Make backup-image of storage node2
5. Shut down the Hyper-V VM’s and bring down the Hyper-V cluster, as we are about to lose storage connectivity.
6. Run starwind5.8 installation on running storage node2, reboot the server (?)
7. Starwind now starts re-syncing, and accepts connections
8. Start Hyper-V cluster, wait X hours for sync to be ready for best performance.

First of all you shouldn`t reboot the server after the installation of 5.8.
ALso, please take a noter that you hsould initiate synchronization manualy after 6th step.

All in all our systems need to go 100% for about 30 minutes (guessing) while I’m upgrading the second storage node and will have a performance hit for a few hours after the upgrade while the HA storage is re-syncing.

Well, I would say that it`ll be less then 30 minutes.
Additionaly I`d like you to know that you can change traffic priority from sync to client requests and backwards by choosing the corresponding option in the dropbox that will appear after right-click on the device.

[Paul W]

Thanks Anatoly, looks like we will be scheduling some maintenance soon

[Anatoly (staff)]

Sounds great!

Keep us updated please!

[camealy]

We have the ability to shut the Hyper-V cluster down during the upgrade, but does this require a full sync or a quick sync? Also, are there any network reconfiguration changes required?

Thanks,

Kurt

[Anatoly (staff)]

Full synchronization is the only option here.

[camealy]

We completed the upgrade without issue last Friday night. Initial results seemed promising with the full sync over 10Gb happening much faster with 5.8 then it ever did before.

We are however on our 2008 R2 guests of the Hyper-V cluster experiencing very strange throughput patterns. It is very linear until you get to the larger transfer sizes where sometimes throughput actually stops.

See ATTO below.

[Anatoly (staff)]

What RAID are you using on SAN boxes please (vendor and type)?

[camealy]

This is a comparison of 5.7 to 5.8 Nothing else has changed. Below is what it used to look like. Very linear.

HP Smart Array P410 1GB Flash Backed Raid 5 6x 450GB 15K SAS

iSCSi.png (28.64 KiB) Viewed 32365 times

[Anatoly (staff)]

OK, so can you confirm that theese two results are pretty new both and that one the first one we can see results for 5.8 and on the latesst one - for 5.7. If yes then I`m just wondering - do you have both installations in your environment?

[camealy]

Here is another example, this one taken a few months before the upgrade from the exact same environment. If you like i have one from 5.6 as well, even taken from the same VM.

[oxyi]

Your benchmark remind me of how mine was before..

You currently have two HA target mounted right ? HA and HAPartner. Can you do me a favor and drop the HAPartner iSCSI target connection completely, so only the HA iSCSI target is left.
And do a benchmark with that, if it an improved benchmark, can you do another one with both target connected ?

Thanks!

[camealy]

This is in the MS initiator on the Hyper-V cluster host? Can I accomplish the same thing with changing the MPIO policy from round-robin to fixed path?

Just want to make sure I don't break anything.

[oxyi]

MS iSCSI initiator

Yea, haven't you test out the HA setup at all ? Like you can lose one of the Starwind server and your server should still run, that's what an active HA setup is.

If you didn't set up it that way, then you shouldn't disconnect it.

Not sure if you can accomplish the same thing via fixed path, but what I found out is that when I have both HA target connected, my performance became like your benchmark.
Search thru my post then you will see. I was just wonder if you have the same issue since you said nothing was changed at all.

[Max (staff)]

2 Camelay,
Fixed path is not an option, HA benefits from Round Robin since it can use both pathes for reading the data. Fixed path should be avoided because it slows down the failover.
By the way, I've seen a similar performance drop with non-equal jumbo frames on the SAN servers. Could you please check the values in both device properties, and HP network utility (if used)