ISCSI lost connection

porkchop85 · Wed May 22, 2013 1:50 pm

Esxi 5 Starwind Iscsi target perc h800 MD1200 Raid 10 Server 2008

So over the weekend something happened in my datacenter. My sysadmin explained that they were replacing batteries in our APCs and somehow blew one up. They then started unplugging things and ended up unplugging my storage server. I come in from vacation and the Vms that I had running on the SAN came up as orphaned or inaccessible. I rebooted the storage server and remounted my deduped disk and had to unmount and remount in esxi. Still no luck then after browsing my datastore I noticed that it was empty. Now the pie chart still shows up like before but the datastore shows no files.

So how can I get access to my vmdks if possible?

porkchop85 · Thu May 23, 2013 1:56 pm

Datastore still shows the right about of data but when I browse its empty. Attached is a picture

Thu May 23, 2013 4:15 pm

Hi Porkchop85,
Could you please tell us which SW version you're using, and what is the cache type and size you've used on the deduplication device?

porkchop85 · Thu May 23, 2013 4:19 pm

Version 6.0.5228 Writeback caching 53.05 GB cache

Fri May 24, 2013 8:46 am

Since write back caching confirms a write before it physically hits the disk there is a big chance that some of these 53 GBs are lost with a power outage.

Right now the recommended way to go is:
1. Recreate the Deduplication device with a smaller cache size, and different cache type. Since your server yields power outages it is recommended not to use write-back caching to prevent data loss. The recommended cache parameters are 1-3 GB, Write through (or write-back at your own risk)
2. When the device is recreated initialize a full restore of a most recent backup to the Datastore.

Alternative route may be using VMFS recovery or similar utilities to attempt to resurrect the existing Datastore.
Although I still recommend to move away from Write-back caching on a non HA storage resource.

porkchop85 · Fri May 24, 2013 12:32 pm

Okay thank you for the help I appreciate it!

epalombizio · Sun May 26, 2013 11:30 am

I had the same exact problem except my issue happened after a graceful shutdown of the service while the Starwind server software was being upgraded. My dedup store shows up on my vSphere server, but the server thinks the volume is empty.

I had write-back cache enabled as well, but since it was a planned graceful shutdown, I wouldn't expect Starwind to have a problem with it.

E

Tue May 28, 2013 8:12 am

Elvis, this happened in one of the older versions, right?

epalombizio · Tue May 28, 2013 10:37 am

Max,
No, this happened when I upgraded to the newest build. Anatoly sent me over a dedup store recovery program which I'll try and run today to see how it works and update him.

Thanks,
Elvis

Thu May 30, 2013 11:26 am

Just update for the comunity: the DD utility helped epalombizio to recover his data. If someone will have identical issue (I really hope that noone will

) just email to support@starwindsoftware.com with reference to this topic and ask for DD Repair utility.

epalombizio · Thu May 30, 2013 1:21 pm

I can confirm that it worked very well. Dmitry and Anatoly were very responsive in getting the utility to me and addressing questions I had. I had to go back 3 restore points, but thankfully I didn't have any recent data changes.

Is this going to be fixed in a minor release to version 6 or is this just going to be rolled into version 8?

Mon Jun 03, 2013 12:39 pm

In version 8 we`ll have great improvement in DD engine, and there will be possibility to avoid the issues like you had. Stay tuned