StarWind Software

Another lsfs issue chiming in here.

I am on the free v8 install, hyperconverged esxi. I have 2tb of data that grew to the 8tb volume until it had 12.5mb free space and all my virtual machines locked up, became unresponsive, and basically flipped out. I realized what happened so I rebooted the vsan machine and installed the latest build claiming to fix the lsfs growth issue, 7774. After it mounted the datastore (hours later as usual) it came online and became available to esxi but there were no storage vmfs partitions showing! according to VMware, that datastore was showing as unformatted!!!
My drive containing the spsx files also now reports 4tb more free space that did not exist. Did lsfs cleanup happen and also cause lost data?

I recovered several of my machines to another datastore to get production partially up however I did not have recent enough functioning backups for everything that was lost and I would prefer to not have to re-create it all.

Last time I lost a datastore I had the l2 cache go corrupt so I tried mounting this one without the l2 - no change.

Am I up a creek or is there something I can do? I only care about not losing my vsan data again.

May I ask you if you have created new devices with 7774 build, or you have created them on previous build, and ran an update after?

Hi Anatoly,
The devices were created prior on 7354.

Here is a shortened timeline:
originally setup datastores (no dedupe, thin prov lsfs)
upgraded incrementally several times, 7354>7471>7509.
lsfs grew and ate all of the space, virtual machines on iscsi had issues
rebooted starwind server, when re-mounted the datastore showed as empty to esxi
rebooted again, stopped starwind services from starting, installed 7774 upgrade hoping it could mount/cleanup lsfs bloat and remedy situation

I just spent 2 1/2 hours on the phone with several engineers at VMware.
they dug into the hex code of the partition tables and really exhausted many avenues. They were perplexed at how this could have happened and unable to recover any data/information. The metadata for the partitions is gone, my data is unrecoverable. Unless starwind can think of some other form of magic then virtual san has lost my production data... again.

Hi! Thanks for the update?

How about jumping on the quick call next week? If you agree please drop me a PM with your phone number.

Thank you

kkuszek wrote:I just spent 2 1/2 hours on the phone with several engineers at VMware.
they dug into the hex code of the partition tables and really exhausted many avenues. They were perplexed at how this could have happened and unable to recover any data/information. The metadata for the partitions is gone, my data is unrecoverable. Unless starwind can think of some other form of magic then virtual san has lost my production data... again.

We had something similar happening. Starwind is trying to fix it by "going back in time" on the log-device...
So far, we haven't it working though

Yeah, sorry about that. Just an update for community about Michaels case:
We are trying to build a new utility which will should get the data back.
But from what we discussed, valid data must be near the end of the LSFS device, so going by 5000 will not really help.
The best you can do now, is going from the point we have finished with steps by 10, while we are working on the new utility.

Yes, I'm currently doing that

My current eta is Thursday, I reached out to Anatoly and am waiting on support.
I restored existing servers to another datastore, I have not deleted or changed things since the disaster so I am hoping but I guess we will see. I will update this thread with my progress as any updates come for other users.

Just checking in here: R&D guys are working to get universal tool for data recovery

Hello.

I have same issue.
After upgrade from 7509 my 3 lsfs storage targets with Hyper-V vhdx disks have empty or buggy.

How to send you logs?

As an update to anyone following with Anatoly's post above they have not contacted me with any tool or fix, just the update he posted that they are working on it.

Yes, it's very sad
I never ever go near an LSFS device again!

Same here,
I do not believe v8 was at all production ready. LSFS was what made me want the product and also my demise. This is only a disaster recovery effort for me at this point. How can I recovery my information to put on reliable storage elsewhere.

Guys, terribly sorry for that. I mean, I know how you guys feel. Just want to assure you that we are doing our best to get all the issues gone far far away as soon as we can.