Recovery of damaged LSFS Device
Posted: Tue Feb 24, 2015 8:16 pm
Hey guys,
I normally never ask for help in a forum, especially not if I'm not paying for the product - it kind of feels like "abuse". However right now I'm in a really desperate situation where i seriously can't help myself and can't find any information, so I'm actually begging you, please help me
So before I attempt to tell you the story with a length equivalent to of half of the bible in a (very-) shortened version, let me say a very very very big thank you for creating such an awesome piece of software and giving free licenses for personal use! You guys seriously are the best. Amazing performance and great stability. It took me more than half a year to break something - which has literally never happened before (I do have a talent for breaking stuff
).
I'll try to keep the story short:
My complete setup is one single physical server built by myself and running at home. I'm currently studying Computer Science and this system is my "playground". But over time this "playground" has collected a lot of (to me personally) valuable stuff, where I put a lot of hard work into, so I hope there is a way to fix this
I'm running StarWind V8 on a Windows Server 2012 R2 Virtual Machine on an ESXi 5.5 host. All Harddisks are on a RAID Controller (but configured as JBOD), passed through to the VM (means the OS has direct access).
On 2 HDD's with I created a Simple Mirror with Storage Spaces. Size = 3TB, FS = NTFS. On this Drive (From now on Called "R") I created a LSFS with L1 and L2 Cache and Size 2TB (yes, not very smart...) for storage of VM's.
So this was running (almost) perfectly since September of last year - until 2 days ago, drive R was full. The Software immediatly switched the LSFS device (lets call it "lsfs1") into read-only mode - that's when I noticed something was wrong.
So I went on and read everything about the infinitely-growing lsfs devices and thought: OK, that should be possible to fix. My idea was to copy the LSFS files over to another Drive (Drive "S", which is a RAID5 [Also Storage Spaces Software RAID] with about 10 TB of free space). After verifying all the files via checksum, I then deleted the files from "R" and created a new LSFS device there, happy to now mount both of them in ESXi and vMotion my VM's over. I tried to mount lsfs1 (the "old" one) by using the "use existing file" feature when creating a new device, but it failed - telling me "Invalid Storage selected". If I click OK and try over and over again (for like 20 - 30 times) I can get past that message, but in the end the mounting of the device fails anyway.
I should add, that everytime I try to add the device I create a new header file (by choosing the .spsp file and pressing "overwrite" button when asked), because using the old headerfile NEVER worked for me (even when all the lsfs devices were ok and mounting just fine).
This is the ultra-short version of the story. If any details about what I did/tried in the meantime or how exactly my setup is built - please don't hesitate to ask!
The relevant part of the Logfile is: (If you need loglevel 3, I will of course provide it immediatly. But since loglevel 3 is really a very very big lot of data, I thought I'd try level 2 first)
I also tried mounting it using the current Beta, but the Beta just crashes and blames it on module ntdll.dll. So I don't have any usable logs from the Beta, sorry 
I now I'm running your software for free and in an ultra-unsupported environment (single host, VM, ...), but I would be unbelievably thankful If someone could point me in the right direction how to save the data on that lsfs volume.
Thanks a lot in advance!
Kindest Regards and Hopes
Daniel Sel
I normally never ask for help in a forum, especially not if I'm not paying for the product - it kind of feels like "abuse". However right now I'm in a really desperate situation where i seriously can't help myself and can't find any information, so I'm actually begging you, please help me
So before I attempt to tell you the story with a length equivalent to of half of the bible in a (very-) shortened version, let me say a very very very big thank you for creating such an awesome piece of software and giving free licenses for personal use! You guys seriously are the best. Amazing performance and great stability. It took me more than half a year to break something - which has literally never happened before (I do have a talent for breaking stuff
).I'll try to keep the story short:
My complete setup is one single physical server built by myself and running at home. I'm currently studying Computer Science and this system is my "playground". But over time this "playground" has collected a lot of (to me personally) valuable stuff, where I put a lot of hard work into, so I hope there is a way to fix this
I'm running StarWind V8 on a Windows Server 2012 R2 Virtual Machine on an ESXi 5.5 host. All Harddisks are on a RAID Controller (but configured as JBOD), passed through to the VM (means the OS has direct access).
On 2 HDD's with I created a Simple Mirror with Storage Spaces. Size = 3TB, FS = NTFS. On this Drive (From now on Called "R") I created a LSFS with L1 and L2 Cache and Size 2TB (yes, not very smart...) for storage of VM's.
So this was running (almost) perfectly since September of last year - until 2 days ago, drive R was full. The Software immediatly switched the LSFS device (lets call it "lsfs1") into read-only mode - that's when I noticed something was wrong.
So I went on and read everything about the infinitely-growing lsfs devices and thought: OK, that should be possible to fix. My idea was to copy the LSFS files over to another Drive (Drive "S", which is a RAID5 [Also Storage Spaces Software RAID] with about 10 TB of free space). After verifying all the files via checksum, I then deleted the files from "R" and created a new LSFS device there, happy to now mount both of them in ESXi and vMotion my VM's over. I tried to mount lsfs1 (the "old" one) by using the "use existing file" feature when creating a new device, but it failed - telling me "Invalid Storage selected". If I click OK and try over and over again (for like 20 - 30 times) I can get past that message, but in the end the mounting of the device fails anyway.
I should add, that everytime I try to add the device I create a new header file (by choosing the .spsp file and pressing "overwrite" button when asked), because using the old headerfile NEVER worked for me (even when all the lsfs devices were ok and mounting just fine).
This is the ultra-short version of the story. If any details about what I did/tried in the meantime or how exactly my setup is built - please don't hesitate to ask!
The relevant part of the Logfile is: (If you need loglevel 3, I will of course provide it immediatly. But since loglevel 3 is really a very very big lot of data, I thought I'd try level 2 first
)
Code: Select all
2/24 20:50:03.403 bf8 Sp: Creating LSFS device:
Directory: S:\BAKVM\
Prefix: VM
Exisiting: 1
Read-only: 1
Block size: 0
Size in blocks: 0
Page size: 0
Pages per superpage: 0
Threads: 1
Packets: 256
DD disabled: 0
Defrag: 0
Verify: 0
2/24 20:50:03.435 bf8 CACHE: *** CacheBase::init: Cache parameters: Mode = 3, Flags = 0, Size = 16 MB, SectorSize = 512, BlockExpiryPeriod = 0, DiskSizeInSectors = 0, Blocks = 0x0000000001F09E50
2/24 20:50:03.435 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [1, 0] size [0]
2/24 20:50:03.435 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [1, 0] size [0] completed
2/24 20:50:03.435 bf8 Sp: *** CStarPackCore::CreateEx Header version is 10
2/24 20:50:03.435 bf8 Sp: * CStarPackCore::CreateEx pHeader->DDStatus is [2], DD is disabled
2/24 20:50:03.435 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [1, 65536] size [0]
2/24 20:50:03.450 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [1, 65536] size [0] completed
2/24 20:50:03.450 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [1, 131072] size [0]
2/24 20:50:03.466 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [1, 131072] size [0] completed
2/24 20:50:03.466 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353130, 0] size [0]
2/24 20:50:03.528 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353130, 0] size [0] completed
2/24 20:50:03.544 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353131, 0] size [0]
2/24 20:50:03.575 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353131, 0] size [0] completed
2/24 20:50:03.606 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353132, 0] size [0]
2/24 20:50:03.622 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353132, 0] size [0] completed
2/24 20:50:03.653 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353133, 0] size [0]
2/24 20:50:03.669 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353133, 0] size [0] completed
2/24 20:50:03.700 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353134, 0] size [0]
2/24 20:50:03.716 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353134, 0] size [0] completed
2/24 20:50:03.747 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353135, 0] size [0]
2/24 20:50:03.763 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353135, 0] size [0] completed
2/24 20:50:03.794 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353136, 0] size [0]
2/24 20:50:03.810 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353136, 0] size [0] completed
2/24 20:50:03.841 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353137, 0] size [0]
2/24 20:50:03.872 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353137, 0] size [0] completed
2/24 20:50:03.888 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353138, 0] size [0]
2/24 20:50:03.919 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353138, 0] size [0] completed
2/24 20:50:03.935 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353139, 0] size [0]
2/24 20:50:03.966 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353139, 0] size [0] completed
2/24 20:50:03.981 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353140, 0] size [0]
2/24 20:50:04.013 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353140, 0] size [0] completed
2/24 20:50:04.028 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353141, 0] size [0]
2/24 20:50:04.028 1b1c Sp: *** CStarPackFileAsync::SeekReadWriteAsync ReadFile\WriteFile failed: Offset 3b000000, WinError 38
2/24 20:50:04.028 1b1c Sp: *** CStarPackSpaceSegmentFile::ReadWritePage Failed to read from file ( Segment index - 306, Offset in bytes - 5284823040, Buffer address - 00000000043B0000, IO length - 8388608 )! Status (38).
2/24 20:50:04.028 1b1c Sp: --- CStarPackSpaceSegmentFile::ReadWritePage(...) (line #959) 38
2/24 20:50:04.028 1b1c Sp: --- CStarPackSpaceSegmented::ReadWritePageInternal(...) (line #2080) 38
2/24 20:50:04.028 1b1c Sp: *** CStarPackSpaceSegmented::ReadWriteInternal ReadWritePageInternal returned error! (PageIndex: 353141, buffer address: 0x00000000043B0000, operation size: 8388608, write: 0, operation offset: 0).
2/24 20:50:04.028 bf8 Sp: --- CStarPackSpaceSegmented::ReadPage(...) (line #298) 38
2/24 20:50:04.028 bf8 Sp: * CStarPackCore::GetActualLastPage Error reading Page 353141
2/24 20:50:04.044 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353131, 0] size [8388608]
2/24 20:50:04.091 1b1c Sp: *** CStarPackSpaceSegmentFile::ReadWritePage Failed to read from file ( Segment index - 306, Offset in bytes - 5200936960, Buffer address - 00000000043B0000, IO length - 8388608 )! Status (38).
2/24 20:50:04.091 1b1c Sp: --- CStarPackSpaceSegmentFile::ReadWritePage(...) (line #959) 38
2/24 20:50:04.091 1b1c Sp: --- CStarPackSpaceSegmented::ReadWritePageInternal(...) (line #2080) 38
2/24 20:50:04.091 1b1c Sp: *** CStarPackSpaceSegmented::ReadWriteInternal ReadWritePageInternal returned error! (PageIndex: 353131, buffer address: 0x00000000043B0000, operation size: 8388608, write: 0, operation offset: 0).
2/24 20:50:04.091 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353131, 0] size [8388608] completed
2/24 20:50:04.106 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353131, 77824] size [0]
2/24 20:50:04.106 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353131, 77824] size [0] completed
2/24 20:50:04.106 bf8 Sp: * CStarPackCore::CreateEx Map at LU 0 is outdated. Restore is needed
2/24 20:50:04.106 bf8 Sp: * CStarPackCore::CreateEx One of the cache parameters [0 0] is 0, so just creating files
2/24 20:50:04.106 bf8 Sp: * CStarPackCore::CreateEx Core [00000000009D0000] created successfully.
2/24 20:50:04.106 bf8 Sp: * CStarPackCore::~CStarPackCore Device was switched to read-only
2/24 20:50:04.106 1b1c CACHE: *CacheBase::opThreadWorker: Stopping delayed write worker thread...
2/24 20:50:04.122 bf8 Sp: * CStarPackThread::~CStarPackThread SpThreadObjectDestroy( 0x0000000000000000 ) 487
2/24 20:50:04.278 bf8 Sw: *DDDisk_ControlRequest: Received control command for plug-in
2/24 20:50:04.278 bf8 Sp: Creating LSFS device:
Directory: S:\BAKVM\
Prefix: VM
Exisiting: 1
Read-only: 1
Block size: 0
Size in blocks: 0
Page size: 0
Pages per superpage: 0
Threads: 1
Packets: 256
DD disabled: 0
Defrag: 0
Verify: 0
2/24 20:50:04.310 bf8 CACHE: *** CacheBase::init: Cache parameters: Mode = 3, Flags = 0, Size = 16 MB, SectorSize = 512, BlockExpiryPeriod = 0, DiskSizeInSectors = 0, Blocks = 0x0000000001F0A0D0
2/24 20:50:04.310 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [1, 0] size [0]
2/24 20:50:04.310 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [1, 0] size [0] completed
2/24 20:50:04.310 bf8 Sp: *** CStarPackCore::CreateEx Header version is 10
2/24 20:50:04.310 bf8 Sp: * CStarPackCore::CreateEx pHeader->DDStatus is [2], DD is disabled
2/24 20:50:04.310 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [1, 65536] size [0]
2/24 20:50:04.325 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [1, 65536] size [0] completed
2/24 20:50:04.325 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [1, 131072] size [0]
2/24 20:50:04.341 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [1, 131072] size [0] completed
2/24 20:50:04.341 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353130, 0] size [0]
2/24 20:50:04.403 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353130, 0] size [0] completed
2/24 20:50:04.419 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353131, 0] size [0]
2/24 20:50:04.450 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353131, 0] size [0] completed
2/24 20:50:04.482 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353132, 0] size [0]
2/24 20:50:04.497 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353132, 0] size [0] completed
2/24 20:50:04.528 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353133, 0] size [0]
2/24 20:50:04.544 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353133, 0] size [0] completed
2/24 20:50:04.575 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353134, 0] size [0]
2/24 20:50:04.591 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353134, 0] size [0] completed
2/24 20:50:04.622 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353135, 0] size [0]
2/24 20:50:04.638 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353135, 0] size [0] completed
2/24 20:50:04.669 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353136, 0] size [0]
2/24 20:50:04.685 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353136, 0] size [0] completed
2/24 20:50:04.716 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353137, 0] size [0]
2/24 20:50:04.747 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353137, 0] size [0] completed
2/24 20:50:04.763 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353138, 0] size [0]
2/24 20:50:04.794 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353138, 0] size [0] completed
2/24 20:50:04.810 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353139, 0] size [0]
2/24 20:50:04.841 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353139, 0] size [0] completed
2/24 20:50:04.856 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353140, 0] size [0]
2/24 20:50:04.888 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353140, 0] size [0] completed
2/24 20:50:04.903 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353141, 0] size [0]
2/24 20:50:04.903 1314 Sp: *** CStarPackFileAsync::SeekReadWriteAsync ReadFile\WriteFile failed: Offset 3b000000, WinError 38
2/24 20:50:04.903 1314 Sp: *** CStarPackSpaceSegmentFile::ReadWritePage Failed to read from file ( Segment index - 306, Offset in bytes - 5284823040, Buffer address - 00000000043B0000, IO length - 8388608 )! Status (38).
2/24 20:50:04.903 1314 Sp: --- CStarPackSpaceSegmentFile::ReadWritePage(...) (line #959) 38
2/24 20:50:04.903 1314 Sp: --- CStarPackSpaceSegmented::ReadWritePageInternal(...) (line #2080) 38
2/24 20:50:04.903 1314 Sp: *** CStarPackSpaceSegmented::ReadWriteInternal ReadWritePageInternal returned error! (PageIndex: 353141, buffer address: 0x00000000043B0000, operation size: 8388608, write: 0, operation offset: 0).
2/24 20:50:04.903 bf8 Sp: --- CStarPackSpaceSegmented::ReadPage(...) (line #298) 38
2/24 20:50:04.903 bf8 Sp: * CStarPackCore::GetActualLastPage Error reading Page 353141
2/24 20:50:04.919 bf8 Sp: * CStarPackSpaceSegmented::ReadPage ReadPage [353131, 0] size [8388608]
2/24 20:50:04.966 1314 Sp: *** CStarPackSpaceSegmentFile::ReadWritePage Failed to read from file ( Segment index - 306, Offset in bytes - 5200936960, Buffer address - 00000000043B0000, IO length - 8388608 )! Status (38).
2/24 20:50:04.966 1314 Sp: --- CStarPackSpaceSegmentFile::ReadWritePage(...) (line #959) 38
2/24 20:50:04.966 1314 Sp: --- CStarPackSpaceSegmented::ReadWritePageInternal(...) (line #2080) 38
2/24 20:50:04.966 1314 Sp: *** CStarPackSpaceSegmented::ReadWriteInternal ReadWritePageInternal returned error! (PageIndex: 353131, buffer address: 0x00000000043B0000, operation size: 8388608, write: 0, operation offset: 0).
2/24 20:50:04.966 bf8 Sp: --- CStarPackSpaceSegmented::ReadPage(...) (line #298) 38
2/24 20:50:04.966 bf8 Sp: --- CStarPackCore::CheckPage(353131) (line #14203) 38
2/24 20:50:04.966 bf8 Sp: *** CStarPackCore::LoadLUList Page with last LU list [10289152] is broken. Device is inconsistent
2/24 20:50:04.966 bf8 Sp: --- CStarPackCore::LoadLUList() (line #19045) 38
2/24 20:50:04.966 bf8 Sp: --- CStarPackCore::CreateEx() (line #3096) 38
2/24 20:50:04.966 bf8 Sp: * CStarPackCore::~CStarPackCore Device was switched to read-only
2/24 20:50:04.966 1314 CACHE: *CacheBase::opThreadWorker: Stopping delayed write worker thread...
2/24 20:50:04.981 bf8 Sp: * CStarPackThread::~CStarPackThread SpThreadObjectDestroy( 0x0000000000000000 ) 487
2/24 20:50:04.981 bf8 Sp: *** SpCoreObjectCreateFromParams 38 0x00000000025D7E60 *ppCore->pCore->Create( 0x00000000025D7E80 ) 38
2/24 20:50:04.981 bf8 Sp: --- SpCoreObjectCreateFromParams(0x00000000025D7E60, 0x00000000025D7E80) (line #4472) 38
2/24 20:50:04.981 bf8 Sw: *** DDDisk_GetDDStorageInformation: Create SP object failed, status ( 38 ).
2/24 20:50:04.981 bf8 Sw: *** DDDisk_ControlQueryStorageCacheInfo: Unable to open storage (error 38).
2/24 20:50:04.981 bf8 Sw: *DDDisk_CompleteControlRequest: Return param: 'Query' = 'Failed: unable to open storage (error 38).'.I now I'm running your software for free and in an ultra-unsupported environment (single host, VM, ...), but I would be unbelievably thankful If someone could point me in the right direction how to save the data on that lsfs volume.
Thanks a lot in advance!
Kindest Regards and Hopes
Daniel Sel