dedup file growth

gstephenson · Tue Jan 17, 2012 7:23 pm

Hi...

Why does a dedup disk continue to grow even when I delete its contents? I created a test dedup disk at 200MB, added some files until it was full, deleted all the files, then added more files in. If I keep doing this the physical size grows beyond 200MB. Is that supposed to happen?

Thanks.

Wed Jan 18, 2012 10:23 am

B/c you store new data and it's just added to database. From block device point of view there's no difference in writing set of 0xFF or 0xAA patterns to some sector - data is always considered new... We'll provide external app you'd run @ background to deal with known file systems and re-assign unused blocks as NTFS does not do it automatically.

gstephenson wrote:Hi...

Why does a dedup disk continue to grow even when I delete its contents? I created a test dedup disk at 200MB, added some files until it was full, deleted all the files, then added more files in. If I keep doing this the physical size grows beyond 200MB. Is that supposed to happen?

Thanks.

gstephenson · Wed Jan 18, 2012 4:43 pm

anton (staff) wrote:B/c you store new data and it's just added to database. From block device point of view there's no difference in writing set of 0xFF or 0xAA patterns to some sector - data is always considered new... We'll provide external app you'd run @ background to deal with known file systems and re-assign unused blocks as NTFS does not do it automatically.

Ok, makes sense. So will this external app be available for download on the site soon?

Wed Jan 18, 2012 9:36 pm

Yes, with V6 pre-release. Early February I think.

gstephenson wrote:
anton (staff) wrote:B/c you store new data and it's just added to database. From block device point of view there's no difference in writing set of 0xFF or 0xAA patterns to some sector - data is always considered new... We'll provide external app you'd run @ background to deal with known file systems and re-assign unused blocks as NTFS does not do it automatically.
Ok, makes sense. So will this external app be available for download on the site soon?

Aitor_Ibarra · Thu Jan 19, 2012 11:47 am

Surely this would have to be run with the target offline? If you have spare disk space, especially on different set of spindles, it would make more sense to me to create a new de-dupe target and then copy the data across from the initiator side. How you do this would depend on the application, but you could probably do it more quickly and with less disruption. And without the worry that this tool won't understand the filesystem - in Windows land you've now got ReFS support to worry about too...

E.g if it's an Exchange mailstore, just create a new one on the new target and migrate the mailboxes. If it's a Hyper-V VM, use SCVMM to do a storage migration...

However if it can be done with the target online, even if it takes forever, it could be a very useful tool for people who don't have spare disks...

Thu Jan 19, 2012 5:46 pm

No, you don't need to put anything off-line. We'll handle time consuming operations @ the background to save IOPS.

ReFS has the same "free space bitmap" concept as it's basically stripped down version of NTFS.

Other file systems (like VMFS) can deallocate free space automatically as they know how to tell to array particular block pattern is not used. We do understand them. NTFS is not that smart.

Aitor_Ibarra wrote:Surely this would have to be run with the target offline? If you have spare disk space, especially on different set of spindles, it would make more sense to me to create a new de-dupe target and then copy the data across from the initiator side. How you do this would depend on the application, but you could probably do it more quickly and with less disruption. And without the worry that this tool won't understand the filesystem - in Windows land you've now got ReFS support to worry about too...

E.g if it's an Exchange mailstore, just create a new one on the new target and migrate the mailboxes. If it's a Hyper-V VM, use SCVMM to do a storage migration...

However if it can be done with the target online, even if it takes forever, it could be a very useful tool for people who don't have spare disks...

ML455SD · Fri Mar 28, 2014 4:03 pm

Hi there,

I am having this same issue. Depulicating Disk growth is out of control. Knocks the iSCSI storage offline.

Looked for the external "cleanup" tool in downloads but I do not see anything.

Help.

Mon Mar 31, 2014 8:12 am

I guess it is a good idea for you to try out our RC of v8. We improved the DD in it. You cna get it by signing up for our beta program by using hte link below:
www.starwindsoftware.com/starwind-v8-beta