Grossly bloated deduplicated target

[ralphw]

This morning I came in and found that my monitoring system was giving me alarms on my SAN server getting close to running out of diskspace.

I have Starwind 5.8 installed on Windows 2008 R2. My target is on a 1.22TB raid array by itself (there is nothing else on that volume). I've got about 62.5GB free on that volume, even though my target is only 1TB.

I look in my vmware server (ESXI 5), which for right now is the only one serving VMs off of that target, and my VM Storage is show 1TB of capacity and about 838GB free!

What's using up all this disk space and how do I correct it?

See my attached screenshots:

Storage volume disk usage

SWSS1.jpg (53.64 KiB) Viewed 27126 times

Data files being used by Starwind

SWSS2.jpg (87.61 KiB) Viewed 27129 times

Starwind target configuration

SWSS3.jpg (98.01 KiB) Viewed 27132 times

[ralphw]

One more screenshot (I could only attach 3 in my last post)

VMware storage use

SWSS4.jpg (34.26 KiB) Viewed 27124 times

Please help.

[ralphw]

It looks like my growth is mostly going into the .spbitmap and .spdata files, with a very small amount of grown in the .spmetadata file. Since my original post, I'm down another 1.2GB, with about half of that space going into the .spbitmap and .spdata files.

It also looks like the gradual consumption has been going on for quite some time. The VMs that are on that machine don't really change much in terms of disk usage, so I'm not really sure what's going on. There was a big drop late in June and I think that's when I migrated some VMs off of another host onto this storage, but I migrated them back after a couple of days, so they are no longer in the SAN.

Graph showing free space on target volume

SWSS5.jpg (45.55 KiB) Viewed 27098 times

[Anatoly (staff)]

Just one question before we proceed - you have written the data,anad deleted it,a after writen some other data and deleted it again, correct?
I`m asking because for now we do not have block re-use or block deletion support, which means that if data will be erased, it doesnt mean that the block, where it was stored will disappear, moreover, nothing will be able to be stored there (on them). This will definitely be fixed in next versions. Stay tuned.

So, if my assumption is correct then the only way that I see now is to simply create new drive (device) and move all the data on it.

[ralphw]

Thanks for the reply, Anatoly.

Wow, if I'm understanding you correctly, that's kind of a big bug. Does this only apply to deduplicated disks? I don't remember this being an issue on other installs I have done where I have used some other type of starwind disk.

Going by my graph, it only gradually uses up space. Except for the big dip late in June, where I moved some VMs around and they were temporary stored on this SAN. So in that case, what you're saying makes sense because after they got moved back off of that SAN, the space was never returned.

As for the gradual decrease...the VMs that are stored there are mostly static VMs. There is a Windows terminal server on there, which only gets very light use, but I suppose as users log in their profile gets copied to the server and then deleted when they log off, so that would explain some usage.

There is also a VM that serves as a Windows DFS recplication point on there which does replicate from some active data. Depending on how that works, I suppose the replica is written and then the data is changed on another server and the new data is written and the old replica is deleted. Maybe that explains the gradual free space decline.

Sounds like I've got alot of work to do pretty soon.

[ralphw]

I'm also finding that copying the data off and creating a new target is more difficult than it sounds in an free ESXi environment. Apparently transferring files (the actual vmware files themselves) off of ESXi using SCP or Veeam's FastSCP is very slow (10-20MB/s), I guess by VMware's design from what I'm finding in google searches, so I'm in a bit of a spot now.

Not only that, but when they copy to my workstation, if they are thin provisioned disks in VMware, they seem to convert to thick disks during that process. I'm not that far yet, but I'm hoping that when I go to put them back into VMware and onto storage, I will have an option to upload them back as thin provisioned disks again...but with how my luck is going so far on this, I doubt it.

Do you guys have any recommendations on how I can speed up that process and/or to keep my thin provisioning I have set up currently in my vmware disks?

[anton (staff)]

You absolutely should NOT use thin-provisioned disks with deduplication. Basically you make double work for everyone. Deduplication IS thin-provisioning by design. Layering extra layer of TP over it makes data more scattered, more random I/O => low performance.

I'm also finding that copying the data off and creating a new target is more difficult than it sounds in an free ESXi environment. Apparently transferring files (the actual vmware files themselves) off of ESXi using SCP or Veeam's FastSCP is very slow (10-20MB/s), I guess by VMware's design from what I'm finding in google searches, so I'm in a bit of a spot now.

Not only that, but when they copy to my workstation, if they are thin provisioned disks in VMware, they seem to convert to thick disks during that process. I'm not that far yet, but I'm hoping that when I go to put them back into VMware and onto storage, I will have an option to upload them back as thin provisioned disks again...but with how my luck is going so far on this, I doubt it.

Do you guys have any recommendations on how I can speed up that process and/or to keep my thin provisioning I have set up currently in my vmware disks?

[ralphw]

You absolutely should NOT use thin-provisioned disks with deduplication. Basically you make double work for everyone. Deduplication IS thin-provisioning by design. Layering extra layer of TP over it makes data more scattered, more random I/O => low performance.

That's good to know now.

I`m asking because for now we do not have block re-use or block deletion support, which means that if data will be erased, it doesnt mean that the block, where it was stored will disappear, moreover, nothing will be able to be stored there (on them). This will definitely be fixed in next versions. Stay tuned.

Does this only apply to deduplacted disks? I need to know this so that I choose the right disk when I recreate the target.

[Anatoly (staff)]

Yes, you can find this when running DeDupe ddevice only.

[ralphw]

Okay, thanks. I'll create some other disk type and maybe give that a try again when that issue gets worked out. But for now, at least I have a plan.

[Anatoly (staff)]

Always glad to help you!

BTW, if this is not production environment you are welcomed to install version 6 that have block re-use.

[dtrounce]

1. I'm using V6.0.4768. The "deletion support" / block re-use option is marked "This feature is experimental".

I'm assuming that this means something like "We've tested it pretty extensively and not found problems. However we don't want to be held responsible if you find an bug and it wipes all your data". How comfortable would you be using deletion support in production, now or soon?


2. I notice that using this version with deduplicated HA targets, the .spdata files seem to constantly grow (by several GBs) even >18 hours after I finished writing the last data. The files are mostly Windows Server 2012 VHDX files for VMs, which are test installs so are pretty empty and are mostly idle. I suppose there may be Windows background processes doing a little work - they have test guest clusters inside them - but they too should be pretty stable.

I was initially assuming that this is due to data from one target still being copied to it's partner. But >18 hours in, that seems unlikely - the targets should have synced by now (I'd be worried if they hadn't, for HA). I have fast disk arrays with a GbE connection between the hosts. The starwindservice.exe on both hosts is frequently reading ~500KB/sec and writing ~50KB/sec to the .spdata files and sending network traffic to its partner. What is going on? When should I expect my .spdata files to stop growing?

[Max (staff)]

Hi David,
The block re-use feature is still experimental so there are several things to be fixed before releasing it.
We're currently working on this and expect an updated version to be available by the end of the month.

[dtrounce]

I have a 100GB de-deduplicated HA target using block reuse. It has 85GB of heavily duplicated VM data which started at 30GB deduplicated.

But after a few weeks of idle activity, this has grown to a pair of 208GB .spdata files, and shows no sign of stopping growing!

I am using 6.0.4768. I see the latest download is now 6.0.4837. Does this fix this problem?

[anton (staff)]

Did you enable delete (makes sense for VM production, no-delete is for "collection" or data backups).