De-Duplication in 5.6

smacrae · Tue May 24, 2011 1:37 am

I have downloaded and installed the free version of StarWind 5.6 and enabled the de-duplication feature.

I am using XenServer(s) to connect to the iSCSI store without any issues.

When I look on the local hard-drive store of the StarWind server I notice 3 files have been created, one is obviously the main store as it is now up to 100Gb's in size as I have moved across 600Gb's of VM's from a pervious version of StarWind, so de-duplication seems to be working very nicely.

My question/problem is on the local StarWind hard-drive server side I see 100Gb's'ish of space has been allocated to the 600Gb's of images that have been moved across which suggested de-duplcation is working extremely well however the problem is when I look on the XenServer side of things it reports that it has used 600Gb's, the VM's true allocated hard-drive size. So XenServer is seeing the size of the HDD's for each VM rather than the actual space they're taking up on the StarWind server.

The problem with the above is I will still only be able to use the allocated space rather than the whole drive. So before I had a 2TB store that I was using up, with de-duplication I will still only be able to store 2TB's of images with what looks like is going to be around 1TB - 1.5Tb's of free space that will not be able to be utilised by StarWind's connections from the XenServer(s).

Can anyone shed some light on this? Is this by design or have I done something wrong? If it is by design how am I meant to use up all the HDD's space that is really free?

Can anyone help?

Scot

Tue May 24, 2011 11:17 am

Scot,
In general, you can create deduplicated virtual device with the size much larger that size of the underlying storage. For example, you can create 2TB virtual disk on the 500GB HDD. Xen Server will get more space than is available on the physical storage.
In a case of 2TB physical storage you can create several virtual disks of size 2TB to utilize HDDs space. In this case you will need to keep an eye on free space on the physical drive.
We will automate the functionality for watching free disk space on the drives that are used by thin-provisioned virtual devices in one of the next versions.
By the way, maximum size for deduplicated device will be increased in the next version.

smacrae · Tue May 24, 2011 11:54 pm

Thanks Alex,

Thanks for the update, based on what you've said below am I correct in assuming that a 1 to 4 ratio is a reasonably safe overcommit level as long as one is keeping an eye on things?

Also, is there a 2TB limit on deduplication now?

Thanks for the update.

Wed May 25, 2011 8:43 am

No. There's no limit on 2TB per deduplication volume. There's a limit of 2^32 addressed deduplication blocks. So with 512 bytes per block and only one LUN allocated for the whole volume you'd indeed have 2TB of space. But 512 bytes is a "hardcore" mode, I don't think you have enough of RAM to store hash tables and enough of CPU cycles to calculate hashes... 4KB per block sounds like a reasonable setting. With 4KB per block you have 16TB of space per LUN. With 256 LUNs allocated it's 4096TB of user-addressable space.

smacrae wrote:Thanks Alex,

Thanks for the update, based on what you've said below am I correct in assuming that a 1 to 4 ratio is a reasonably safe overcommit level as long as one is keeping an eye on things?

Also, is there a 2TB limit on deduplication now?

Thanks for the update.

smacrae · Wed May 25, 2011 4:09 pm

Thanks again Alex,

I the 4k block size is unable to be modified on this release of StarWind? I'm assuming this is the case?

I'm running a dedicated Xenon 5620 server with 8Gb's of RAM and a dedicated RAID 10, would this be okay for the deduplication option? I'm also assuming I cannot run two targets off of this server in terms of the hash/RAM issues you mentioned before?

Wed May 25, 2011 4:42 pm

You can pick up block size of your choice. 4KB is recommended as a good compromise between performance and deduplication ratio.

smacrae wrote:Thanks again Alex,

I the 4k block size is unable to be modified on this release of StarWind? I'm assuming this is the case?

I'm running a dedicated Xenon 5620 server with 8Gb's of RAM and a dedicated RAID 10, would this be okay for the deduplication option? I'm also assuming I cannot run two targets off of this server in terms of the hash/RAM issues you mentioned before?

smacrae · Wed May 25, 2011 5:52 pm

OK, when I added the store it seemed to be locked on 8K, I guess I was wrong. If I have filled up the store that is currently set 8K Blocks, Metadata of 3998 and Data Cache size of 10 am I able to delete this target and rea-dd without loosing what I have already stored or am I force to start again?

Thanks again...

Wed May 25, 2011 7:30 pm

The whole idea is to create target "from scratch".

smacrae wrote:OK, when I added the store it seemed to be locked on 8K, I guess I was wrong. If I have filled up the store that is currently set 8K Blocks, Metadata of 3998 and Data Cache size of 10 am I able to delete this target and rea-dd without loosing what I have already stored or am I force to start again?

Thanks again...

smacrae · Thu May 26, 2011 1:22 am

Sure, that I understand...but what if you need to move the store or increase the space, you could on a non-deduplicated target.

So I am assuming from your answer it is not possible on a deduplicated target?

Thu May 26, 2011 6:33 am

You cannot mix different volume types. You can add extra LUN to already created target (since 5.7 or 5.

but you would not be able to do any mixes.

smacrae wrote:Sure, that I understand...but what if you need to move the store or increase the space, you could on a non-deduplicated target.

So I am assuming from your answer it is not possible on a deduplicated target?

smacrae · Thu May 26, 2011 10:40 pm

Let me ask the question as simply as I can, sorry I am no expert (as I am sure you can see):

Question 1:

- I have created the store using de-duplication
- After creating this store and reading your advice I would like to change the store from 8K blocks to 4K blocks, can I do this or must all of the data be lost?

Question 2

- I have created the store using de-duplcation
- I max out space and decide I would like to move the store to a larger RAID Array and extend the size. I can do this easily when using a normal store without de-duplication (an IMG file). However can I extend with a de-duplication store?

Sorry, but I still really do not have an answers to the above two questions which I've attempted to ask....I guess you may have answered and I am too stupid to see the answer. If you could be so kind as to tell me in layman's terms it would be greatly appreciated.

Fri May 27, 2011 7:21 am

A1: You cannot change deduplication block size on already created storage.

A2: You need to create extra LUN and extend volume on it (the same you do with hardware RAID). But it's not in the current experimental version, 5.7 release or 5.8

We're not stupid here. Neither you nor me. Just using non-native language for communications

smacrae wrote:Let me ask the question as simply as I can, sorry I am no expert (as I am sure you can see):

Question 1:
- I have created the store using de-duplication
- After creating this store and reading your advice I would like to change the store from 8K blocks to 4K blocks, can I do this or must all of the data be lost?
Question 2
- I have created the store using de-duplcation
- I max out space and decide I would like to move the store to a larger RAID Array and extend the size. I can do this easily when using a normal store without de-duplication (an IMG file). However can I extend with a de-duplication store?
Sorry, but I still really do not have an answers to the above two questions which I've attempted to ask....I guess you may have answered and I am too stupid to see the answer. If you could be so kind as to tell me in layman's terms it would be greatly appreciated.