Is dedupe a realistic requirement?

[ppcrobcole]

I understand this is an old thread but and FYI I've been able to use Dedupe on my QNAP by Creating and ISCSI share that I map onto a 2012 Server either via Native 2012 ISCSI or via VMWARE Mapping.
Then add the Volume as a 2012 Server Volume And then from the 2012 Server enable dedupe on the Volume.

It is great for storing backups etc and general use data.

[Don]

Well that is not really using dedupe on the NAS. Dedupe is running on a 2012 server and not on the NAS. The NAS is only the storage medium.

[sWORDs]

+1

[cbeerse]

Thinking about deduplication on disks (as in spindles, not ssd-s): blocks are compared and double blocks are stored only once. Hence, the first file with the block in it is stored continuously on disk, no problem so far. The next file with the same block is stored on disk with the double block pointing to the first one. This new file is fragmented by design. In the end, files with random double blocks can be stored in a spagetti-fragmentation due to the deduplicaiton. Hence, if it is for usability, I might not like this kind of fragmentation-by-deduplication.

On the other hand, the way Apple-s time-machine does its file-based deduplication can be used on qnaps since ages (since it uses rsync and a unix style filesystem): The first backup does create a fresh tree of files. Successive backups only add new or changed files. The files that have not been touched are linked from the previous backup using the '--link-dest=dir' option from rsync.

[schumaku]

On the other hand, the way Apple-s time-machine does its file-based deduplication can be used on qnaps since ages (since it uses rsync and a unix style filesystem): The first backup does create a fresh tree of files.

When I think about the regular Time Machine indications of having to create a fresh full backup (informing the user to loose the history) on _both_ "native" Apple Time Capsules and NAS Time Machine destinations - not sure that's a feature is something I want on the regular storage

[cbeerse]

On the other hand, the way Apple-s time-machine does its file-based deduplication can be used on qnaps since ages (since it uses rsync and a unix style filesystem): The first backup does create a fresh tree of files.

When I think about the regular Time Machine indications of having to create a fresh full backup (informing the user to loose the history) on _both_ "native" Apple Time Capsules and NAS Time Machine destinations - not sure that's a feature is something I want on the regular storage

I have to admit, I donnot know details on apples time capsules and such. The rsync way I'm using at home does create a complete tree-structure. The files are effectively the same where they have not been changed and they are new for changed and new files. That is all based on the unix filesystem feature of hard-links. If an old version is removed, the tree-structure is removed and all the hardlinks in it to the files are cleared. Only the files that have no hardlinks left are effectively removed. So there is no need to do a fresh full backup since every backup is effectively a full backup, only the old data is not transferred during incremental backups.

Some detail I found out with the above: if a backup is stopped/crashed half-way that backup is not complete as you'd expect. The next backup will find this half-way backup and use that as its base, hence, more data is transferred and the deduplication has a glitch there.

[schumaku]

Neither Time Machine nor these applications based on rsync and making use of hard links are anywhere near to an universal de-duplication I'm afraid.

[cbeerse]

Neither Time Machine nor these applications based on rsync and making use of hard links are anywhere near to an universal de-duplication I'm afraid.

True as in a violin is a musical instrument but a musical instrument is not necessary a violin, it can also be a piano. Universal de-duplication is no implementation, it is a method to store a lot of data in less space by recognizing doubles and storing them less frequent. What kind of deduplication do you fancy, each one has its advantages and its disadvantages, both on writing on reading and in maintenance. In disk-storage I see block-deduplication and file-deduplication as reasonable implementations.

Since it is fairly easy to recognize file duplication in successive backups, it is my preferred way on low-power systems like my qnap 419 nas. The ms-exchange mail server at the office has a similar way of deduplication by message. The big san in the server-room does its block-level deduplication at night and needs its memory-buffers and cpu-power, where this memory buffers are used in day-time for file-reconstruction, to overcome the massive file fragmentation.

[schumaku]

That's an application to avoid duplication to be correct. What-if say Mac users, Windows users, and Linux users write the same file n-times by SAMBA, AFP, and NFS ...? I don't want to fight with you, ok agree - it's de-duplication by an rsync based application.

[cbeerse]

That's an application to avoid duplication to be correct. What-if say Mac users, Windows users, and Linux users write the same file n-times by SAMBA, AFP, and NFS ...? I don't want to fight with you, ok agree - it's de-duplication by an rsync based application.

Well, if these users write the same file, it might be the same file to your view, but on most network-filesystems that is a different file as it has a different timestamp. With file deduplication, the file-properties as name, size and (write) timestamps are the first items to check.

How do you think block-based deduplication will find such a file being the same? It does not, as it does not look at the file specification. It only looks at the blocks that make the file. It does compare every block with other blocks, maybe on hash-base, maybe bit-by-bit (likely both). Then it removes the new block and changes the pointers to using the old one. It does this on every write action. Hence in your occasion, it accepts 3 different files, generates hashes for every block (maybe the same as for raid5, maybe different onces...) and start comparing those. Then you ask yourself why writing to such a device is slow... Next step, if you read those files, you wonder why the disks are crunching. That's because the files are fragmented block by block.

[Anata mo]

Well as you write:

...The NAS is only the storage medium.

then that fits just fine, because since he's storing the files on the iSCSI target, that is on the NAS, then data is only being written once with Dedup... So the description is fine "Dedupe on the NAS" it's just not being performed by the NAS it self

I'm also using Windows Server 2012 R2 with Dedupe and that saves like +80% of my usage on the NAS in this way... so I would agrre with "ppcrobcole" that its a great way of doing it... at least until QNAP support it natively with fx. Permabit or the likes

So:
+1
from me too!

[storageman]

All forms of dedupe have considerable load on processors (as much as 30%).
Therefore dedupe is more relevant to backup targets than day to day live storage.
Only storage with Xeon type processors can cope with the kind of load it imposes.

[forkless]

You have to look at the environment in which you want to apply a de-duplication mechanism. For most of QNAPs users it's not really hugely beneficial. In theory it sounds nice to have no duplicates on your filesystem, but most of us here are in a single (few) user environment. Duplication will be practically non-existent.

If you are running a QNAP for business purposes in a production environment... well then shame on you. I would use QNAP at best for lab/proof of concept set-ups.

[Briain]

...In theory it sounds nice to have no duplicates on your filesystem, but most of us here are in a single (few) user environment. Duplication will be practically non-existent.

Ah, if only I were that organised/competent!

Actually, being serious for a moment - unusual for me - I'm just wondering if there's a known utility to seek out and log exact duplicates and their paths (such that I could manually investigate/delete them)? It isn't something I'd even considered until reading this thread (I doubt I have many, but it would be mildly interesting to find that out; I'd bet there will be a few).

Bri

[lleone]

I'm running a Windows Home Server v1 (based on Win2003R2) on a single core 512MB ram virtual machine just to backup 7 family client (personal/work).
It does single instance storage (block based dedupe) , bare metal restore, backup rotation ( it keeps the last 54 backups for every single client, one a week ) and it copes it very well using standard ntfs formatted disks (unlike W2012r2 as far as I know).

I have not found out yet a similar (low cost) backup software and I plan to run it as a virtual machine on a TS-453pro.

I'd love if Qnap came out with native dedupe and such a backup software.
(or if someone could point me to alternatives)

Thanks,
L.