Why backup with QuDedupe is 10x faster?

Backup, Restore, Netbak Replicator, Cloud Storage Services
Post Reply
janwer
Getting the hang of things
Posts: 97
Joined: Wed Dec 26, 2018 10:16 pm

Why backup with QuDedupe is 10x faster?

Post by janwer »

HBS3. Two Qnaps on 1Gbps LAN. RTRR. TVS-951x to TS-431P2. 4.3.6 on both.

New backup without versioning, without QuDedup 8MB/s. CPU1 8-16%, CPU2 50%.
Another new backup with QuDedup 70MB/s. Both CPUs 70-80%

Why?
TVS-951x 16GB RAM, 3x 0,5 TB SSD RAID5, 5x 8 TB RAID5
TS-431P2 8GB RAM, 4x 3 TB HGST RAID5
Unraid 8GB RAM, 10x 3TB HGST
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Why backup with QuDedupe is 10x faster?

Post by P3R »

Not much is known about QuDedup but I can guess that it could be because it use deduplication (effectiveness depend on your data), it may also use compression and it only need to write to a single backup file instead of creating a complete folder structure with a lot of small files?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
janwer
Getting the hang of things
Posts: 97
Joined: Wed Dec 26, 2018 10:16 pm

Re: Why backup with QuDedupe is 10x faster?

Post by janwer »

10x speed is observed on network. After compression, deduplication. I set up few backup jobs for testing and with Qudedupe is 50-70MB/s. I don't know what should be speed with older HBS? Was it 8MB/s on 1Gbps LAN?
TVS-951x 16GB RAM, 3x 0,5 TB SSD RAID5, 5x 8 TB RAID5
TS-431P2 8GB RAM, 4x 3 TB HGST RAID5
Unraid 8GB RAM, 10x 3TB HGST
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: Why backup with QuDedupe is 10x faster?

Post by Moogle Stiltzkin »

NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Why backup with QuDedupe is 10x faster?

Post by P3R »

janwer wrote: Fri Sep 13, 2019 7:44 am I don't know what should be speed with older HBS? Was it 8MB/s on 1Gbps LAN?
There is no speed that it "should" be. Speed will depend on NASes used, their configuration, job configuration and most importantly what the data being transferred consist of. Depending on those variables different users will most likely see different incomparable results.

If you want to know the effect of folder structure and file creation only, test with a single very large file and see how the two different backup methods compare.

If you want to see the effect of deduplication/compression only, test with a single very large strongly encrypted file. That should render deduplication/compression virtually useless so any differences you see then will be from other effects.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
User avatar
Toxic17
Ask me anything
Posts: 6469
Joined: Tue Jan 25, 2011 11:41 pm
Location: Planet Earth
Contact:

Re: Why backup with QuDedupe is 10x faster?

Post by Toxic17 »

P3R wrote: Fri Sep 13, 2019 2:57 pm There is no speed that it "should" be. Speed will depend on NASes used, their configuration, job configuration and most importantly what the data being transferred consist of. Depending on those variables different users will most likely see different incomparable results.
I'm afraid some users think its just a straight file copy and look at their own copying experiences from PC to NAS as a benchmark.
Regards Simon

Qnap Downloads
MyQNap.Org Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following


NAS: TS-673A QuTS hero h5.1.2.2534 • TS-121 4.3.3.2420 • APC Back-UPS ES 700G
Network: VM Hub3: 500/50 • UniFi UDM Pro: 3.2.9 • UniFi Network Controller: 8.0.28
USW-Aggregation: 6.6.61 • US-16-150W: 6.6.61 • 2x USW Mini Flex 2.0.0 • UniFi AC Pro 6.6.62 • UniFi U6-LR 6.6.62
UniFi Protect: 2.11.21/8TB Skyhawk AI • 3x G3 Instants: 4.69.55 • UniFi G3 Flex: 4.69.55 • UniFi G5 Flex: 4.69.55
janwer
Getting the hang of things
Posts: 97
Joined: Wed Dec 26, 2018 10:16 pm

Re: Why backup with QuDedupe is 10x faster?

Post by janwer »

Qdedup is more CPU consuming so it should be faster.
I did test copying large movie file. No need to compress.
Manual recomends do not use Qudedup for movie files and
I did 2TB first backup with versioning and Qudedupe. It took about 10 hours.
Now I have to backup 2,4TB without versioning and without Qudedupe. Estimated time is 70 hours.

Why simpler and less CPU consuming way is so much slower?
Why 8MB/s on my 1Gbps LAN?
Source CPU's is 5-17%, destination CPU 34%.
What transfer should I expect? Is it normal or error?

If I copy this data between two QNAPs using PC/SMB full 1Gbps speed is no problem - 60MB/s.

Now I run simultaneously two backup jobs, both without Qudedupe. I have double speed! 16MB/s. Source CPU 6-36%, destination CPU 60-80%.

Edit.
I deleted backup job. Creating the same job but with Qudedupe. Transfer 50MB/s, estimated time 7-8 hours. CPUs 42% and 58%.
TVS-951x 16GB RAM, 3x 0,5 TB SSD RAID5, 5x 8 TB RAID5
TS-431P2 8GB RAM, 4x 3 TB HGST RAID5
Unraid 8GB RAM, 10x 3TB HGST
Post Reply

Return to “Backup & Restore”