Why backup with QuDedupe is 10x faster?

Backup, Restore, Netbak Replicator, Cloud Storage Services
Post Reply
janwer
Starting out
Posts: 29
Joined: Wed Dec 26, 2018 10:16 pm

Why backup with QuDedupe is 10x faster?

Post by janwer » Wed Sep 11, 2019 11:28 pm

HBS3. Two Qnaps on 1Gbps LAN. RTRR. TVS-951x to TS-431P2. 4.3.6 on both.

New backup without versioning, without QuDedup 8MB/s. CPU1 8-16%, CPU2 50%.
Another new backup with QuDedup 70MB/s. Both CPUs 70-80%

Why?
TVS-951x TS-431P2 Unraid

P3R
Guru
Posts: 10892
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Why backup with QuDedupe is 10x faster?

Post by P3R » Fri Sep 13, 2019 7:09 am

Not much is known about QuDedup but I can guess that it could be because it use deduplication (effectiveness depend on your data), it may also use compression and it only need to write to a single backup file instead of creating a complete folder structure with a lot of small files?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

janwer
Starting out
Posts: 29
Joined: Wed Dec 26, 2018 10:16 pm

Re: Why backup with QuDedupe is 10x faster?

Post by janwer » Fri Sep 13, 2019 7:44 am

10x speed is observed on network. After compression, deduplication. I set up few backup jobs for testing and with Qudedupe is 50-70MB/s. I don't know what should be speed with older HBS? Was it 8MB/s on 1Gbps LAN?
TVS-951x TS-431P2 Unraid

User avatar
Moogle Stiltzkin
Ask me anything
Posts: 7562
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: Why backup with QuDedupe is 10x faster?

Post by Moogle Stiltzkin » Fri Sep 13, 2019 10:09 am

NAS
[Main Server] QNAP TS-877 w. 4tb [ 3x HGST Deskstar NAS (HDN724040ALE640) & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 + 16gb ddr4 Crucial + QWA-AC2600 wireless adapter.
[Backup] QNAP TS-653A w. 5x 2TB Samsung F3 (HD203WI) EXT4 Raid5
[^] QNAP TS-659 Pro II 1x 4TB HGST Deskstar NAS
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-228 w. 1x 1TB WD RE3 (WD1002FBYS)
[^] QNAP TS-128
Mobile NAS TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Asus AC68U Router|100dl/50ul MBPS FTTH Internet | Windows 10, WC PC-Intel i7 920 Ivy bridge desktop (1x 512gb Samsung 850 Pro SSD + 1x 4tb HGST Ultrastar 7K4000)


Guides & articles
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review

https://www.patreon.com/mooglestiltzkin

P3R
Guru
Posts: 10892
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Why backup with QuDedupe is 10x faster?

Post by P3R » Fri Sep 13, 2019 2:57 pm

janwer wrote:
Fri Sep 13, 2019 7:44 am
I don't know what should be speed with older HBS? Was it 8MB/s on 1Gbps LAN?
There is no speed that it "should" be. Speed will depend on NASes used, their configuration, job configuration and most importantly what the data being transferred consist of. Depending on those variables different users will most likely see different incomparable results.

If you want to know the effect of folder structure and file creation only, test with a single very large file and see how the two different backup methods compare.

If you want to see the effect of deduplication/compression only, test with a single very large strongly encrypted file. That should render deduplication/compression virtually useless so any differences you see then will be from other effects.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

User avatar
Toxic17
Ask me anything
Posts: 5107
Joined: Tue Jan 25, 2011 11:41 pm
Location: Planet Earth
Contact:

Re: Why backup with QuDedupe is 10x faster?

Post by Toxic17 » Fri Sep 13, 2019 5:11 pm

P3R wrote:
Fri Sep 13, 2019 2:57 pm
There is no speed that it "should" be. Speed will depend on NASes used, their configuration, job configuration and most importantly what the data being transferred consist of. Depending on those variables different users will most likely see different incomparable results.
I'm afraid some users think its just a straight file copy and look at their own copying experiences from PC to NAS as a benchmark.
Regards Simon

QTS 4.x User Guidex

QNAP Club Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following


NAS: TS-473-32GB QM2-2P QXG-10G1T 4.3.6.0993 • TVS-463-16GB 4.3.6.0993 QM2-2S10G1TB • TS-459 Pro 2GB 4.2.6 • TS-121 4.3.3.0998 • APC Back-UPS ES 700G •
QPKG's: TwonkyServer 8.51 • Apache73 • QSonarr 3.0.1.503 • QNBZGet 21.0 • phpMyAdmin 4.9.0.1 • Qmono 5.20.1.19 • McAfee 2.2.1 • Lychee 3.2.15 • HBS 3.0.190802 • LEgo v3.0.0
Network: VM Hub 3.0 <500/35> • UniFi USG Pro 4 • UniFi USW-16-150W • UniFi USW-8-60W • UniFi CloudKey Gen2+• UniFi G3-Flex • UAP AC Pro • UAP AC Lite • SLM2008 • Dell 7050 MFF •

janwer
Starting out
Posts: 29
Joined: Wed Dec 26, 2018 10:16 pm

Re: Why backup with QuDedupe is 10x faster?

Post by janwer » Fri Sep 13, 2019 9:34 pm

Qdedup is more CPU consuming so it should be faster.
I did test copying large movie file. No need to compress.
Manual recomends do not use Qudedup for movie files and
I did 2TB first backup with versioning and Qudedupe. It took about 10 hours.
Now I have to backup 2,4TB without versioning and without Qudedupe. Estimated time is 70 hours.

Why simpler and less CPU consuming way is so much slower?
Why 8MB/s on my 1Gbps LAN?
Source CPU's is 5-17%, destination CPU 34%.
What transfer should I expect? Is it normal or error?

If I copy this data between two QNAPs using PC/SMB full 1Gbps speed is no problem - 60MB/s.

Now I run simultaneously two backup jobs, both without Qudedupe. I have double speed! 16MB/s. Source CPU 6-36%, destination CPU 60-80%.

Edit.
I deleted backup job. Creating the same job but with Qudedupe. Transfer 50MB/s, estimated time 7-8 hours. CPUs 42% and 58%.
TVS-951x TS-431P2 Unraid

Post Reply

Return to “Backup & Restore”