Hybrid Backup RTRR impossibly slow
-
- Starting out
- Posts: 30
- Joined: Sun Dec 26, 2010 8:35 am
Hybrid Backup RTRR impossibly slow
I have recently started using HBS to backup files from a QNAP TS-410 to a QNAP TS-119. Having read the docco, HBS sounds great and just what I was looking for to schedule and store different versions of backups. I have approx 2TB of data to backup.
However, in practice the RTRR backups are so painfully slow that it's almost unusable. As an example, the initial backup of one person's folders from TS-410 to TS-119 takes 7.5 hrs to backup only 13GB of data. The average backup speed is approx 900KB/s. But when I do an RSYNC backup the speeds are much higher at approx 5.7MB/s. Both NAS devices have gigabit ethernet connections. These are the settings for the rync job:
So I could use the rsync protocol, but it's a sync, not a backup that retains versions histories. I really want to use rtrr but at these speeds it's not possible. Why is rtrr so slow compared to rsync, and what can I do to speed it up and make it usable?
However, in practice the RTRR backups are so painfully slow that it's almost unusable. As an example, the initial backup of one person's folders from TS-410 to TS-119 takes 7.5 hrs to backup only 13GB of data. The average backup speed is approx 900KB/s. But when I do an RSYNC backup the speeds are much higher at approx 5.7MB/s. Both NAS devices have gigabit ethernet connections. These are the settings for the rync job:
So I could use the rsync protocol, but it's a sync, not a backup that retains versions histories. I really want to use rtrr but at these speeds it's not possible. Why is rtrr so slow compared to rsync, and what can I do to speed it up and make it usable?
-
- Getting the hang of things
- Posts: 56
- Joined: Wed Apr 16, 2008 1:12 am
Re: Hybrid Backup RTRR impossibly slow
On the 2nd ov Nov. we made a ticket for this problem. Because is was very fast until, the "must have" update to HBS3. In the old backub station rtrr was extremly fast.
But Qnap never realy helped us until today. Only hints like "firmware update" "reset and restore" but it doesn't help. Now it took them almost 20 days to respond again in our ticket.
The have send it to the development in Taiwan. Lets hope they find out something.
I recommend you, open a ticket, if more people send in this problem it gets hopefully solved.
https://service.qnap.com/
But Qnap never realy helped us until today. Only hints like "firmware update" "reset and restore" but it doesn't help. Now it took them almost 20 days to respond again in our ticket.
The have send it to the development in Taiwan. Lets hope they find out something.
I recommend you, open a ticket, if more people send in this problem it gets hopefully solved.
https://service.qnap.com/
-
- Starting out
- Posts: 30
- Joined: Sun Dec 26, 2010 8:35 am
Re: Hybrid Backup RTRR impossibly slow
Wow, that's really poor. I've also seen other similar posts that didn't get resolved. And over 200 people have read this with no solution offered so I guess everyone has the same problem that QNAP have implemented a backup solutio that's unusable. They ought to be onto that quick smart as backups are crucial to any storage solution.hallohome wrote: ↑Tue Dec 29, 2020 4:15 pm On the 2nd ov Nov. we made a ticket for this problem. Because is was very fast until, the "must have" update to HBS3. In the old backub station rtrr was extremly fast.
But Qnap never realy helped us until today. Only hints like "firmware update" "reset and restore" but it doesn't help. Now it took them almost 20 days to respond again in our ticket.
The have send it to the development in Taiwan. Lets hope they find out something.
I recommend you, open a ticket, if more people send in this problem it gets hopefully solved.
https://service.qnap.com/
I'll raise a ticket and see what they say. Probably nothing by the looks of things!
Meantime I've stopped using their unusable HFS app for backups and am using Rsnapshot instead. This achieves the same end result of incremental backups, but uses rsync so the performance is respectable.
-
- New here
- Posts: 3
- Joined: Tue Mar 01, 2011 12:34 am
Re: Hybrid Backup RTRR impossibly slow
I have had similar problems backing up from a TS-541+ to a TS410 using HBS 3.
I had 750GB in 1.5M files. The first 50% went reasonably quickly. Then it slowed to a crawl. Stopping the job and restarting would get a 100,000 done fairly fast, before slowing to a crawl. For the last 100,000 had to restart the job every few thousand in order to get a reasonable speed.
At a guess, I would say a log on the server is getting full, and somehow it is not purging old entries fast enough to allow the backup to continue at a sensible speed.
One thing I noted was the files it seemed to hang up on were very small byte sized files. The large ones didn't seem to be a problem.
Hopefully QNAP technical support/development can reproduce this and develop a fix.
I had 750GB in 1.5M files. The first 50% went reasonably quickly. Then it slowed to a crawl. Stopping the job and restarting would get a 100,000 done fairly fast, before slowing to a crawl. For the last 100,000 had to restart the job every few thousand in order to get a reasonable speed.
At a guess, I would say a log on the server is getting full, and somehow it is not purging old entries fast enough to allow the backup to continue at a sensible speed.
One thing I noted was the files it seemed to hang up on were very small byte sized files. The large ones didn't seem to be a problem.
Hopefully QNAP technical support/development can reproduce this and develop a fix.
- dolbyman
- Guru
- Posts: 35215
- Joined: Sat Feb 12, 2011 2:11 am
- Location: Vancouver BC , Canada
Re: Hybrid Backup RTRR impossibly slow
if it's the destination RRTR, then don't bank on QNAP, no more feature or security updates for x19 devices starting today
-
- New here
- Posts: 2
- Joined: Mon Jul 15, 2019 10:24 pm
Re: Hybrid Backup RTRR impossibly slow
Same issue here. Backup from a TVS-471 to a TVS-472XT. Initial backup speed was at least 50 Mb/sec, processing 1.5Tb in a couple of hours.
But then it slowed down to a crawl (< 4 Kb/s). The last 10% of the files will take a full day this way.
I am 100% sure it is not the CPU load (<10%), memory usage (<30%) on either NAS and it is definitely
not the network congestion (using Gbit ethernet, dedicated).
Also, when I start another backup job (in parallel), the total speed is back to >50 Mb/sec instantly.
So it must be a bug in HBS. Hopefully the QNAP guys will fix it soon.
But then it slowed down to a crawl (< 4 Kb/s). The last 10% of the files will take a full day this way.
I am 100% sure it is not the CPU load (<10%), memory usage (<30%) on either NAS and it is definitely
not the network congestion (using Gbit ethernet, dedicated).
Also, when I start another backup job (in parallel), the total speed is back to >50 Mb/sec instantly.
So it must be a bug in HBS. Hopefully the QNAP guys will fix it soon.
-
- New here
- Posts: 2
- Joined: Mon Jul 15, 2019 10:24 pm
Re: Hybrid Backup RTRR impossibly slow
Did a quick test. Split the same backup job in 3 parts, ran then one by one and they were all processed at a speed of approx. 110MB/s.
So it looks like the issue may have something to do with the large number of (small) files.
PS: in my previous post I mentioned Kb/s, Mb/s, Tb. That should have been KB/s, MB/s, TB. Oops.
So it looks like the issue may have something to do with the large number of (small) files.
PS: in my previous post I mentioned Kb/s, Mb/s, Tb. That should have been KB/s, MB/s, TB. Oops.
-
- Know my way around
- Posts: 129
- Joined: Mon Oct 22, 2018 2:56 pm
Re: Hybrid Backup RTRR impossibly slow
I have also found that a large number of small files creates problems. Unfortunately "large" is not all that large. I find that if you I the backup and run it again, I can get a bit further and eventually get it done. Something is definitely amiss.
-
- Easy as a breeze
- Posts: 298
- Joined: Fri Apr 25, 2014 10:29 am
Re: Hybrid Backup RTRR impossibly slow
Not sure if this works for newer models, but if HBS is removed the built-in Backup Station returns. I have no need for cloud based backups or other features of HBS on my old, but still running TS-212P.
TS-212P QTS: 4.3.3.1864 build 20211212
2 x Seagate ST3000VN000 (RAID 1)
2 x Seagate ST3000VN000 (RAID 1)
-
- Starting out
- Posts: 25
- Joined: Fri Mar 03, 2017 11:24 pm
Re: Hybrid Backup RTRR impossibly slow
I had this exact same problem - I think a suggestion either on this thread or similar on this board fixed this for me
I increased the size of the log file to the maximum 1024 mb
instead of the estimated 12 days, took less than 8 hours to complete a full backup of approx 4.5 TB to an attached USB drive
not a "guru" in this area, but I wonder if a backup with many small files causes log file bloat
hopefully, QNAP will address this
I increased the size of the log file to the maximum 1024 mb
instead of the estimated 12 days, took less than 8 hours to complete a full backup of approx 4.5 TB to an attached USB drive
not a "guru" in this area, but I wonder if a backup with many small files causes log file bloat
hopefully, QNAP will address this
-
- Easy as a breeze
- Posts: 495
- Joined: Tue Aug 25, 2015 7:06 pm
Re: Hybrid Backup RTRR impossibly slow
Progress
I increased the log file to 1024 MB.
Now, the BU proceeds rapidly to 5% and then slows down to a crawl. The initial time to finish is 22 hrs (which is reasonable), and then grows steadily to many days.
At least HBS was able to calculate the job size
I'll try splitting the job into 2-3 parts..
I increased the log file to 1024 MB.
Now, the BU proceeds rapidly to 5% and then slows down to a crawl. The initial time to finish is 22 hrs (which is reasonable), and then grows steadily to many days.
At least HBS was able to calculate the job size
I'll try splitting the job into 2-3 parts..
You do not have the required permissions to view the files attached to this post.
NAS: TS-453Be
RAM:Crucial 8GB Kit (2 x 4GB) DDR3L-1600 SODIMM CT2KIT51264BF160B
QTS: 5.1.4
HDD's: RAID 6: Four 8TB WD Red (WD80EFAX)
USB HDD: One 12 TB WD Elements (WDBWLG0120HBK-NESN)
Switch: Netgear GS108
RAM:Crucial 8GB Kit (2 x 4GB) DDR3L-1600 SODIMM CT2KIT51264BF160B
QTS: 5.1.4
HDD's: RAID 6: Four 8TB WD Red (WD80EFAX)
USB HDD: One 12 TB WD Elements (WDBWLG0120HBK-NESN)
Switch: Netgear GS108
-
- First post
- Posts: 1
- Joined: Sat Feb 13, 2021 12:01 am
Re: Hybrid Backup RTRR impossibly slow
I am having the same issue. I had a 4TB USB drive that seemed to backup incremental files quickly. When I replaced the smaller external drive with a new larger one I started having this problem. I am backing up 3TB. At about 31% is slows to a crawl. Looking at the details it's backing up small image files when it starts to choke.
Is QNAP looking into this?
Is QNAP looking into this?
-
- New here
- Posts: 7
- Joined: Sun Jun 28, 2015 11:21 am
Re: Hybrid Backup RTRR impossibly slow
Hmm - I tried this and Backup Station did not show up, nor do I see it in the app center to install. I did a reboot. Any other steps you did to get it?
-
- Easy as a breeze
- Posts: 298
- Joined: Fri Apr 25, 2014 10:29 am
Re: Hybrid Backup RTRR impossibly slow
It probably depends on the firmware for your model. I remember trying HBS when it first came out some 4 or so years ago and was buggy then. All I did was uninstall it and magically Backup Station was there.
TS-212P QTS: 4.3.3.1864 build 20211212
2 x Seagate ST3000VN000 (RAID 1)
2 x Seagate ST3000VN000 (RAID 1)
-
- Been there, done that
- Posts: 530
- Joined: Wed Apr 28, 2010 9:22 pm
Re: Hybrid Backup RTRR impossibly slow
As of firmware 4.5.1 good old Backup Station is removed, as mentioned in the release notes. Unfortunately.
1x TS251, 1x TS251D, 1x TS253