Sync NAS to USB HDD, HBS always does full backup instead of only changes

Backup, Restore, Netbak Replicator, Cloud Storage Services
Post Reply
Firun
New here
Posts: 8
Joined: Wed Jul 31, 2019 1:29 am
Location: Switzerland

Sync NAS to USB HDD, HBS always does full backup instead of only changes

Post by Firun » Sun Aug 04, 2019 7:20 pm

So, I've been trying to create a backup / one way sync job that would only copy files that have been changed to the external HDD I've connected to my TS-253Be via USB. However, no matter what I do, it seems like it wipes the entire directory, then backups everything all over again. From my understanding, this is expected behavior for backup, but sync should, by default, be smart enough to only backup changes and delete files that are not in the source anymore - like rsync does.

Ideally, I'd like to run a backup to HDD every night, with only changes being transferred. I do not need versioning. However, no matter whether I use backup or one way sync, whether I check exclude duplicate files or not, it always copies everything all over again. What can I do to make this work? Additionally, what exactly does "exclude duplicate files" do? It's not quite clear to me from the description. Any help would be appreciated.

Edit: The HDD is formatted in ExFAT and I've purchased the according license. Also, I'd rather not use QuDedup since I'd prefer to have the regular file names for easy copying off the backup HDD.

Edit2: I have a Raspberry Pi in my LAN that runs Nagios and PiHole. If all else fails I'll hook the HDD up to my Pi, share it via Samba, and then rsync to it - but that's not what I bought a rather capable Nas for...
QNAP TS-253Be, 2x 4TB WD Red, RAID1, 16GB Ram (-> HX318LS11IBK2/16)

User avatar
spikemixture
Been there, done that
Posts: 677
Joined: Wed Mar 07, 2018 11:04 pm
Location: 3rd World

Re: Sync NAS to USB HDD, HBS always does full backup instead of only changes

Post by spikemixture » Mon Aug 05, 2019 4:18 pm

I just today setup an 8Bay USB3 Box,

It has 27TB of older/smaller drives currently setup as Raid0.

Now as a test i just copied a 30GB folder from my NAS to the USB box. (This folder does have some new/changed files, but not many)
3 times .
Doesn't look right to me !
hbs1Capture.JPG
You do not have the required permissions to view the files attached to this post.
Qnap TS-1277 1700 - 6x8TB WD White, 2x8TB WD Red - Raid 5
Qnap TVS-1282 - 8x10TB WD White - Raid 6
Synology -1817+ - 2x10,2x8,2x6 (Ironwolf) 2x6 WD Red - SHR
Drobo 5n - 5x2TB WD Green - Drobo Raid
Buffalo LinkStation - 2x3TB WD
ProBox 8 Bay USB3 - 8x4TB Seagate - JBOD
All software is updated as soon as it's available.
I give my opinion from my experience i.e. I have (or had) that piece of equipment/software and used it! :roll:

Firun
New here
Posts: 8
Joined: Wed Jul 31, 2019 1:29 am
Location: Switzerland

Re: Sync NAS to USB HDD, HBS always does full backup instead of only changes

Post by Firun » Mon Aug 05, 2019 4:23 pm

Hmm but it does seem to not always write all files! For me, it always writes the entire folder all over again. Which settings did you use for that job? I'd be very grateful if you could detail the settings you used.
QNAP TS-253Be, 2x 4TB WD Red, RAID1, 16GB Ram (-> HX318LS11IBK2/16)

User avatar
spikemixture
Been there, done that
Posts: 677
Joined: Wed Mar 07, 2018 11:04 pm
Location: 3rd World

Re: Sync NAS to USB HDD, HBS always does full backup instead of only changes

Post by spikemixture » Mon Aug 05, 2019 4:44 pm

Mine looks like it is backing up the whole lot also.

I feel the settings are not the cause

I have 25TB i want to put on the USB so obviously don't want to copy that much more than once.
hbs2Capture.JPG
You do not have the required permissions to view the files attached to this post.
Qnap TS-1277 1700 - 6x8TB WD White, 2x8TB WD Red - Raid 5
Qnap TVS-1282 - 8x10TB WD White - Raid 6
Synology -1817+ - 2x10,2x8,2x6 (Ironwolf) 2x6 WD Red - SHR
Drobo 5n - 5x2TB WD Green - Drobo Raid
Buffalo LinkStation - 2x3TB WD
ProBox 8 Bay USB3 - 8x4TB Seagate - JBOD
All software is updated as soon as it's available.
I give my opinion from my experience i.e. I have (or had) that piece of equipment/software and used it! :roll:

Firun
New here
Posts: 8
Joined: Wed Jul 31, 2019 1:29 am
Location: Switzerland

Re: Sync NAS to USB HDD, HBS always does full backup instead of only changes

Post by Firun » Mon Aug 05, 2019 5:01 pm

Thank you! Yes, I don't think it's the settings either... What FS is on your HDD? Mine is formatted in exFAT (with the driver from QNAP). Somehow, it seems to me as if the current HBS version is not quite working as intended...

In the meantime I've opened a support ticket with QNAP. I'll update the thread when I hear back from them.
QNAP TS-253Be, 2x 4TB WD Red, RAID1, 16GB Ram (-> HX318LS11IBK2/16)

User avatar
spikemixture
Been there, done that
Posts: 677
Joined: Wed Mar 07, 2018 11:04 pm
Location: 3rd World

Re: Sync NAS to USB HDD, HBS always does full backup instead of only changes

Post by spikemixture » Mon Aug 05, 2019 6:13 pm

I formatted mine in NTFS so the format appears to be irrelevant.
I trying my img again.
hbs2Capture.JPG
HBSV3 is beta so opening a ticket helps them undertand

Keep me updated and i will keep testing.
You do not have the required permissions to view the files attached to this post.
Qnap TS-1277 1700 - 6x8TB WD White, 2x8TB WD Red - Raid 5
Qnap TVS-1282 - 8x10TB WD White - Raid 6
Synology -1817+ - 2x10,2x8,2x6 (Ironwolf) 2x6 WD Red - SHR
Drobo 5n - 5x2TB WD Green - Drobo Raid
Buffalo LinkStation - 2x3TB WD
ProBox 8 Bay USB3 - 8x4TB Seagate - JBOD
All software is updated as soon as it's available.
I give my opinion from my experience i.e. I have (or had) that piece of equipment/software and used it! :roll:

Firun
New here
Posts: 8
Joined: Wed Jul 31, 2019 1:29 am
Location: Switzerland

Re: Sync NAS to USB HDD, HBS always does full backup instead of only changes

Post by Firun » Mon Aug 05, 2019 8:34 pm

I've received a reply from QNAP: It first has to build the database before syncing works correctly. The first time, it will wipe the disk and transfer everything.

To test it I've just started the backup job again, and what do you know... it's only transferring changes. My best guess is that it takes a while after completion of the backup job to build said database. When I tested it, I transferred a small amount of files and immediately restarted the sync job... maybe that was the error.

Perhaps it's the same for you? Do the sync, and then wait for a good while before doing it again...

EDIT: In fact, it still transfers everything. It just does not wipe the disk beforehand. What the... I've updated the ticket and asked the guy for what's going on here. Will update when I get a reply.
QNAP TS-253Be, 2x 4TB WD Red, RAID1, 16GB Ram (-> HX318LS11IBK2/16)

User avatar
spikemixture
Been there, done that
Posts: 677
Joined: Wed Mar 07, 2018 11:04 pm
Location: 3rd World

Re: Sync NAS to USB HDD, HBS always does full backup instead of only changes

Post by spikemixture » Tue Aug 06, 2019 10:25 am

Will try that.
Here are my speeds.
hbs3Capture.JPG
You do not have the required permissions to view the files attached to this post.
Qnap TS-1277 1700 - 6x8TB WD White, 2x8TB WD Red - Raid 5
Qnap TVS-1282 - 8x10TB WD White - Raid 6
Synology -1817+ - 2x10,2x8,2x6 (Ironwolf) 2x6 WD Red - SHR
Drobo 5n - 5x2TB WD Green - Drobo Raid
Buffalo LinkStation - 2x3TB WD
ProBox 8 Bay USB3 - 8x4TB Seagate - JBOD
All software is updated as soon as it's available.
I give my opinion from my experience i.e. I have (or had) that piece of equipment/software and used it! :roll:

Firun
New here
Posts: 8
Joined: Wed Jul 31, 2019 1:29 am
Location: Switzerland

Re: Sync NAS to USB HDD, HBS always does full backup instead of only changes

Post by Firun » Tue Aug 06, 2019 3:54 pm

So, Qnap asked me to allow them to log in to my NAS remotely. Before activating the Helpdesk I've removed a lot of files (down to ~200GB from ~500GB). I then triggered multiple backups during the evening as well as the usual nightly backup. The result is attached.

Evidently, some caching / database building is going on, but it is either horribly slow or the algorithm is borked in some way since it still transferred the same 47.16GB each time, but skipped the rest of the ~200GB (11186 files, apparently). Judging from how slow the indexing and thumbnailing of files generally is I am inclined to think that the database is super slow. However, with the miniscule amount HBS exposes it's hard to tell what's actually going on. Seeing how long it took for it to build a rather small database I'd guess that the time needed to build one for your amount of data would be measured in months... which is unacceptable, especially seeing how your backup generally takes several days at least :/

Let's see what Qnap says after accessing my NAS. The rep said he'd hand it over to R&D for a detailed examination if he can reproduce the issue and can't isolate a cause. In the meantime I might SSH into my NAS to poke around in Linux and check the various logs. Maybe there is some kind of log that is more verbose and details what HBS is actually doing. I'll update here when I hear / find anything.
You do not have the required permissions to view the files attached to this post.
QNAP TS-253Be, 2x 4TB WD Red, RAID1, 16GB Ram (-> HX318LS11IBK2/16)

Firun
New here
Posts: 8
Joined: Wed Jul 31, 2019 1:29 am
Location: Switzerland

Re: Sync NAS to USB HDD, HBS always does full backup instead of only changes

Post by Firun » Tue Aug 06, 2019 7:49 pm

After much digging through the NAS via SSH I could isolate a handful of 7z files as the culprits. For some reason it always transfers those, yet leaves other 7z files in the same folder alone. The files are partially compressed using the LZMA algorithm, partially using the LZMA2 algorithm - perhaps this plays into it? Relayed that to support, let's see what they'll say.
QNAP TS-253Be, 2x 4TB WD Red, RAID1, 16GB Ram (-> HX318LS11IBK2/16)

Firun
New here
Posts: 8
Joined: Wed Jul 31, 2019 1:29 am
Location: Switzerland

Re: Sync NAS to USB HDD, HBS always does full backup instead of only changes

Post by Firun » Wed Aug 07, 2019 7:13 pm

Update: I've analysed all files in the folder that houses the *.7z files that cause problems. All the files that were compressed using the LZMA2 algorithm are transferred each time. 1/4 of the files compressed using the LZMA algorithm are transferred each time. Interestingly, an archive without compression is also transferred each time. I've sent the info to QNAP for them to make sense of it. So far, it definitely looks like *.7z and especially those compressed with LZMA2 are problematic.

EDIT: After taking both a problematic LZMA and LZMA2 file and recompressing it several times with different settings I found out that the problem is a combination of 7z files and external storage. When I back up NAS -> NAS the backup works and files are only transferred once. When I backup NAS -> USB it transfers all the test files every time. I've also tried reformatting and renaming my USB HDD, but it changes nothing. Let's see what QNAP finds out.
QNAP TS-253Be, 2x 4TB WD Red, RAID1, 16GB Ram (-> HX318LS11IBK2/16)

Firun
New here
Posts: 8
Joined: Wed Jul 31, 2019 1:29 am
Location: Switzerland

Re: Sync NAS to USB HDD, HBS always does full backup instead of only changes

Post by Firun » Tue Sep 10, 2019 5:14 pm

If anyone is still interested in this... QNAP support has fixed the issue. I know no specifics, but it was a bug. Topic can be closed :)
QNAP TS-253Be, 2x 4TB WD Red, RAID1, 16GB Ram (-> HX318LS11IBK2/16)

User avatar
spikemixture
Been there, done that
Posts: 677
Joined: Wed Mar 07, 2018 11:04 pm
Location: 3rd World

Re: Sync NAS to USB HDD, HBS always does full backup instead of only changes

Post by spikemixture » Wed Sep 11, 2019 11:19 pm

Firun wrote:
Tue Sep 10, 2019 5:14 pm
If anyone is still interested in this... QNAP support has fixed the issue. I know no specifics, but it was a bug. Topic can be closed :)
Fixed ?

So the next app update ?
Qnap TS-1277 1700 - 6x8TB WD White, 2x8TB WD Red - Raid 5
Qnap TVS-1282 - 8x10TB WD White - Raid 6
Synology -1817+ - 2x10,2x8,2x6 (Ironwolf) 2x6 WD Red - SHR
Drobo 5n - 5x2TB WD Green - Drobo Raid
Buffalo LinkStation - 2x3TB WD
ProBox 8 Bay USB3 - 8x4TB Seagate - JBOD
All software is updated as soon as it's available.
I give my opinion from my experience i.e. I have (or had) that piece of equipment/software and used it! :roll:

Post Reply

Return to “Backup & Restore”