Sync NAS to USB HDD, HBS always does full backup instead of only changes

[Firun]

So, I've been trying to create a backup / one way sync job that would only copy files that have been changed to the external HDD I've connected to my TS-253Be via USB. However, no matter what I do, it seems like it wipes the entire directory, then backups everything all over again. From my understanding, this is expected behavior for backup, but sync should, by default, be smart enough to only backup changes and delete files that are not in the source anymore - like rsync does.

Ideally, I'd like to run a backup to HDD every night, with only changes being transferred. I do not need versioning. However, no matter whether I use backup or one way sync, whether I check exclude duplicate files or not, it always copies everything all over again. What can I do to make this work? Additionally, what exactly does "exclude duplicate files" do? It's not quite clear to me from the description. Any help would be appreciated.

Edit: The HDD is formatted in ExFAT and I've purchased the according license. Also, I'd rather not use QuDedup since I'd prefer to have the regular file names for easy copying off the backup HDD.

Edit2: I have a Raspberry Pi in my LAN that runs Nagios and PiHole. If all else fails I'll hook the HDD up to my Pi, share it via Samba, and then rsync to it - but that's not what I bought a rather capable Nas for...

[spikemixture]

I just today setup an 8Bay USB3 Box,

It has 27TB of older/smaller drives currently setup as Raid0.

Now as a test i just copied a 30GB folder from my NAS to the USB box. (This folder does have some new/changed files, but not many)
3 times .
Doesn't look right to me !

hbs1Capture.JPG

[Firun]

Hmm but it does seem to not always write all files! For me, it always writes the entire folder all over again. Which settings did you use for that job? I'd be very grateful if you could detail the settings you used.

[spikemixture]

Mine looks like it is backing up the whole lot also.

I feel the settings are not the cause

I have 25TB i want to put on the USB so obviously don't want to copy that much more than once.

hbs2Capture.JPG

[Firun]

Thank you! Yes, I don't think it's the settings either... What FS is on your HDD? Mine is formatted in exFAT (with the driver from QNAP). Somehow, it seems to me as if the current HBS version is not quite working as intended...

In the meantime I've opened a support ticket with QNAP. I'll update the thread when I hear back from them.

[spikemixture]

I formatted mine in NTFS so the format appears to be irrelevant.
I trying my img again.

hbs2Capture.JPG

HBSV3 is beta so opening a ticket helps them undertand

Keep me updated and i will keep testing.

[Firun]

I've received a reply from QNAP: It first has to build the database before syncing works correctly. The first time, it will wipe the disk and transfer everything.

To test it I've just started the backup job again, and what do you know... it's only transferring changes. My best guess is that it takes a while after completion of the backup job to build said database. When I tested it, I transferred a small amount of files and immediately restarted the sync job... maybe that was the error.

Perhaps it's the same for you? Do the sync, and then wait for a good while before doing it again...

EDIT: In fact, it still transfers everything. It just does not wipe the disk beforehand. What the... I've updated the ticket and asked the guy for what's going on here. Will update when I get a reply.

[spikemixture]

Will try that.
Here are my speeds.

hbs3Capture.JPG

[Firun]

So, Qnap asked me to allow them to log in to my NAS remotely. Before activating the Helpdesk I've removed a lot of files (down to ~200GB from ~500GB). I then triggered multiple backups during the evening as well as the usual nightly backup. The result is attached.

Evidently, some caching / database building is going on, but it is either horribly slow or the algorithm is borked in some way since it still transferred the same 47.16GB each time, but skipped the rest of the ~200GB (11186 files, apparently). Judging from how slow the indexing and thumbnailing of files generally is I am inclined to think that the database is super slow. However, with the miniscule amount HBS exposes it's hard to tell what's actually going on. Seeing how long it took for it to build a rather small database I'd guess that the time needed to build one for your amount of data would be measured in months... which is unacceptable, especially seeing how your backup generally takes several days at least :/

Let's see what Qnap says after accessing my NAS. The rep said he'd hand it over to R&D for a detailed examination if he can reproduce the issue and can't isolate a cause. In the meantime I might SSH into my NAS to poke around in Linux and check the various logs. Maybe there is some kind of log that is more verbose and details what HBS is actually doing. I'll update here when I hear / find anything.

[Firun]

After much digging through the NAS via SSH I could isolate a handful of 7z files as the culprits. For some reason it always transfers those, yet leaves other 7z files in the same folder alone. The files are partially compressed using the LZMA algorithm, partially using the LZMA2 algorithm - perhaps this plays into it? Relayed that to support, let's see what they'll say.

[Firun]

Update: I've analysed all files in the folder that houses the *.7z files that cause problems. All the files that were compressed using the LZMA2 algorithm are transferred each time. 1/4 of the files compressed using the LZMA algorithm are transferred each time. Interestingly, an archive without compression is also transferred each time. I've sent the info to QNAP for them to make sense of it. So far, it definitely looks like *.7z and especially those compressed with LZMA2 are problematic.

EDIT: After taking both a problematic LZMA and LZMA2 file and recompressing it several times with different settings I found out that the problem is a combination of 7z files and external storage. When I back up NAS -> NAS the backup works and files are only transferred once. When I backup NAS -> USB it transfers all the test files every time. I've also tried reformatting and renaming my USB HDD, but it changes nothing. Let's see what QNAP finds out.

[Firun]

If anyone is still interested in this... QNAP support has fixed the issue. I know no specifics, but it was a bug. Topic can be closed

[spikemixture]

If anyone is still interested in this... QNAP support has fixed the issue. I know no specifics, but it was a bug. Topic can be closed

Fixed ?

So the next app update ?

[doolio]

Hi, Unfortunately, I continue to have this problem. I have a TS-251+ running the latest firmware v4.4 and bought an exFat license. I placed a brand new WD 8TB drive into a external USB3 enclosure and formatted it with exFat. I wanted to use exFat since I would like to be able to use the external drive with both Windows and MacOS. The external drive is recognized with no problems.

I then set up an AutoBackUp to synchronize all files from the NAS to the External drive. The first time it took 6 hours since it was copying everything for the first time. I then copied 2-3 new files to the NAS that were 5GB in total size. Then I re-launched the backup for a second time expecting that it would only take a few minutes. It it took 2.5 hours!. Previously I had an HGST 4TB drive in the external enclosure formatted with NTFS and it would finish a similar backup in 2-5 minutes. When I launched the second backup it looks like the new exFat drive is re-reading all files in order to decide what to synch.

I know that NTFS is a journaled file system which can very quickly know which files to synch. I Googled exFat and it turns out that exFAT is *not* a journaled file system. Could this be the reason why the exFAT synch takes so much longer than the NTFS synch?