Page 1 of 1

[review] HybridBackupSync 3 beta

Posted: Sat May 25, 2019 5:39 am
by Moogle Stiltzkin
You can join the contest here to participate

QNAP TBS 453DX NAS Book - Hardware Review *on your marks get set....GO... :D

For qudedupe i'll probably use some data test samples rather than my official stuff in a test share :) still have to figure out if this is something i'd do for backing up. but at least for filesharing i probly use it.

some of the new features of HBS 3

TCP BBR : Magic dust for network performance.

It’s lost on most people that today’s internet is running on algorithms and protocols that were designed back in the 80s. You know, back when things like “message boards” where the most exciting thing going on.

But that’s not today’s internet. Today’s internet is distributed across 7 billion people on devices and network connectivity ranging from awesome great 1GB / sec wired connections to wireless ones that… well.. Aren’t.

And that brings up the question that perhaps today’s internet needs require and upgrade to those algorithms we made back 30 years ago.

This is where TCP BBR comes in. It’s a TCP congestion control algorithm built for the congestion of the modern internet. ... a5f1ccf437

QuDedupe bandwidth reduction & encryption?

faster versioning?

cross device access

also new UI



the review will cover my experience testing with this, thoughts on the features and UI, whether it helped or what needs to be changed/tweaked. This is something i actually use for my NAS backup 8)

Re: [review] HybridBackupSync 3 beta

Posted: Sat May 25, 2019 6:36 am
by Moogle Stiltzkin
Just a small preview of what i found out so far.

QuDedupe requires that BOTH the sauce and destination QNAP NAS have it installed (so if you got an older HBS on your NAS e.g. an EOL model, well thats not gonna fly i'm afraid)

So i'm gonna have to run 2 tests

- HBS 3 beta on both a TS-877 (sauce) to TS-653a (destination)

- HBS3 beta on Ts-877 to TS-509 Pro (this is eol and using a much older hbs :( )

Re: [review] HybridBackupSync 3 beta

Posted: Sat May 25, 2019 6:56 am
by Moogle Stiltzkin
okay i discovered at least 1-2 weird bugs. but i managed to get around it by trying somethings. so if you are wondering, yes there are definitely some bugs *hence it's beta

not sure i recommend this for production just yet :)

i created a share for testing purposes for now.

Re: [review] HybridBackupSync 3 beta

Posted: Sat May 25, 2019 7:14 am
by Moogle Stiltzkin
one nice option is the client side encryption. basically you encrypt your data PRIOR to backing it up remotely (in my case onto the backup NAS on the same lan). So my guess is, the stored backup will be encrypted (because in the settings for client side encryption, it requires a password to decrypt). So if you want a secure backup, this is probably ideal.

This is probably something i'd use for backing up my encrypted shares, so that the backed up data is encrypted. Encrypting the share on the backup NAS is one possible solution, but then you would need to remember to unlock before running your HBS backup, and then to lock it back again manually. It seems like just using the client side encryption for the backup is the simpler solution to go with.



the remove deleted data from destination option is useful for me. because i don't want to retain data on the backup that is no longer on the main NAS (though i have to be mindful of not accidentally deleting something i actually want, and end up removing it on the backup too). There is also an option to retain the data before deleting after x days which is useful as a precaution for accidents (for what i just mentioned).

Re: [review] HybridBackupSync 3 beta

Posted: Sat May 25, 2019 7:34 am
by Moogle Stiltzkin
This is my first test with the following settings configured

- create a backup job to remote (a ts-877 to a ts-653a on the local lan)
- enable QuDedupe
- enable client side encryption (i'll probably only use this for data stored on my encrypted shares, so i'd have a separate job just for that for my own production backup. If i was doing this over the internet, i'd probably enable this for all, but for local lan i'm not sure it's needed unless it's for sensitive data)
- delete data from destination (did they rename the delete extra files ? well it's explained what it does, and seems to do the same thing)

I wanted to use the backup only new files, but that seemed to be greyed out, no idea why. it wasn't explained :'



you can run your backup either from the 1st or 2nd screenshot locations. but usually i prefer doing mine from the 2nd screenshot. also as you can see the recover jobs i have disabled. for safety sake, i recommend disabling them when not actively using. It would be a disaster if i overwrote my main NAS from backup by hitting a wrong button :shock: i only unlock that when i actually am going to recover for realz only.

Here is the first successful backup job. The sample size of the backup was too small, so i could not really test out the effectiveness of the TCP BBR just yet (also how much of a performance impact would using client side encryption cause. i'll run a complete test later for that)

as you can see, QuDedupe created a bunch of weird stuff in it. Well as long as the file size is same or smaller than not using it, and i can successfully recover my files, then i don't particularly mind. But it's clear that you won't be able to just simply open up your files on your backup just like that. You'd have to recover them back to your main NAS for them to work again (which i will test in a bit by the way by deleting the sauce data then using the recover and see how swellingly that works out)


Re: [review] HybridBackupSync 3 beta

Posted: Sat May 25, 2019 8:40 am
by Moogle Stiltzkin
ok now testing recovery.

delete the sauce data using filestation (sometimes when i do it from win explorer it MAY not actually be deleted. so to be certain, i use filestation)

next is to recover in HBS.

okay it worked kinda (got confused by some of the settings that initially would not let me procede). i found some issue which i will report :S

all files restored, crc checks out. tested files as working

next test i will run is a less restrictive setup that i would probably use for most of my backup. i'll try without client side encryption. I'll however enable SSL encryption. Will test with qudedupe enabled still. Will run a separate test later with it disabled.

Re: [review] HybridBackupSync 3 beta

Posted: Sat May 25, 2019 9:15 am
by Moogle Stiltzkin
new test

one way sync rtrr with ssl encryption enabled.

In 1 way sync, there is no client side encryption and qudedupe option. other options get unlocked instead for this mode.


anyway it worked

this is the setting i most often use and am familiar with, so i'll do a bigger backup test to get a feel for transfer speed improvements.

i'll run separate tests for hbs BACKUP to check out qudedupe more though.

from what i've noticed so far, the only time TCP BBR got mentioned was in an option during the recovery test (for the HBS backup). i did not see it mentioned anywhere else :'

i just assumed that if u run HBS 3 beta and RTRR that TCP BBR will be working. Not sure if running the new HBS on the destination is also a requirement for it (only qudedupe so far mentioned it as a requirement)

Re: [review] HybridBackupSync 3 beta

Posted: Sat May 25, 2019 9:27 am
by Moogle Stiltzkin
Creating a HBS BACKUP test 2.

Won't use client side encryption. Will only test with qudedupe enabled and the other settings from before. I'll leave SSL enabled though.

I'll also use a bigger sample size of source data to backup (roughly 300 gb). Will take a while to copy over to the new test hbs 2 share i made for the purpose.

will be testing selectively deleting files and seeing if that correctly replicates over to the backup and just testing how it works.

also testing if qudedupe windows app for decrypting actually works or not (at least previously it did decrypt fine via HBS when running the restore)

Re: [review] HybridBackupSync 3 beta

Posted: Sat May 25, 2019 10:57 am
by Moogle Stiltzkin
After doing a bit more research, wondering if the new HBS needs to be on both the source and destination qnap for TCP BBR to work, i don't think that it does
Doesn’t require the client to implement BBR. This one is the magic pixie dust. Prior algorithms like QUIC required client & server to have both implemented the algorithm. BBR doesn’t require the client to be using BBR as well. This is especially relevant in the developing world which use older mobile platforms and have limited bandwidth, or areas where websites & services haven’t made the switch yet. ... a5f1ccf437

Re: [review] HybridBackupSync 3 beta

Posted: Sat May 25, 2019 3:06 pm
by Moogle Stiltzkin
performance test


first did an extra step i ran a test to ensure there is no issues with ethernet cables on switch. so performance issues can probably look elsewhere :'

both qnaps ts-877 and ts-653a updated qts and using the hbs 3 beta, latest switch and router firmwares, all devices rebooted. test prep ready.


my used configuration

RTRR, one way sync, no filters, remove additional files at destination, check file contents, exclude system generated temp files, do not take a snapshot, SSL disabled.

I'll try later with ssl, but i'm not expecting there to be issue with it enabled. It's just simply i am running over local lan, so not sure that i require it for my case scenario :)

side note: had SSD cache acceleration enabled, random I/O , bypass 1MB block. I was reading another thread where it implied ssd cache acceleration was slowing down performance for transfer, but i'm not really having this specific issue, so i did not bother to disable (although i did disable then re-enable so the capacity utility is on the low end. in thread they mention issues when it is almost full)

then delete data on main NAS, and perform a restore from backup NAS to main NAS. speed is more or less the same

so the rtrr 1 way sync + restore seems to work fine.

only the backup mode and with qudedupe requires more testing :'


ran a complete backup (5tb ish), same settings, everything seems to have worked as intended.




difference in folder and files is probably thumbnails. the file size storage at least is the same.

Re: [review] HybridBackupSync 3 beta

Posted: Sun May 26, 2019 5:49 am
by Moogle Stiltzkin
did some digging, yes they do explain the difference between a backup and sync. although maybe they should make it into the intro slider a quick summary about it perhaps? with links that go more indepth into an explanation comparing the 2. i'll bet you a bunch of newbies will be wondering which they should be using to perform a backup. ... p-station/

About Synchronization (One-Way,Two-Way), Backup and Replication
File Synchronization (a.k.a. File Sync) is the process of ensuring that two or more locations contain the same, up-to-date, files. If you add, change, or delete a file from one location, the synchronization process will add, change, or delete the corresponding file at the other location.

Synchronization can be “two-way” or “one-way”.

Two-Way Synchronization (a.k.a. bi-directional synchronization or both-ways synchronization):

This synchronization process copies files in both directions to reconcile changes as needed. Files are expected to change in both locations. The two locations are considered equivalent.

Example: if a file is newer in Location A, it will be copied to Location B. But if a file is newer in Location B it will be copied to Location A. Similarly, if a file is deleted from Location A, it will be deleted from Location B as well and vice versa.

One-Way Synchronization (a.k.a. file mirroring / file replication / file backup):

Files are expected to change in one location only. To reconcile the changes, the synchronization process copies files only in one direction. The two locations are not considered equivalent. One location is considered the Source and the other is considered the Target. Files are pushed from Source to Target (or files are pulled from Source to Target, but always in one direction only). Source is said to be mirrored to Target.

This creates an exact 1:1 replica of all files in Source to Target. This is very useful and effective for backup purposes since only changed/new files are copied.

Example: If a file is newer in Location A (Source), it will be copied to Location B (Target). But if a file is newer in Location B, it will NOT be copied to Location A. Similarly, if a file is deleted from Location A, it will be deleted from Location B. But a file deleted in Location B is restored using the file in Location A.

this is why i tend to use rtrr one way sync. and have in the past used it successfully as a backup.

and in the current hbs 3 beta test, the performance i'm getting is 80 MB/s + despite having things like checksum verification, delete extra files etc enabled. So thats all fine and dandy for me.

Sync vs. Backup Explained

What’s the real difference between sync and backup? What are the pros and cons of them? We often get confused when hearing terms like that, and conclude in our minds that they are the same. Since our lives have basically shifted towards the digital realm where we are constantly focused on the phone, the tablet, computers and even now smart watches ; yet some of us don’t even know if the data is backed up or synced somewhere.

Sync, means synchronization. Therefore a ‘syncing’ action happens with your photos on your phone in the background… And hopefully you’ve registered yourself to sync your photos to iCloud (if you’re using an Apple iPhone) or for the recent Android devices they should be syncing somewhere onto the cloud. One should also note that sync is not a backup. Syncing lets you copy or delete files as needed to ensure that two locations have the same exact file or version of the files. Examples of syncing is Dropbox, Google Drive as the files stored on them are the same exact versions of each other whether you view it on your laptop, phone, or computer etc. So make sure, to also backup apart from syncing your files.

On a QNAP NAS… You can always backup AND sync your files at the same time.
but my one way sync is an exact replica and works for recovery. so why wouldn't it be considered a backup? :'

Re: [review] HybridBackupSync 3 beta

Posted: Sun May 26, 2019 5:53 pm
by Moogle Stiltzkin
testing HBS Backup with qudedupe disabled.

but using client side encryption, remove additional files, check file contents, exclude system generated temp files, do not take snapshots, ssl disabled.

once it completes i'll test if it encrypts client side or not. supposedly it requires a password to unlock. checking how that works.


ok weird error. the backup gets disrupted, then it resumes. i also notice the destination NAS had triggered the alarm for reach threshold 80% (related?). Anyway it's resumed so keeping an eye on it.


this broken pipe error happened 2 more times. each time the backup process resets back to 0% so you start over again from the start :shock: very doubtful this will finish completing. nor would i use this setting for my own backup until it actually works :(

for now i deleted anything at destination (but i don't see why i should. by rights the settings should delete anything not matching source to begin with), re-created the job same settings, and run again. see if it works this time.

nope it didn't.

fyi, client side encryption was added sometime since last year. it's not something i used at the time. so i'm not sure if the issue is new and only occuring with this latest hbs beta or what. i'm merely reporting my results as is :(
Hybrid Backup Sync 2.1.180816
( 2018/08/22 )

- Added support for enabling client-side encryption with RTRR backup jobs.
- Added support for restarting tasks after an abnormal NAS shutdown.
- Clarified the wording of tooltip descriptions.
- Clarified the wording of warning messages.
- Upgraded the OneDrive / OneDrive for Business API.
- Upgraded rsync to the latest version (3.1.3).

i wonder if this is related to this?
viewtopic.php?f=15&t=147968&hilit=clien ... on#p712276

i created a helpdesk ticket to report the issues i've found so far.

anyway, so now i'm testing the same settings minus client side encryption. i'm trying to check if the issue is specific to this setting or not :'

okay it works fine

so i think the previous bug is related to client side encryption :'

hm the restore job had sort of a problem. although it kinda says it completed, at the same time it says it didn't and was stuck at 0% in the UI. but i check share refresh the data seemed to have already transferred over. even on the source nas incoming jobs says restore job was completed. so there was a bug where the HBS job UI will show a "0%" completion and a probably false transfer speed reading. And in the log window it will show the restore job completed. hence the contradiction :S

Re: [review] HybridBackupSync 3 beta

Posted: Mon May 27, 2019 6:43 am
by Moogle Stiltzkin
next test is ts-877 to ts-509 pro, one way sync rtrr my normal settings like before.

it works. speed maxes out roughly 50-60 MB/s which is basically the best the ts-509 pro can do (it's a rather old model pretty much EOL at this point, which patches only up to 2020-12)


*just an update, it completes the sync backup just fine. 1.62tb transfered over.

the ts-509 pro is stuck using an older hbs version
Hybrid Backup Sync 1.3.180327
( 2018/05/11 )
but apparently it doesn't care, it still works. but i'd do the backup/sync/restore from the main nas since it has the newer HBS 8)

Re: [review] HybridBackupSync 3 beta

Posted: Sat Jun 15, 2019 6:32 pm
by Moogle Stiltzkin
resuming more testing of HBS.... i'll only post more if there are anything significantly different from the previous results i disclosed so far.

because i had to delete my initial restore because of a separate non related issue that caused random reboots. so rather than wonder if anything got corrupted, i'm just gonna restore again from backup.

I would just like to say that maybe HBS should be more clearer to the user as to how to perform verification of source compared to destination.

like there is the suggestion to run it twice to get such verification of integrity check that everything got copied over correctly and no deviation.
viewtopic.php?f=15&t=126139&p=708666&hi ... ts#p708666

maybe this is something that ought to be explained in the HBS slider intro, or as part of "tips and tricks" in the manual guide to teach the user how to perform this important task of verification.

also perhaps if there is a discrepancy, maybe some way to point out which exactly, then offer the user a choice as to what to do. Maybe the option to select those mismatched files and to sync or skip.

this kind of interface looks handy, where you can filter for which files exactly have mistmatched checksums when running a comparison check between source and destination.

Re: [review] HybridBackupSync 3 beta

Posted: Mon Jun 17, 2019 4:36 pm
by Moogle Stiltzkin
It's great that you can do scheduling in HBS 3

but..... problem is there is no unified management page for scheduling for the QNAP as a whole.

There is scheduling for HBS, then there is scheduling for raid scrub in storage manager, then there is the smart test scheduling in a sub section in storage manager, then another in power (restart/shutdown) scheduling.

Scheduling all over the place, and you never quite know for sure (unless you take the time to check each and everyone to be sure there is no conflicting scheduled task) if there is a conflict.

it's fine having those settings in those different areas, but it would ALSO be tremendously helpful if we had a unified page AS WELL where all these scheduled tasks can be seen from an overview and to make adjustments accordingly.

qnap needs to take a step back and see for settings like scheduling, also requiring a unified management page for it as well. especially when such settings may have conflicts with other settings.