[review] HybridBackupSync 3 beta

Introduce yourself to us and other members here, or share your own product reviews, suggestions, and tips and tricks of using QNAP products.
Post Reply
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

[review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

You can join the contest here to participate
viewtopic.php?f=45&t=148657
https://www.qnap.com/solution/hbs3/en/#f11

QNAP TBS 453DX NAS Book - Hardware Review *on your marks get set....GO... :D
https://www.youtube.com/watch?v=ucVlERwlR2A
https://www.youtube.com/watch?v=dXM1XBbVhQo


For qudedupe i'll probably use some data test samples rather than my official stuff in a test share :) still have to figure out if this is something i'd do for backing up. but at least for filesharing i probly use it.


some of the new features of HBS 3

TCP BBR
Image
TCP BBR : Magic dust for network performance.

It’s lost on most people that today’s internet is running on algorithms and protocols that were designed back in the 80s. You know, back when things like “message boards” where the most exciting thing going on.

But that’s not today’s internet. Today’s internet is distributed across 7 billion people on devices and network connectivity ranging from awesome great 1GB / sec wired connections to wireless ones that… well.. Aren’t.

And that brings up the question that perhaps today’s internet needs require and upgrade to those algorithms we made back 30 years ago.

This is where TCP BBR comes in. It’s a TCP congestion control algorithm built for the congestion of the modern internet.
https://medium.com/google-cloud/tcp-bbr ... a5f1ccf437

https://www.youtube.com/watch?v=PkWleEfaGAU



QuDedupe bandwidth reduction & encryption?
Image

https://www.youtube.com/watch?v=KeQyptqo33M
https://www.youtube.com/watch?v=xsSPnBbuDNU


faster versioning?
Image


cross device access
Image


also new UI
Image

Image

Image


the review will cover my experience testing with this, thoughts on the features and UI, whether it helped or what needs to be changed/tweaked. This is something i actually use for my NAS backup 8)
Last edited by Moogle Stiltzkin on Sun Sep 29, 2019 12:45 pm, edited 11 times in total.
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

Just a small preview of what i found out so far.

QuDedupe requires that BOTH the sauce and destination QNAP NAS have it installed (so if you got an older HBS on your NAS e.g. an EOL model, well thats not gonna fly i'm afraid)
viewtopic.php?f=45&t=148657&p=716090#p716090


So i'm gonna have to run 2 tests

- HBS 3 beta on both a TS-877 (sauce) to TS-653a (destination)

- HBS3 beta on Ts-877 to TS-509 Pro (this is eol and using a much older hbs :( )
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

okay i discovered at least 1-2 weird bugs. but i managed to get around it by trying somethings. so if you are wondering, yes there are definitely some bugs *hence it's beta

not sure i recommend this for production just yet :)

i created a share for testing purposes for now.
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

one nice option is the client side encryption. basically you encrypt your data PRIOR to backing it up remotely (in my case onto the backup NAS on the same lan). So my guess is, the stored backup will be encrypted (because in the settings for client side encryption, it requires a password to decrypt). So if you want a secure backup, this is probably ideal.

This is probably something i'd use for backing up my encrypted shares, so that the backed up data is encrypted. Encrypting the share on the backup NAS is one possible solution, but then you would need to remember to unlock before running your HBS backup, and then to lock it back again manually. It seems like just using the client side encryption for the backup is the simpler solution to go with.

Image

Image


the remove deleted data from destination option is useful for me. because i don't want to retain data on the backup that is no longer on the main NAS (though i have to be mindful of not accidentally deleting something i actually want, and end up removing it on the backup too). There is also an option to retain the data before deleting after x days which is useful as a precaution for accidents (for what i just mentioned).
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

This is my first test with the following settings configured

- create a backup job to remote (a ts-877 to a ts-653a on the local lan)
- enable QuDedupe
- enable client side encryption (i'll probably only use this for data stored on my encrypted shares, so i'd have a separate job just for that for my own production backup. If i was doing this over the internet, i'd probably enable this for all, but for local lan i'm not sure it's needed unless it's for sensitive data)
- delete data from destination (did they rename the delete extra files ? well it's explained what it does, and seems to do the same thing)

I wanted to use the backup only new files, but that seemed to be greyed out, no idea why. it wasn't explained :'


Image

Image


you can run your backup either from the 1st or 2nd screenshot locations. but usually i prefer doing mine from the 2nd screenshot. also as you can see the recover jobs i have disabled. for safety sake, i recommend disabling them when not actively using. It would be a disaster if i overwrote my main NAS from backup by hitting a wrong button :shock: i only unlock that when i actually am going to recover for realz only.


Here is the first successful backup job. The sample size of the backup was too small, so i could not really test out the effectiveness of the TCP BBR just yet (also how much of a performance impact would using client side encryption cause. i'll run a complete test later for that)
Image


as you can see, QuDedupe created a bunch of weird stuff in it. Well as long as the file size is same or smaller than not using it, and i can successfully recover my files, then i don't particularly mind. But it's clear that you won't be able to just simply open up your files on your backup just like that. You'd have to recover them back to your main NAS for them to work again (which i will test in a bit by the way by deleting the sauce data then using the recover and see how swellingly that works out)

Image
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

ok now testing recovery.

delete the sauce data using filestation (sometimes when i do it from win explorer it MAY not actually be deleted. so to be certain, i use filestation)
Image


next is to recover in HBS.

okay it worked kinda (got confused by some of the settings that initially would not let me procede). i found some issue which i will report :S
Image

all files restored, crc checks out. tested files as working
Image



next test i will run is a less restrictive setup that i would probably use for most of my backup. i'll try without client side encryption. I'll however enable SSL encryption. Will test with qudedupe enabled still. Will run a separate test later with it disabled.
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

new test

one way sync rtrr with ssl encryption enabled.

In 1 way sync, there is no client side encryption and qudedupe option. other options get unlocked instead for this mode.
Image

Image

anyway it worked
Image


this is the setting i most often use and am familiar with, so i'll do a bigger backup test to get a feel for transfer speed improvements.


i'll run separate tests for hbs BACKUP to check out qudedupe more though.



from what i've noticed so far, the only time TCP BBR got mentioned was in an option during the recovery test (for the HBS backup). i did not see it mentioned anywhere else :'

i just assumed that if u run HBS 3 beta and RTRR that TCP BBR will be working. Not sure if running the new HBS on the destination is also a requirement for it (only qudedupe so far mentioned it as a requirement)
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

Creating a HBS BACKUP test 2.

Won't use client side encryption. Will only test with qudedupe enabled and the other settings from before. I'll leave SSL enabled though.

I'll also use a bigger sample size of source data to backup (roughly 300 gb). Will take a while to copy over to the new test hbs 2 share i made for the purpose.
Image


will be testing selectively deleting files and seeing if that correctly replicates over to the backup and just testing how it works.

also testing if qudedupe windows app for decrypting actually works or not (at least previously it did decrypt fine via HBS when running the restore)
Last edited by Moogle Stiltzkin on Sat May 25, 2019 3:01 pm, edited 2 times in total.
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

After doing a bit more research, wondering if the new HBS needs to be on both the source and destination qnap for TCP BBR to work, i don't think that it does
Doesn’t require the client to implement BBR. This one is the magic pixie dust. Prior algorithms like QUIC required client & server to have both implemented the algorithm. BBR doesn’t require the client to be using BBR as well. This is especially relevant in the developing world which use older mobile platforms and have limited bandwidth, or areas where websites & services haven’t made the switch yet.
https://medium.com/google-cloud/tcp-bbr ... a5f1ccf437
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

performance test

prepping..

first did an extra step i ran a test to ensure there is no issues with ethernet cables on switch. so performance issues can probably look elsewhere :'
Image


both qnaps ts-877 and ts-653a updated qts and using the hbs 3 beta, latest switch and router firmwares, all devices rebooted. test prep ready.


results
Image


my used configuration

RTRR, one way sync, no filters, remove additional files at destination, check file contents, exclude system generated temp files, do not take a snapshot, SSL disabled.

I'll try later with ssl, but i'm not expecting there to be issue with it enabled. It's just simply i am running over local lan, so not sure that i require it for my case scenario :)

side note: had SSD cache acceleration enabled, random I/O , bypass 1MB block. I was reading another thread where it implied ssd cache acceleration was slowing down performance for transfer, but i'm not really having this specific issue, so i did not bother to disable (although i did disable then re-enable so the capacity utility is on the low end. in thread they mention issues when it is almost full)
viewtopic.php?f=25&t=124852


then delete data on main NAS, and perform a restore from backup NAS to main NAS. speed is more or less the same
Image


so the rtrr 1 way sync + restore seems to work fine.

only the backup mode and with qudedupe requires more testing :'




*update

ran a complete backup (5tb ish), same settings, everything seems to have worked as intended.

results
Image


source
Image


destination
Image

difference in folder and files is probably thumbnails. the file size storage at least is the same.
Last edited by Moogle Stiltzkin on Sun May 26, 2019 5:33 pm, edited 2 times in total.
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

did some digging, yes they do explain the difference between a backup and sync. although maybe they should make it into the intro slider a quick summary about it perhaps? with links that go more indepth into an explanation comparing the 2. i'll bet you a bunch of newbies will be wondering which they should be using to perform a backup.

https://blog.qnap.com/sync-vs-backup-explained/

https://www.qnap.com/en-in/how-to/knowl ... p-station/


About Synchronization (One-Way,Two-Way), Backup and Replication
File Synchronization (a.k.a. File Sync) is the process of ensuring that two or more locations contain the same, up-to-date, files. If you add, change, or delete a file from one location, the synchronization process will add, change, or delete the corresponding file at the other location.

Synchronization can be “two-way” or “one-way”.

Two-Way Synchronization (a.k.a. bi-directional synchronization or both-ways synchronization):

This synchronization process copies files in both directions to reconcile changes as needed. Files are expected to change in both locations. The two locations are considered equivalent.

Example: if a file is newer in Location A, it will be copied to Location B. But if a file is newer in Location B it will be copied to Location A. Similarly, if a file is deleted from Location A, it will be deleted from Location B as well and vice versa.

One-Way Synchronization (a.k.a. file mirroring / file replication / file backup):

Files are expected to change in one location only. To reconcile the changes, the synchronization process copies files only in one direction. The two locations are not considered equivalent. One location is considered the Source and the other is considered the Target. Files are pushed from Source to Target (or files are pulled from Source to Target, but always in one direction only). Source is said to be mirrored to Target.

This creates an exact 1:1 replica of all files in Source to Target. This is very useful and effective for backup purposes since only changed/new files are copied.

Example: If a file is newer in Location A (Source), it will be copied to Location B (Target). But if a file is newer in Location B, it will NOT be copied to Location A. Similarly, if a file is deleted from Location A, it will be deleted from Location B. But a file deleted in Location B is restored using the file in Location A.
https://www.tgrmn.com/web/kb/item78.htm


this is why i tend to use rtrr one way sync. and have in the past used it successfully as a backup.

and in the current hbs 3 beta test, the performance i'm getting is 80 MB/s + despite having things like checksum verification, delete extra files etc enabled. So thats all fine and dandy for me.


Sync vs. Backup Explained

What’s the real difference between sync and backup? What are the pros and cons of them? We often get confused when hearing terms like that, and conclude in our minds that they are the same. Since our lives have basically shifted towards the digital realm where we are constantly focused on the phone, the tablet, computers and even now smart watches ; yet some of us don’t even know if the data is backed up or synced somewhere.

Sync, means synchronization. Therefore a ‘syncing’ action happens with your photos on your phone in the background… And hopefully you’ve registered yourself to sync your photos to iCloud (if you’re using an Apple iPhone) or for the recent Android devices they should be syncing somewhere onto the cloud. One should also note that sync is not a backup. Syncing lets you copy or delete files as needed to ensure that two locations have the same exact file or version of the files. Examples of syncing is Dropbox, Google Drive as the files stored on them are the same exact versions of each other whether you view it on your laptop, phone, or computer etc. So make sure, to also backup apart from syncing your files.

On a QNAP NAS… You can always backup AND sync your files at the same time.
but my one way sync is an exact replica and works for recovery. so why wouldn't it be considered a backup? :'
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

testing HBS Backup with qudedupe disabled.

but using client side encryption, remove additional files, check file contents, exclude system generated temp files, do not take snapshots, ssl disabled.
Image


once it completes i'll test if it encrypts client side or not. supposedly it requires a password to unlock. checking how that works.



*update

ok weird error. the backup gets disrupted, then it resumes. i also notice the destination NAS had triggered the alarm for reach threshold 80% (related?). Anyway it's resumed so keeping an eye on it.
Image

*update

this broken pipe error happened 2 more times. each time the backup process resets back to 0% so you start over again from the start :shock: very doubtful this will finish completing. nor would i use this setting for my own backup until it actually works :(


for now i deleted anything at destination (but i don't see why i should. by rights the settings should delete anything not matching source to begin with), re-created the job same settings, and run again. see if it works this time.

nope it didn't.

fyi, client side encryption was added sometime since last year. it's not something i used at the time. so i'm not sure if the issue is new and only occuring with this latest hbs beta or what. i'm merely reporting my results as is :(
Hybrid Backup Sync 2.1.180816
( 2018/08/22 )

[Enhancements]
- Added support for enabling client-side encryption with RTRR backup jobs.
- Added support for restarting tasks after an abnormal NAS shutdown.
- Clarified the wording of tooltip descriptions.
- Clarified the wording of warning messages.
- Upgraded the OneDrive / OneDrive for Business API.
- Upgraded rsync to the latest version (3.1.3).

i wonder if this is related to this?
viewtopic.php?f=15&t=147968&hilit=clien ... on#p712276

i created a helpdesk ticket to report the issues i've found so far.


anyway, so now i'm testing the same settings minus client side encryption. i'm trying to check if the issue is specific to this setting or not :'

okay it works fine
Image



so i think the previous bug is related to client side encryption :'


hm the restore job had sort of a problem. although it kinda says it completed, at the same time it says it didn't and was stuck at 0% in the UI. but i check share refresh the data seemed to have already transferred over. even on the source nas incoming jobs says restore job was completed. so there was a bug where the HBS job UI will show a "0%" completion and a probably false transfer speed reading. And in the log window it will show the restore job completed. hence the contradiction :S
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

next test is ts-877 to ts-509 pro, one way sync rtrr my normal settings like before.

it works. speed maxes out roughly 50-60 MB/s which is basically the best the ts-509 pro can do (it's a rather old model pretty much EOL at this point, which patches only up to 2020-12)

Image

*just an update, it completes the sync backup just fine. 1.62tb transfered over.


the ts-509 pro is stuck using an older hbs version
Hybrid Backup Sync 1.3.180327
( 2018/05/11 )
but apparently it doesn't care, it still works. but i'd do the backup/sync/restore from the main nas since it has the newer HBS 8)
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

resuming more testing of HBS.... i'll only post more if there are anything significantly different from the previous results i disclosed so far.

because i had to delete my initial restore because of a separate non related issue that caused random reboots. so rather than wonder if anything got corrupted, i'm just gonna restore again from backup.

I would just like to say that maybe HBS should be more clearer to the user as to how to perform verification of source compared to destination.

like there is the suggestion to run it twice to get such verification of integrity check that everything got copied over correctly and no deviation.
viewtopic.php?f=15&t=126139&p=708666&hi ... ts#p708666


maybe this is something that ought to be explained in the HBS slider intro, or as part of "tips and tricks" in the manual guide to teach the user how to perform this important task of verification.


also perhaps if there is a discrepancy, maybe some way to point out which exactly, then offer the user a choice as to what to do. Maybe the option to select those mismatched files and to sync or skip.

this kind of interface looks handy, where you can filter for which files exactly have mistmatched checksums when running a comparison check between source and destination.
Image
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [review] HybridBackupSync 3 beta

Post by Moogle Stiltzkin »

It's great that you can do scheduling in HBS 3
Image

but..... problem is there is no unified management page for scheduling for the QNAP as a whole.

There is scheduling for HBS, then there is scheduling for raid scrub in storage manager, then there is the smart test scheduling in a sub section in storage manager, then another in power (restart/shutdown) scheduling.

Scheduling all over the place, and you never quite know for sure (unless you take the time to check each and everyone to be sure there is no conflicting scheduled task) if there is a conflict.

it's fine having those settings in those different areas, but it would ALSO be tremendously helpful if we had a unified page AS WELL where all these scheduled tasks can be seen from an overview and to make adjustments accordingly.

qnap needs to take a step back and see for settings like scheduling, also requiring a unified management page for it as well. especially when such settings may have conflicts with other settings.
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
Post Reply

Return to “Users' Corner”