Synchronising Two QNAPs accross the internet HELP

Discussion on remote replication.
Post Reply
thomas.dahl
Getting the hang of things
Posts: 53
Joined: Sat Apr 09, 2011 6:39 pm

Synchronising Two QNAPs accross the internet HELP

Post by thomas.dahl »

I know I know, this topic has been discussed a million times in this forum, but all comments about Rsync, RTRR etc etc.. do not really bring me forward. I am now way more confused than before.

I have two Qnaps (TS453B and TVS463) each sitting in its own network, one in Germany and one in Sweden. The two networks are VPN into each other using a router capability and access between the two networks is actually very quick. I am using Fritz routers. So far so good. 100mbit/100mbit in Sweden and 50mbit/10mbit in Germany.

I can see both networks and can share printers etc across both networks.

The data folder is around 1.8 Tb on both machines. I physically synchronized them 5 weeks ago while they were both on the same network (Germany) and it took a couple of days to copy and sync.

Now, using HBS3 it says it will take 83 DAYS to synchronize them !!

1. Only a few MB of data has changed between the two machines. Maybe 20 files. So why 83 DAYS?? Surely the two machines can check file names and modification dates of each file on each machine and see what needs to be copied and then only copy what is needed. What about using file checksums to verify files? Why on earth are huge amounts of data being shifted across the internet?? 99% of all this data is irrelevant as the files already exist on the other QNAP.

2. HBS3 2 way sync is not using RTRR or Rsync? Well at least there seems to be no way to choose. So why can these two services be started inside HBS3 if they are not used? Are they used? Why is HBS3 so confusing?

If HBS3 is a dogs breakfast, what other services or programs can I use to get the job done? Should I use a PC based program to do this as opposed to QNAP Apps? This will tie up a PC. SyncToy used to be a great tool, but no longer supported.

This seems to be a very common topic in the forum and across the internet (just google QNAP sync) and yet confusion still rules.

Please help me ! I am stuck in the forest in sweden and cannot see the wood for the trees.

Thomas
P3R
Guru
Posts: 13183
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Synchronising Two QNAPs accross the internet HELP

Post by P3R »

Hej Thomas!
thomas.dahl wrote: Tue Oct 13, 2020 2:13 pm 100mbit/100mbit in Sweden and 50mbit/10mbit in Germany.
Germany will be your bottleneck. 10 Mbit/s isn't much when trying to synchronize 1.8 TB.
Surely the two machines can check file names and modification dates of each file on each machine and see what needs to be copied and then only copy what is needed. What about using file checksums to verify files?
That's how Rsync work. RTRR is different so it depend on the protocol used.
HBS3 2 way sync is not using RTRR or Rsync?
Two-Way Sync uses RTRR.
Well at least there seems to be no way to choose.
You only have a choice of protocols when doing One-Way Sync (push) and Active Sync (pull), otherwise it's always RTRR.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
thomas.dahl
Getting the hang of things
Posts: 53
Joined: Sat Apr 09, 2011 6:39 pm

Re: Synchronising Two QNAPs accross the internet HELP

Post by thomas.dahl »

Thanks P3R. I will try one way sync an see if the speed becomes barable. I'll report back as this is an application that many many users seem to be looking for.

Why does HBS3 not mention this? Seriouly, why?

If you cannot sync then the QNAPS are useless as real time storage for teleworkers, and if they are not intended as back-ups, then I am starting to question what they should be used for...
P3R
Guru
Posts: 13183
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Synchronising Two QNAPs accross the internet HELP

Post by P3R »

thomas.dahl wrote: Tue Oct 13, 2020 3:50 pm I will try one way sync an see if the speed becomes barable.
Rsync is probably your only realistic alternative with the relatively slow connection (in relation to the amount of data). Even when using Rsync, it will be slow initially as the CPUs will struggle with calculating checksums for 1.8 TB data but it will become faster on subsequent runs as checksums will over time be cached.
Why does HBS3 not mention this? Seriouly, why?
To begin with I'm not with Qnap, I'm independent and have only learnt this from a long experience of working with their and other products so my attempt of an explanation is only mine.

Qnap-to-Qnap sync is what you happen to be interested in but it's only one of MANY different backup and sync scenarios built into a single interface. It's not easy to explain all those different technical details and keep the easy-to-use look-and-feel. Also a large majority of Qnap users are non-professionals that become confused and scared by technical terms and a lot of information.

Apart from some obvious details, like renaming "Active Sync" to "One-Way Pull Sync", I don't know how it could be done much better.
If you cannot sync then the QNAPS are useless as real time storage for teleworkers, and if they are not intended as back-ups, then I am starting to question what they should be used for...
I configure backups and syncs all the time so it's of course possible but require some understanding of how the system work. It's only that it's not always as easy to configure as Qnap and the customer wish it was... :wink:
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
thomas.dahl
Getting the hang of things
Posts: 53
Joined: Sat Apr 09, 2011 6:39 pm

Re: Synchronising Two QNAPs accross the internet HELP

Post by thomas.dahl »

Thanks for your help P3R. I now have a crazy patchwork of one-way syncs working now and at least my data seems safe.

I am having everyone from the second network VPN in to the first and only store data on the first QNAP. The second QNAP is now simply only a back-up and nothing else.

The is so totally not what I wanted to achieve with the two QNAPS. I wanted the two offices (Sweden and Germany) to be able to work independently of each other on their own QNAPs, and for them to be syncronised queitly in the back-ground. Seems this was never possible with QNAP products if you have a large data set that only changes slowly. 53 days to synchronise 2Tb of data, where only around 50Mb of file have changed. Nuts!

If anyone has an idea of another product, other than a MS server, that could do this I would be very appreciative your advice. No MS please any more..
dottcorbelli
New here
Posts: 5
Joined: Wed Jan 13, 2021 1:36 am

Re: Synchronising Two QNAPs accross the internet HELP

Post by dottcorbelli »

It can quite easily make a copy between the two systems, if the data is not changed significantly on a daily basis, using the ZPAQ program or its evolutions (eg zpaqfranz).
It has the feature of compressing and deduplicating but, above all, the generated file is only hanging. Therefore rsync only sends the data actually changed in the single day.

I use it to make a night (~1 hour) backup of about 1TB of data with a 2MB/s upload bandwith (Italy to France)
P3R
Guru
Posts: 13183
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Synchronising Two QNAPs accross the internet HELP

Post by P3R »

dottcorbelli wrote: Wed Jan 13, 2021 1:51 am It can quite easily make a copy between the two systems, if the data is not changed significantly on a daily basis, using the ZPAQ program or its evolutions (eg zpaqfranz).
Isn't ZPAQ an archiving tool rather than useful for synchronizations, that is the topic here?

Synchronization here mean creating an exact copy of the original source files at the destination.

HBS can also be used for backups with a specific backup format and then it's most likely considerably faster than for synchronizations. Deduplication, compression and encryption are also supported during backups.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
ForMar
New here
Posts: 3
Joined: Mon Jan 27, 2020 2:49 pm

Re: Synchronising Two QNAPs accross the internet HELP

Post by ForMar »

. I wanted the two offices (Sweden and Germany) to be able to work independently of each other on their own QNAPs, and for them to be syncronised queitly in the back-ground.
You can't achieve that with qnap and most of the other NAS Vendors as well( even Netapp ), because there is no file locking, so if 2 guys work on the same file, the last writer will win.
To achieve something like you want, you need tool like ClusterFS, PeerSync, or similar.

Kind regards
dottcorbelli
New here
Posts: 5
Joined: Wed Jan 13, 2021 1:36 am

Re: Synchronising Two QNAPs accross the internet HELP

Post by dottcorbelli »

P3R wrote: Wed Jan 13, 2021 4:05 am Isn't ZPAQ an archiving tool rather than useful for synchronizations, that is the topic here?
In fact it's zpaqfranz, a zpaq fork
Synchronization here mean creating an exact copy of the original source files at the destination.
In this case I use a rather different approach: zfs replica of iSCSI target (of course with UNIX, FreeBSD in fact)

They are clearly not simple NAS SMB shares.
Who knows, maybe the more advanced versions (with zfs) allow you to do it natively
Davo456
New here
Posts: 8
Joined: Sat Apr 10, 2021 11:24 pm

Re: Synchronising Two QNAPs accross the internet HELP

Post by Davo456 »

I sync 2 NAS's across 2 locations (UK and Switzerland), with about 3TB of data - initially sync did take a long time - I actually split into 3 jobs, and each job took around 5-10 days each the first time.
Now about 6 months later, each job takes between 15-30 minutes with 15 minutes, being no file changes, and 30 minutes for significant changes.
the 3 jobs, cascade each other, so I set the first one-off, and the second just follows when the first is complete. ditto for job 3.

my internet speeds are similar to the OP.
TS453A 2016-current
TS653A 2018-2021 RIP (backplane failed)
TS253B 2020-current
TS653D 2021-current
thomas.dahl
Getting the hang of things
Posts: 53
Joined: Sat Apr 09, 2011 6:39 pm

Re: Synchronising Two QNAPs accross the internet HELP

Post by thomas.dahl »

I am now doing something similar, except I have 20 jobs running at different times. The advantage for me is speed and the fact that I synchronize some folders more often where the data is likely to change more often.

5-10 days the first time ??? Yikes. I started getting these sorts of numbers but most of the data (also about 3Tb) was already synchronized when I had both QNAPS next to each other in the office. As I did not expect that there was much to synchronize I assumed that this was going to be the same every time.

THEN I discovered that one user was backing up his emails to one of the Qnaps (without my knowledge) and as this was a huge set of files the data was never synchronized fully... so it carried on, and on, and on... I would love to have some sort of reality check wizard that can catch and alarm these things happening (I know I know..tricky and newby mistake) because it can cost a lot of time finding out where the issues are, particularly when you are new to synching.

I ended up setting up a VPN using internal tools available in the routers I used. Was quite tricky to do and I nearly gave up, but now it works and it rock solid and fast. I can certainly recommend doing it this way. One of the routers had to have a fixed IP address for this to work which cost me a little more money each month with German Telekom.
Post Reply

Return to “Remote Replication/ Disaster Recovery”