Synchronising Two QNAPs accross the internet HELP

Discussion on remote replication.
Post Reply
thomas.dahl
Starting out
Posts: 47
Joined: Sat Apr 09, 2011 6:39 pm

Synchronising Two QNAPs accross the internet HELP

Post by thomas.dahl » Tue Oct 13, 2020 2:13 pm

I know I know, this topic has been discussed a million times in this forum, but all comments about Rsync, RTRR etc etc.. do not really bring me forward. I am now way more confused than before.

I have two Qnaps (TS453B and TVS463) each sitting in its own network, one in Germany and one in Sweden. The two networks are VPN into each other using a router capability and access between the two networks is actually very quick. I am using Fritz routers. So far so good. 100mbit/100mbit in Sweden and 50mbit/10mbit in Germany.

I can see both networks and can share printers etc across both networks.

The data folder is around 1.8 Tb on both machines. I physically synchronized them 5 weeks ago while they were both on the same network (Germany) and it took a couple of days to copy and sync.

Now, using HBS3 it says it will take 83 DAYS to synchronize them !!

1. Only a few MB of data has changed between the two machines. Maybe 20 files. So why 83 DAYS?? Surely the two machines can check file names and modification dates of each file on each machine and see what needs to be copied and then only copy what is needed. What about using file checksums to verify files? Why on earth are huge amounts of data being shifted across the internet?? 99% of all this data is irrelevant as the files already exist on the other QNAP.

2. HBS3 2 way sync is not using RTRR or Rsync? Well at least there seems to be no way to choose. So why can these two services be started inside HBS3 if they are not used? Are they used? Why is HBS3 so confusing?

If HBS3 is a dogs breakfast, what other services or programs can I use to get the job done? Should I use a PC based program to do this as opposed to QNAP Apps? This will tie up a PC. SyncToy used to be a great tool, but no longer supported.

This seems to be a very common topic in the forum and across the internet (just google QNAP sync) and yet confusion still rules.

Please help me ! I am stuck in the forest in sweden and cannot see the wood for the trees.

Thomas

P3R
Guru
Posts: 12334
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Synchronising Two QNAPs accross the internet HELP

Post by P3R » Tue Oct 13, 2020 2:53 pm

Hej Thomas!
thomas.dahl wrote:
Tue Oct 13, 2020 2:13 pm
100mbit/100mbit in Sweden and 50mbit/10mbit in Germany.
Germany will be your bottleneck. 10 Mbit/s isn't much when trying to synchronize 1.8 TB.
Surely the two machines can check file names and modification dates of each file on each machine and see what needs to be copied and then only copy what is needed. What about using file checksums to verify files?
That's how Rsync work. RTRR is different so it depend on the protocol used.
HBS3 2 way sync is not using RTRR or Rsync?
Two-Way Sync uses RTRR.
Well at least there seems to be no way to choose.
You only have a choice of protocols when doing One-Way Sync (push) and Active Sync (pull), otherwise it's always RTRR.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

thomas.dahl
Starting out
Posts: 47
Joined: Sat Apr 09, 2011 6:39 pm

Re: Synchronising Two QNAPs accross the internet HELP

Post by thomas.dahl » Tue Oct 13, 2020 3:50 pm

Thanks P3R. I will try one way sync an see if the speed becomes barable. I'll report back as this is an application that many many users seem to be looking for.

Why does HBS3 not mention this? Seriouly, why?

If you cannot sync then the QNAPS are useless as real time storage for teleworkers, and if they are not intended as back-ups, then I am starting to question what they should be used for...

P3R
Guru
Posts: 12334
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Synchronising Two QNAPs accross the internet HELP

Post by P3R » Tue Oct 13, 2020 5:07 pm

thomas.dahl wrote:
Tue Oct 13, 2020 3:50 pm
I will try one way sync an see if the speed becomes barable.
Rsync is probably your only realistic alternative with the relatively slow connection (in relation to the amount of data). Even when using Rsync, it will be slow initially as the CPUs will struggle with calculating checksums for 1.8 TB data but it will become faster on subsequent runs as checksums will over time be cached.
Why does HBS3 not mention this? Seriouly, why?
To begin with I'm not with Qnap, I'm independent and have only learnt this from a long experience of working with their and other products so my attempt of an explanation is only mine.

Qnap-to-Qnap sync is what you happen to be interested in but it's only one of MANY different backup and sync scenarios built into a single interface. It's not easy to explain all those different technical details and keep the easy-to-use look-and-feel. Also a large majority of Qnap users are non-professionals that become confused and scared by technical terms and a lot of information.

Apart from some obvious details, like renaming "Active Sync" to "One-Way Pull Sync", I don't know how it could be done much better.
If you cannot sync then the QNAPS are useless as real time storage for teleworkers, and if they are not intended as back-ups, then I am starting to question what they should be used for...
I configure backups and syncs all the time so it's of course possible but require some understanding of how the system work. It's only that it's not always as easy to configure as Qnap and the customer wish it was... :wink:
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

thomas.dahl
Starting out
Posts: 47
Joined: Sat Apr 09, 2011 6:39 pm

Re: Synchronising Two QNAPs accross the internet HELP

Post by thomas.dahl » Wed Oct 14, 2020 5:06 pm

Thanks for your help P3R. I now have a crazy patchwork of one-way syncs working now and at least my data seems safe.

I am having everyone from the second network VPN in to the first and only store data on the first QNAP. The second QNAP is now simply only a back-up and nothing else.

The is so totally not what I wanted to achieve with the two QNAPS. I wanted the two offices (Sweden and Germany) to be able to work independently of each other on their own QNAPs, and for them to be syncronised queitly in the back-ground. Seems this was never possible with QNAP products if you have a large data set that only changes slowly. 53 days to synchronise 2Tb of data, where only around 50Mb of file have changed. Nuts!

If anyone has an idea of another product, other than a MS server, that could do this I would be very appreciative your advice. No MS please any more..

Post Reply

Return to “Remote Replication/ Disaster Recovery”