Synchronising Two QNAPs accross the internet HELP
-
- Getting the hang of things
- Posts: 53
- Joined: Sat Apr 09, 2011 6:39 pm
Synchronising Two QNAPs accross the internet HELP
I know I know, this topic has been discussed a million times in this forum, but all comments about Rsync, RTRR etc etc.. do not really bring me forward. I am now way more confused than before.
I have two Qnaps (TS453B and TVS463) each sitting in its own network, one in Germany and one in Sweden. The two networks are VPN into each other using a router capability and access between the two networks is actually very quick. I am using Fritz routers. So far so good. 100mbit/100mbit in Sweden and 50mbit/10mbit in Germany.
I can see both networks and can share printers etc across both networks.
The data folder is around 1.8 Tb on both machines. I physically synchronized them 5 weeks ago while they were both on the same network (Germany) and it took a couple of days to copy and sync.
Now, using HBS3 it says it will take 83 DAYS to synchronize them !!
1. Only a few MB of data has changed between the two machines. Maybe 20 files. So why 83 DAYS?? Surely the two machines can check file names and modification dates of each file on each machine and see what needs to be copied and then only copy what is needed. What about using file checksums to verify files? Why on earth are huge amounts of data being shifted across the internet?? 99% of all this data is irrelevant as the files already exist on the other QNAP.
2. HBS3 2 way sync is not using RTRR or Rsync? Well at least there seems to be no way to choose. So why can these two services be started inside HBS3 if they are not used? Are they used? Why is HBS3 so confusing?
If HBS3 is a dogs breakfast, what other services or programs can I use to get the job done? Should I use a PC based program to do this as opposed to QNAP Apps? This will tie up a PC. SyncToy used to be a great tool, but no longer supported.
This seems to be a very common topic in the forum and across the internet (just google QNAP sync) and yet confusion still rules.
Please help me ! I am stuck in the forest in sweden and cannot see the wood for the trees.
Thomas
I have two Qnaps (TS453B and TVS463) each sitting in its own network, one in Germany and one in Sweden. The two networks are VPN into each other using a router capability and access between the two networks is actually very quick. I am using Fritz routers. So far so good. 100mbit/100mbit in Sweden and 50mbit/10mbit in Germany.
I can see both networks and can share printers etc across both networks.
The data folder is around 1.8 Tb on both machines. I physically synchronized them 5 weeks ago while they were both on the same network (Germany) and it took a couple of days to copy and sync.
Now, using HBS3 it says it will take 83 DAYS to synchronize them !!
1. Only a few MB of data has changed between the two machines. Maybe 20 files. So why 83 DAYS?? Surely the two machines can check file names and modification dates of each file on each machine and see what needs to be copied and then only copy what is needed. What about using file checksums to verify files? Why on earth are huge amounts of data being shifted across the internet?? 99% of all this data is irrelevant as the files already exist on the other QNAP.
2. HBS3 2 way sync is not using RTRR or Rsync? Well at least there seems to be no way to choose. So why can these two services be started inside HBS3 if they are not used? Are they used? Why is HBS3 so confusing?
If HBS3 is a dogs breakfast, what other services or programs can I use to get the job done? Should I use a PC based program to do this as opposed to QNAP Apps? This will tie up a PC. SyncToy used to be a great tool, but no longer supported.
This seems to be a very common topic in the forum and across the internet (just google QNAP sync) and yet confusion still rules.
Please help me ! I am stuck in the forest in sweden and cannot see the wood for the trees.
Thomas
-
- Guru
- Posts: 13190
- Joined: Sat Dec 29, 2007 1:39 am
- Location: Stockholm, Sweden (UTC+01:00)
Re: Synchronising Two QNAPs accross the internet HELP
Hej Thomas!
Germany will be your bottleneck. 10 Mbit/s isn't much when trying to synchronize 1.8 TB.thomas.dahl wrote: ↑Tue Oct 13, 2020 2:13 pm 100mbit/100mbit in Sweden and 50mbit/10mbit in Germany.
That's how Rsync work. RTRR is different so it depend on the protocol used.Surely the two machines can check file names and modification dates of each file on each machine and see what needs to be copied and then only copy what is needed. What about using file checksums to verify files?
Two-Way Sync uses RTRR.HBS3 2 way sync is not using RTRR or Rsync?
You only have a choice of protocols when doing One-Way Sync (push) and Active Sync (pull), otherwise it's always RTRR.Well at least there seems to be no way to choose.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
-
- Getting the hang of things
- Posts: 53
- Joined: Sat Apr 09, 2011 6:39 pm
Re: Synchronising Two QNAPs accross the internet HELP
Thanks P3R. I will try one way sync an see if the speed becomes barable. I'll report back as this is an application that many many users seem to be looking for.
Why does HBS3 not mention this? Seriouly, why?
If you cannot sync then the QNAPS are useless as real time storage for teleworkers, and if they are not intended as back-ups, then I am starting to question what they should be used for...
Why does HBS3 not mention this? Seriouly, why?
If you cannot sync then the QNAPS are useless as real time storage for teleworkers, and if they are not intended as back-ups, then I am starting to question what they should be used for...
-
- Guru
- Posts: 13190
- Joined: Sat Dec 29, 2007 1:39 am
- Location: Stockholm, Sweden (UTC+01:00)
Re: Synchronising Two QNAPs accross the internet HELP
Rsync is probably your only realistic alternative with the relatively slow connection (in relation to the amount of data). Even when using Rsync, it will be slow initially as the CPUs will struggle with calculating checksums for 1.8 TB data but it will become faster on subsequent runs as checksums will over time be cached.thomas.dahl wrote: ↑Tue Oct 13, 2020 3:50 pm I will try one way sync an see if the speed becomes barable.
To begin with I'm not with Qnap, I'm independent and have only learnt this from a long experience of working with their and other products so my attempt of an explanation is only mine.Why does HBS3 not mention this? Seriouly, why?
Qnap-to-Qnap sync is what you happen to be interested in but it's only one of MANY different backup and sync scenarios built into a single interface. It's not easy to explain all those different technical details and keep the easy-to-use look-and-feel. Also a large majority of Qnap users are non-professionals that become confused and scared by technical terms and a lot of information.
Apart from some obvious details, like renaming "Active Sync" to "One-Way Pull Sync", I don't know how it could be done much better.
I configure backups and syncs all the time so it's of course possible but require some understanding of how the system work. It's only that it's not always as easy to configure as Qnap and the customer wish it was...If you cannot sync then the QNAPS are useless as real time storage for teleworkers, and if they are not intended as back-ups, then I am starting to question what they should be used for...
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
-
- Getting the hang of things
- Posts: 53
- Joined: Sat Apr 09, 2011 6:39 pm
Re: Synchronising Two QNAPs accross the internet HELP
Thanks for your help P3R. I now have a crazy patchwork of one-way syncs working now and at least my data seems safe.
I am having everyone from the second network VPN in to the first and only store data on the first QNAP. The second QNAP is now simply only a back-up and nothing else.
The is so totally not what I wanted to achieve with the two QNAPS. I wanted the two offices (Sweden and Germany) to be able to work independently of each other on their own QNAPs, and for them to be syncronised queitly in the back-ground. Seems this was never possible with QNAP products if you have a large data set that only changes slowly. 53 days to synchronise 2Tb of data, where only around 50Mb of file have changed. Nuts!
If anyone has an idea of another product, other than a MS server, that could do this I would be very appreciative your advice. No MS please any more..
I am having everyone from the second network VPN in to the first and only store data on the first QNAP. The second QNAP is now simply only a back-up and nothing else.
The is so totally not what I wanted to achieve with the two QNAPS. I wanted the two offices (Sweden and Germany) to be able to work independently of each other on their own QNAPs, and for them to be syncronised queitly in the back-ground. Seems this was never possible with QNAP products if you have a large data set that only changes slowly. 53 days to synchronise 2Tb of data, where only around 50Mb of file have changed. Nuts!
If anyone has an idea of another product, other than a MS server, that could do this I would be very appreciative your advice. No MS please any more..
-
- New here
- Posts: 5
- Joined: Wed Jan 13, 2021 1:36 am
Re: Synchronising Two QNAPs accross the internet HELP
It can quite easily make a copy between the two systems, if the data is not changed significantly on a daily basis, using the ZPAQ program or its evolutions (eg zpaqfranz).
It has the feature of compressing and deduplicating but, above all, the generated file is only hanging. Therefore rsync only sends the data actually changed in the single day.
I use it to make a night (~1 hour) backup of about 1TB of data with a 2MB/s upload bandwith (Italy to France)
It has the feature of compressing and deduplicating but, above all, the generated file is only hanging. Therefore rsync only sends the data actually changed in the single day.
I use it to make a night (~1 hour) backup of about 1TB of data with a 2MB/s upload bandwith (Italy to France)
-
- Guru
- Posts: 13190
- Joined: Sat Dec 29, 2007 1:39 am
- Location: Stockholm, Sweden (UTC+01:00)
Re: Synchronising Two QNAPs accross the internet HELP
Isn't ZPAQ an archiving tool rather than useful for synchronizations, that is the topic here?dottcorbelli wrote: ↑Wed Jan 13, 2021 1:51 am It can quite easily make a copy between the two systems, if the data is not changed significantly on a daily basis, using the ZPAQ program or its evolutions (eg zpaqfranz).
Synchronization here mean creating an exact copy of the original source files at the destination.
HBS can also be used for backups with a specific backup format and then it's most likely considerably faster than for synchronizations. Deduplication, compression and encryption are also supported during backups.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
-
- New here
- Posts: 4
- Joined: Mon Jan 27, 2020 2:49 pm
Re: Synchronising Two QNAPs accross the internet HELP
You can't achieve that with qnap and most of the other NAS Vendors as well( even Netapp ), because there is no file locking, so if 2 guys work on the same file, the last writer will win.. I wanted the two offices (Sweden and Germany) to be able to work independently of each other on their own QNAPs, and for them to be syncronised queitly in the back-ground.
To achieve something like you want, you need tool like ClusterFS, PeerSync, or similar.
Kind regards
-
- New here
- Posts: 5
- Joined: Wed Jan 13, 2021 1:36 am
Re: Synchronising Two QNAPs accross the internet HELP
In fact it's zpaqfranz, a zpaq fork
In this case I use a rather different approach: zfs replica of iSCSI target (of course with UNIX, FreeBSD in fact)Synchronization here mean creating an exact copy of the original source files at the destination.
They are clearly not simple NAS SMB shares.
Who knows, maybe the more advanced versions (with zfs) allow you to do it natively
-
- New here
- Posts: 8
- Joined: Sat Apr 10, 2021 11:24 pm
Re: Synchronising Two QNAPs accross the internet HELP
I sync 2 NAS's across 2 locations (UK and Switzerland), with about 3TB of data - initially sync did take a long time - I actually split into 3 jobs, and each job took around 5-10 days each the first time.
Now about 6 months later, each job takes between 15-30 minutes with 15 minutes, being no file changes, and 30 minutes for significant changes.
the 3 jobs, cascade each other, so I set the first one-off, and the second just follows when the first is complete. ditto for job 3.
my internet speeds are similar to the OP.
Now about 6 months later, each job takes between 15-30 minutes with 15 minutes, being no file changes, and 30 minutes for significant changes.
the 3 jobs, cascade each other, so I set the first one-off, and the second just follows when the first is complete. ditto for job 3.
my internet speeds are similar to the OP.
TS453A 2016-current
TS653A 2018-2021 RIP (backplane failed)
TS253B 2020-current
TS653D 2021-current
TS653A 2018-2021 RIP (backplane failed)
TS253B 2020-current
TS653D 2021-current
-
- Getting the hang of things
- Posts: 53
- Joined: Sat Apr 09, 2011 6:39 pm
Re: Synchronising Two QNAPs accross the internet HELP
I am now doing something similar, except I have 20 jobs running at different times. The advantage for me is speed and the fact that I synchronize some folders more often where the data is likely to change more often.
5-10 days the first time ??? Yikes. I started getting these sorts of numbers but most of the data (also about 3Tb) was already synchronized when I had both QNAPS next to each other in the office. As I did not expect that there was much to synchronize I assumed that this was going to be the same every time.
THEN I discovered that one user was backing up his emails to one of the Qnaps (without my knowledge) and as this was a huge set of files the data was never synchronized fully... so it carried on, and on, and on... I would love to have some sort of reality check wizard that can catch and alarm these things happening (I know I know..tricky and newby mistake) because it can cost a lot of time finding out where the issues are, particularly when you are new to synching.
I ended up setting up a VPN using internal tools available in the routers I used. Was quite tricky to do and I nearly gave up, but now it works and it rock solid and fast. I can certainly recommend doing it this way. One of the routers had to have a fixed IP address for this to work which cost me a little more money each month with German Telekom.
5-10 days the first time ??? Yikes. I started getting these sorts of numbers but most of the data (also about 3Tb) was already synchronized when I had both QNAPS next to each other in the office. As I did not expect that there was much to synchronize I assumed that this was going to be the same every time.
THEN I discovered that one user was backing up his emails to one of the Qnaps (without my knowledge) and as this was a huge set of files the data was never synchronized fully... so it carried on, and on, and on... I would love to have some sort of reality check wizard that can catch and alarm these things happening (I know I know..tricky and newby mistake) because it can cost a lot of time finding out where the issues are, particularly when you are new to synching.
I ended up setting up a VPN using internal tools available in the routers I used. Was quite tricky to do and I nearly gave up, but now it works and it rock solid and fast. I can certainly recommend doing it this way. One of the routers had to have a fixed IP address for this to work which cost me a little more money each month with German Telekom.