RTRR syncing all files instead of changed

Backup, Restore, Netbak Replicator, Cloud Storage Services
Post Reply
RoelVB
New here
Posts: 5
Joined: Sat Jan 26, 2019 7:45 pm

RTRR syncing all files instead of changed

Post by RoelVB » Tue Feb 12, 2019 2:49 am

Hi there,

I've copied a couple of terabytes from one QNAP to another using RTRR over the LAN.
Now one of the NAS's is on a remove site, so I changed the RTRR target IP-adres. But if I start a replication it starts to sync all the files again, instead of only new files.
A replication job ran when the target NAS was still in the same LAN, that time it skipped all the files. Since nothing was changed.

These are the policy settings (I've tried different settings without any change in behavior):
997fc080-22e3-4d05-9f55-c8c9269ff09d.jpg

Can someone explain why this is happening?
You do not have the required permissions to view the files attached to this post.

P3R
Guru
Posts: 10632
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: RTRR syncing all files instead of changed

Post by P3R » Tue Feb 12, 2019 8:04 am

RTRR will always compare all files but only copy those that have changed.

The comparison done by RTRR is for filename, size and date/time (this is generally enough and pretty fast on hi-bandwidth connections). If you tick the Check file contents option it will also compare the file contents (this is slow when there's much data).

So it's not really copying the files over but reading the files to do the comparison of the contents that you told it to do.

What's the speed of the connection between the two NASes?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

RoelVB
New here
Posts: 5
Joined: Sat Jan 26, 2019 7:45 pm

Re: RTRR syncing all files instead of changed

Post by RoelVB » Thu Feb 14, 2019 2:31 am

Thanks you for your reply.

Turning of "Check file contents" does help (a little). Strange thing is, with this option checked the logs showed no skipped files. With this option disabled it does log a lot of skipped files.
I would expect this option to check file hashes and not the complete content.

Another think is. I have one folder which was previously synced using RSync and is now syncing using RTRR, this one is resyncing everything. Not really a big deal if it's only this once.

P3R
Guru
Posts: 10632
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: RTRR syncing all files instead of changed

Post by P3R » Thu Feb 14, 2019 3:56 am

RoelVB wrote:
Thu Feb 14, 2019 2:31 am
I would expect this option to check file hashes and not the complete content.
Maybe it use a hash internally to do the comparison but with RTRR the master will do the comparison of both source and destination. And how can the master compare it's own file with the file on the other unit? The answer is that it have to download the complete remote file from the other side when you've ticked the Check file contents option to be able to do the comparison.

Rsync is much more clever in this regard as each side calculate the hashes for it's files and then only the hash values are sent over to the master.

RTRR is very feature rich and offer the fantastic versioning feature but it's pretty demanding for bandwidth so unless using a LAN-like connection, the backup/sync times will be very long. Also, depending on which Qnap is the master doing the backup the huge file copying necessary when using Check file contents may travel in a different direction than the administrator expected, which can become a huge problem on assymetric connections like with xDSL and some cable connections (that usually have a high download speed but low upload). RTRR Is great on symmetric gigabit, decent on 100 Mbps but many times questionable on 10 Mbps.

It's of course depending on the data volumes and the patience of the administrator but generally speaking, Rsync is always preferable on slower site-to-site connections like 10 Mbps and less.
Another think is. I have one folder which was previously synced using RSync and is now syncing using RTRR, this one is resyncing everything. Not really a big deal if it's only this once.
What do you mean by resyncing?

Both RTRR and Rsync will always compare everything that the job is configured to compare but only new and changed files will be copied to the destination. The details on how that comparison is done differ hugely though.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

RoelVB
New here
Posts: 5
Joined: Sat Jan 26, 2019 7:45 pm

Re: RTRR syncing all files instead of changed

Post by RoelVB » Thu Feb 14, 2019 5:36 am

P3R wrote:
Thu Feb 14, 2019 3:56 am
What do you mean by resyncing?
I mean it is copying every file again instead of skipping them. But as you explained RSync and RTRR compare files differently, so that would be the "problem" here.

P3R
Guru
Posts: 10632
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: RTRR syncing all files instead of changed

Post by P3R » Thu Feb 14, 2019 5:44 am

Disabling the Check file contents option will make a huge difference but Rsync have another advantage as well. Being block based it only need to transfer the blocks that have changed, not the whole file as RTRR always do. This can make a huge difference as well with small changes in large files. Rsync is the undisputed champion of slow connection backups while RTRR have several advantages on fast LAN-like connections.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

Post Reply

Return to “Backup & Restore”