NAS to NAS or RTRR

[P3R]

I have yet to explore the impact of 'check file contents' on the performance of the sync job. I am expecting it to render the sync unusable in my situation, as I only have ~5Mbps upload speed...

Most likely yes. Your application sounds like a perfect match for NAS to NAS, as that's based on rsync.

I agree that 'check contents' would have caught this corruption, but ironically it would have been caught even without it (I believe), since the corruption rendered the file smaller (I understand that, without 'check contents', it will rely on name/date/size).

Yes if smaller it should be caught. But I think that you said the you did run RTRR it and that it wasn't caught? Perhaps the file corruption happened later?

I can imagine some efficient checksum calculations though ...

You mean like NAS to NAS do?

[Steerpike]

Your application sounds like a perfect match for NAS to NAS, as that's based on rsync.
...

I started out testing NAS-NAS, when both my NAS's were on the same LAN, but moved on to RTRR for various reasons - possibly in part because posters here suggested it was going away in the next release? Is it fair to say, even if NAS-NAS goes away, the more generic 'rsync' option will remain, and will perform not much differently from nas-nas?

Yes if smaller it should be caught. But I think that you said the you did run RTRR it and that it wasn't caught? Perhaps the file corruption happened later?

No, sadly, I 'corrected' the problem immediately by re-copying from the source (laptop). That was a mistake from a troubleshooting perspective.

[Moogle Stiltzkin]

There is a check box in the RTRR job 'options' screen to enable detailed logs. On the RTRR screen, at the very top (not specific to any one job), click on the options button and then choose to enable detailed logs. I set that when I first started working with the QNAP so I can't say what a 'non-detailed' job log looks like, but with this checked, you get a log entry for every single file sync'd by the job. I also found the log file at /etc/logs/qsync/detail. Sadly it gets overwritten each time the job runs, so I have a script to rename it and move it to another location.

In my case, I was getting a lot more files sync'd than I expected but the log revealed it was due to the contents of @recycle being included; I've subsequently disabled that setting.

this one? yeah i had it enabled when i ran the rtrr tasks

rtrr p37.jpg

ooo it's at that location? Why didn't they tell me then where it was :[ i was looking through the whole UI and did not find it. Is that an oversight for not integrating the logs into the UI for quick access and browsing? Also i agree that deleting after each rtrr performed is not something i like either; maybe give us a choice to retain old rtrr detailed logs or not.


that aside, with "verify contents" doesn't seem it impacts performance if the destination is empty of any data to begin with cause i'm maxing out my transfer speed locally.

[P3R]

I started out testing NAS-NAS, when both my NAS's were on the same LAN, but moved on to RTRR...

Well RTRR is in most cases better on a LAN. Rsync is the undisputed champion on slow connections, like 5 Mbit/sec and below.

...for various reasons - possibly in part because posters here suggested it was going away in the next release?

If I understand things correctly all current backup options will be replaced by Hybrid Backup Sync in 4.3 but why worry about that now? Use what works best of the currently available options is my advice.

Is it fair to say, even if NAS-NAS goes away, the more generic 'rsync' option will remain, and will perform not much differently from nas-nas?

I'm sorry, the only Qnaps I currently have is in production and Hybrid Backup Sync is still only in beta so I haven't tried it.

I would however be very surprised and consider it a huge mistake by Qnap if there is no rsync-based option in Hybrid Backup Sync as there are still many slow internet connections in use. I have one remote location myself that's on a 4/1 Mbit/sec DSL-connection. I'm have hopes for fiber there as well but it's at least one year away.

[Spider99]

i would hope its a checksum hash check!

[Steerpike]

Well I tried 'check file contents' on a test sync of a large structure with a few files deleted (one that completed in about 10 minutes without it) and ... after 2+ hours ... I gave up. So check file contents just doesn't cut it on a slow connection, for sure.

I'm now back to testing with Nas-to-Nas, trying to compare to RTRR over my slow WAN connection. But I can't find any log file anywhere; I'm looking in /etc/logs/rsync, /etc/logs/qsnapsync/brief, /etc/logs/qsnapsync/detail - all empty. There's even a button right there in the UI for the Nas to Nas job - Rsync log - but there's nothing 'there'.

NTNjob.jpg

Without being able to view the log, it's a lot harder to do the analysis.

Edit to add:
I ran the Nas to Nas job for the same set of files (deleted one directory of about 150 files, 124 MB, out of a much larger folder structure that remained otherwise unchanged) and it took 3 minutes with Nas to Nas, compared to 11 minutes with RTRR.

So that would suggest Nas to Nas definitely has the edge on slower connections.

Regarding logging for Nas to Nas - I found my notes from earlier; they say to add

Code: Select all

log file = /var/log/rsyncd.log
max verbosity = 3

to /etc/config/rsyncd.conf , but doing so had no effect.

[Moogle Stiltzkin]

found it where you said it was.

however looking at it, the filenames that popped up were things i'm 100% sure was already at destination, so they ought to have been skipped over.

Considering that during the process it mentions files skipped and files backed up, i'm assuming that it is possible to skip file files if it already existed at the destination. identical filename at the same pathing folders/sub folders would have sufficed even with verify contents disabled wouldn't it ? But now i'm not so sure :X

Side note wish they could integrate these logs into UI and add a button for download logs as txt or csv format. cauz i had to use winscp to browse to the location to find it

rtrr p38.jpg

[P3R]

Side note wish they could integrate these logs into UI and add a button for download logs as txt or csv format. cauz i had to use winscp to browse to the location to find it

If you mean like this it's there already:

RTRR_logs.jpg

[P3R]

i would hope its a checksum hash check!

But to do the hash calculation on the remote data the NAS doing the backup needs to get the complete file...

[Spider99]

it should not have to as rtrr is a qnap protocol the remote nas should just send the hash for the other nas to check - minimal bandwidth

[Moogle Stiltzkin]

Side note wish they could integrate these logs into UI and add a button for download logs as txt or csv format. cauz i had to use winscp to browse to the location to find it

If you mean like this it's there already:

RTRR_logs.jpg

oops my bad, yep its the right log

[P3R]

it should not have to as rtrr is a qnap protocol the remote nas should just send the hash for the other nas to check - minimal bandwidth

RTRR is only one of the transport protocols supported. There's also Ftp and CIFS/SMB so it's obviously meant to work with general non-Qnap servers as well.

What you want is already available, it's called Rsync (and NAS to NAS when both sides are Qnap).

[Steerpike]

Side note wish they could integrate these logs into UI and add a button for download logs as txt or csv format. cauz i had to use winscp to browse to the location to find it

If you mean like this it's there already:

RTRR_logs.jpg

That button to download the log, for RTRR, is great, but it can only access the latest log. So if you run RTRR every hour, and a major change happened 4 hours ago, you have no visibility into that. I've somewhat addressed that with this script I've concocted;

Code: Select all

cp -u -v /etc/logs/qsync/detail/qsync-Job0{.log,-AZ-"$(date -r /etc/logs/qsync/detail/qsync-Job0.log +%Y%m%d-%H%M%S)".log}

it copies the qsync-Job0.log to qsync-Job0-AZ-(YYMMDD-HHMMSS).log, thus keeping a copy even when qsync-Job0 gets overwritten. I currently have it run from the crontab, on the same schedule as my RTRR job, but that's problematic. I would prefer to add it to the actual SH script file that runs the RTRR job itself, if possible, so I can be sure to run it only when the job has completed.

So does anyone have any insight into the log location for the Nas-to-Nas (NTN) job? Clearly, for my situation, NTN is 3x faster so I want to pursue it but I really want to be able to review logs if I'm going to commit to it as my production solution. I posted above about the entry "log file = (path/logname)" in the /etc/config/rsyncd.conf file, but that didn't seem to help.

[Steerpike]

Oh my - just made an important discovery / observation during my testing. Both the Nas To Nas (NTN, using rsync) and RTRR sync processes will OVERWRITE newer files at the destination, and there doesn't seem to be a switch to modify this behavior.

I have two locations - CA and AZ. in CA, I updated file 'x'. last modified date went from 12/5 to 12/10 as expected. I then ran NTN sync from AZ to CA, and found that the file 'x' was reset back to it's 12/5 state. I then re-edited the file x, thus resetting last modified date to 12/10, and ran the RTRR sync from AZ to CA, and exactly the same behavior was observed.

In the 'filter' section for both job types there is no setting for 'copy only newer versions ...'. It seems like such an obvious option. Oh well.

I then setup a two-way RTRR sync, same folder pair, and found that that does behave more as expected - newer files get copied in either direction. but I also discovered that two-way RTRR has no 'real time' option; you just have to schedule it. and hourly seems to be the most frequent.

[P3R]

Oh my - just made an important discovery / observation during my testing. Both the Nas To Nas (NTN, using rsync) and RTRR sync processes will OVERWRITE newer files at the destination, and there doesn't seem to be a switch to modify this behavior.

Yes they are both one-way replications unless you use RTRR in 2-way Synchronization mode.