NAS to NAS or RTRR

[Steerpike]

Oh my - just made an important discovery / observation during my testing. Both the Nas To Nas (NTN, using rsync) and RTRR sync processes will OVERWRITE newer files at the destination, and there doesn't seem to be a switch to modify this behavior.

Yes they are both one-way replications unless you use RTRR in 2-way Synchronization mode.

And 2-way sync mode can only be scheduled at most hourly; no 'real time' mode - am I correct in that?

[P3R]

And 2-way sync mode can only be scheduled at most hourly; no 'real time' mode - am I correct in that?

As I said earlier in this thread, I'm not a fan of 2-way sync so I have never even used it and therefore unfortunately isn't familiar with the options.

2-way sync sounds fantastic if it works perfectly but because of the complexity and risks associated with a feature like that I prefer a different approach. I consider the site I'm mostly at the main site and do one-way synchronizations from that to the remote sites nightly. If I update a file when on the remote site I manually upload them to the main site. This of course require that all users are aware of how this works and follow that procedure.

[Spider99]

it should not have to as rtrr is a qnap protocol the remote nas should just send the hash for the other nas to check - minimal bandwidth

RTRR is only one of the transport protocols supported. There's also Ftp and CIFS/SMB so it's obviously meant to work with general non-Qnap servers as well.

What you want is already available, it's called Rsync (and NAS to NAS when both sides are Qnap).

FTp and smb are not qnap protocols and just under the rtrr banner so just lazy way to implement ftp or smb backup/sync etc - they smb if very flaky

RTRR is under qnap control - so in theory very easy to implement hash compare

[P3R]

RTRR is under qnap control - so in theory very easy to implement hash compare

RTRR is clearly an application created to be able to talk with both Qnap and other general file servers whether you like it or not. It won't change because of you moaning about it in post after post here.

What you want is already available, it's called Rsync (and NAS to NAS when both sides are Qnap).

[Spider99]

Prey tell how you connect to another machine with rtrr thats not a qnap machine - i.e. the rtrr protocol not ftp or smb

[Steerpike]

And 2-way sync mode can only be scheduled at most hourly; no 'real time' mode - am I correct in that?

As I said earlier in this thread, I'm not a fan of 2-way sync so I have never even used it and therefore unfortunately isn't familiar with the options.

2-way sync sounds fantastic if it works perfectly but because of the complexity and risks associated with a feature like that I prefer a different approach. I consider the site I'm mostly at the main site and do one-way synchronizations from that to the remote sites nightly. If I update a file when on the remote site I manually upload them to the main site. This of course require that all users are aware of how this works and follow that procedure.

Yeah, I'm not willing to rely on 2-way sync either; I decided to try it out yesterday after seeing one-way sync overwrite a newer file with an older one; I wanted to be sure 2-way at least handled that (it did). I'm a little disappointed that neither RTRR nor rsync have an option to check for newer files at the destination, but it is what it is ... now I know. My use case won't actually allow that to happen in real life, so not a big deal.

I'm currently only testing all the options for a future deployment, so I have the luxury of trying things out.

The biggest issues/observations so far are:
1) Rsync is approx 3x faster than RTRR over my slow WAN connection (5mbps)
2) RTRR has good logging while rsync does not.

The first item strongly favors rsync, but the second item strongly favors RTRR. I'm very reluctant to commit to something in production with real files when I don't have a basic log of what happened.

Given that I'm still not in production, I'm tempted to try the new hybrid backup solution. From what I've read, it sounds like they've put a ton of effort into supporting 3rd party cloud backup repositories (Amazon S3, etc), but I suspect that if you select the Nas to Nas options, it won't be doing anything different - including lack of logging perhaps. But I can spend time today playing with it.

Edit To Add: Yikes, reading about Hybrid Backup / Sync ... seems like a big pile of poo; unstable beta code released as production, then pulled, then re-released as beta ... But ... I guess I can give it a whirl ...

[kherr4377]

i use rtrr exclusively between my 2 469l's. with early 4 firmware i was getting 100+MB average transfer, can't get better than that. somewhere around 4.1 speeds slowed to 75%, they broke it. but ever since 4.2.1/4.2.2 it's back to 100%. it's still solid with 4.3.2 ....

[Steerpike]

I "Upgraded" to Hybrid Backup/Sync, and I can pretty much confirm that - when it comes to NTN (rsync) and RTRR jobs, nothing has changed. Still took 3 minutes with rsync, and 11 minutes with RTRR over the slow WAN network. Same (non-existent) logging for rsync.

Weird thing is, I can't seem to delete any job, and all jobs are now lumped into one big list, so a bit harder to keep track. But my sense is, they didn't do anything with rsync or RTRR; all the effort went into a new front end, and all the 'new stuff'.

[Steerpike]

OK, I've been testing with the new Hybrid Backup Sync app, and I would say my comments earlier still hold - it's basically a new front-end for job definition and control, but nothing has changed behind the scenes in terms of services, protocols, and logging (or lack thereof). My RTRR jobs are all working fine. But today I tried to set up a new Rsync job, and could not get past the basic 'test' phase of setting up the job, despite double-checking all settings. In desperation, I uninstalled Hybrid Backup Sync (which immediately revealed the good old 'Backup Station' again) and without even a restart, I was able to quickly configure a new Nas-To-NAS (rsync) job, same settings. So I'm currently giving up on Hybrid Backup Sync.

Is there a specific forum area to discuss this 'beta' product?

[njarvisto]

is there any other solution other than these? both are heinously slow when it comes to a file sync between two sites - GoodSync or similar is significantly faster but im wanting an unattended option without having PC etc on.
eg, one of my jobs through RTRR takes ~2 hours, vs 5 minutes via GoodSync. This is a big enough difference to take note of for me.

[Spider99]

without and information cant comment - sounds like very different setups to have that much difference

[Steerpike]

is there any other solution other than these? both are heinously slow when it comes to a file sync between two sites - GoodSync or similar is significantly faster but im wanting an unattended option without having PC etc on.
eg, one of my jobs through RTRR takes ~2 hours, vs 5 minutes via GoodSync. This is a big enough difference to take note of for me.

That makes little sense. I've done a good deal of testing with RTRR, and I'm getting close to 60% of maximum theoretical bandwidth on some large files. That is - my max upload bandwidth at one site is about 6 Mbps, and I can 'sync' two or three large files and get a throughput of about 4Mbps. Considering that the entire structure I'm replicating has thousands of files, all of which have to be checked, that's not too bad. There's no way anything could do the same job much faster, given the bandwidth available.

In your case, what is your bandwidth (upload, not download) from the source site? How many files total, and how many changed for your test? RTRR is file-level, not block-level, so if a file is changed at all the entire file is going to be transmitted. Maybe GoodSync is block level, and your file is a database, with only a few blocks changed? Try deleting the entire file and then syncing - in that case, the entire file 'must' be copied, and in that case, the time taken should be somewhat related to your bandwidth plus overhead.

[njarvisto]

I'll send some screen grabs when back in front of PC - the most painful part is that this is the smallest job I'm looking to complete.
3x folders,
404mb with 1452 files
128mb with 320 files
40mb with 107 files

No DB files, PDF/DOC/XLS/RTF mainly

[Spider99]

do you have compression on and /or check file contents on the rtrr?

[markopas]

How's it going everyone - I have been seriously enjoying reading this thread and the wealth of information in it.

I had a few questions I hope that a few of you could chime in on.

I recently purchased a TVS-682T (i3 w/16GB) and have 4 - 10TB 3.5" Seagate NAS drives and 1 - 1TB m.2 SSD which I use for SSD caching and have the NAS setup as a RAID5.

I currently have a two Cat6a network cables the unit came with plugged in to the 10GbE ports and then directly plugged in to the back of my cable modem/router (Brand new DOCSIS 3.1 router/modem from my ISP all on Gb connections using Cat6 cabling). I also have my other (POS) D-Link DNS-343 - 4 x 4TB as my primary storage which I will either sell after all my content has been transferred from it or use it as a backup of sorts also plugged in to the modem with Cat6 cable. I have been playing around with all the settings on QNAP to get acquainted with it and was using the 4.3.x beta firmware which I think is fantastic but today decided to go back and try the 4.2 f/w as there are still more supported apps that are not yet available on 4.3 beta.

I have been having serious issues trying to figure out how to remote backup from the DNS-343 => TVS-682T but all the options that that I go through are pretty much useless as they are very basic and the instructions make no sense. It seems as though that the remote backup feature (which they say is RSYNC) but I think was only designed for other D-LINK DNS-343 units (or newer since this product is EoL). I have seriously been racking my brain on this, I can only find threads on how to backup from DNS-343 <=> DNS-343 but nothing else. I have been looking through the Hybrid Backup App features which I think is decent but still buggy, so I removed it and went back to the pre-installed backup tool.

The only feature that pertains so me was the one way sync option going from remote to local, everything else pertaining to backup features are local to remote. So using the RTRR with either FTP or CIFS works but I initially ran a test (the button to test the connection) and only came back finding out that I was only getting 12MBps but actualy transfer speeds was only giving me ~5MBps which made no sense and when I setup a test job using CIFS i was only getting in the neighbourhood of 3MBps which was even worse. I recently made a change and thought I would give it a shot and turned on jumbo frames to 9000 on both the old and new NAS. I then tested out the FTP test feature and it came back with 65MBps (what a difference) but when I ran the job, it only was doing about 23-28MBps which I found even more weird but was still a significant increase compared to before. I then cancelled the job and then decided to run it as a CIFS restore and the test button doesn't tell you how quick the job is going to be other than it saying either SUCCESSFUL or FAILED. I then processed the same folder for the job and was able to get anywhere between 40-55MBps which made me super happy but still does not explain why I am only getting 40-55MBps on a 1Gbps connection when I should be getting somewhere closer to ~100MBps.

Is it possible that this is the limitation of array from the D-Link?

After reading Spider99's post, I connected to the QNAP and ran the two tests.

Vol_ID Vol_Name Pool_ID Mapping_Name Throughput Mount_Path Throughput
1 DataVol1 288 /dev/mapper/cachedev1 499.43 MB/s /share/CACHEDEV1_DATA 516.13 MB/s

Enclosure Port Sys_Name Throughput RAID RAID_Type RAID_Throughput Pool
NAS_HOST 1 /dev/sda 536.39 MB/s /dev/md2 Single 538.23 MB/s 256
NAS_HOST 5 /dev/sdb 226.87 MB/s /dev/md1 RAID 5 658.45 MB/s 288
NAS_HOST 6 /dev/sdc 232.49 MB/s /dev/md1 RAID 5 658.45 MB/s 288
NAS_HOST 7 /dev/sde 228.82 MB/s /dev/md1 RAID 5 658.45 MB/s 288
NAS_HOST 8 /dev/sdd 227.74 MB/s /dev/md1 RAID 5 658.45 MB/s 288

Can anyone help provide any thoughts on other things I could try to help with performance loss I am experiencing.

Regards,
Marko