NAS to NAS or RTRR

Discussion on remote replication.
Locked
User avatar
Moogle Stiltzkin
Guru
Posts: 11445
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: NAS to NAS or RTRR

Post by Moogle Stiltzkin »

okay tested.

I want to backup storage2 source to storage2 destination.

But the result in destination was storage2 > the task name > storage2 > latest > then all the contents from storage 2....

Whereas with NAS to NAS using the same source and destination settings, i get storage 2 > then all the contents from storage 2.


How do i get the backup pathing same like NAS to NAS >->; because some of my files were already quite long pathing so adding 2-3 extra sub directories is not gonna work out for me :(

bad... (pathing as result of rtrr)
rtrr p18.jpg

good (pathing as result of nas to nas)
rtrr p19.jpg
rtrr p20.jpg

any ideas :[
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
P3R
Guru
Posts: 13192
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: NAS to NAS or RTRR

Post by P3R »

Moogle Stiltzkin wrote:How do i get the backup pathing same like NAS to NAS >->; because some of my files were already quite long pathing so adding 2-3 extra sub directories is not gonna work out for me :(
Use Synchronize instead of Backup in RTRR.

There is absolutely no need to obfuscate your RFC1918 private address information. It's useless to everyone outside of your network as they can't be reached over the internet and impossible for you to hide for anyone inside your network.

It's your public ip address (usually on the WAN interface of your router/firewall) that you shouldn't reveal to anyone.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
User avatar
Moogle Stiltzkin
Guru
Posts: 11445
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: NAS to NAS or RTRR

Post by Moogle Stiltzkin »

P3R wrote:Use Synchronize instead of Backup in RTRR.
You mean this?
rtrr p21.jpg

local folder to remote folder same like backup, just that now the main header if syncronization mode? :)



Anyhow i did as you suggested and maybe it's working as i wanted it to. Here are the steps i followed and to verify it's working
rtrr p22.jpg
rtrr p23.jpg
rtrr p24.jpg
rtrr p25.jpg
rtrr p26.jpg
rtrr p27.jpg
rtrr p28.jpg
Few things though. Because i had already previously done the backup using NAS to NAS, the destination already has the files. So i'm just running RRsync to test the feature works.

But based on configuration it will check if data is same or not, and to delete extra files. Meaning it ought to be a 1:1 copy from destination to folder. Anything extra in destination not found on source will be deleted. I assumed this did not include deleting the exact file already there on destination, seeing there is already a policy to first check if it's same or not ?

Cause why is it still take 35 hours just to complete this task if all is needed to do is first is to verify if file exist, if it does and no changes is needed, then there should be no need to replace the file. Or is my settings wrong ? :S
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
P3R
Guru
Posts: 13192
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: NAS to NAS or RTRR

Post by P3R »

The option Check file contents will do exactly that. It will compare the data in the file on the source with the one on the destination to determine if there are any differences. Checking the contents of files in both source and destination will take a long time if it's much data.

If you disable the Check file contents option RTRR will still check size, date and time to determine if the files are the same and that will be much faster.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
User avatar
Moogle Stiltzkin
Guru
Posts: 11445
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: NAS to NAS or RTRR

Post by Moogle Stiltzkin »

P3R wrote:The option Check file contents will do exactly that. It will compare the data in the file on the source with the one on the destination to determine if there are any differences. Checking the contents of files in both source and destination will take a long time if it's much data.

If you disable the Check file contents option RTRR will still check size, date and time to determine if the files are the same and that will be much faster.
thx. yeah i did suspect it takes longer (cauz i do hash checks all the time), but i didn't think it would be this long... oh well.. since it's a beta test on my part, i'll do it using check file contents first. Then i'll test the other method later with it disabled.

:)


The qts help was very brief was rtrr was. Had to use the online resource to find the detailed guide for 4.3.x manual

https://www.qnap.com/en/support/con_show.php?cid=11

http://files.qnap.com/manualpdf/4.3/cat ... SMB_en.zip


ctrl+f to find "rtrr"
rtrr p29.jpg

Didn't see mention about difference between backup and sync mode for the same source folder to destination had a different output :S guess they need to add that into the manual kek. Or was the first bullet point explaining exactly that? :) Anyway problem solved for me.


Just a recap for other readers testing rtrr

Solution

source share: storage2

then create destination a share also storage2.

RTRR, Sync mode, local folder to remote folder, storage2 (source) > storage2 (destination)

result = contents of storage 2 is placed in destination storage2, so you don't get storage2/storage2. For anyone else who needs to do this but did not know :)


alternate output

But what happens if use backup mode, but also local folder to remote folder, storage2 (source) > storage2 (destination) ?

apparently the destination output will be storage2/task name/latest/ (the contents of storage 2)

The manual does not tell you this :)
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
Steerpike
Starting out
Posts: 43
Joined: Tue Nov 22, 2016 11:39 pm
Location: Bay Area, CA

Re: NAS to NAS or RTRR

Post by Steerpike »

P3R wrote: ...
RTRR have been constantly developed and have more features and advantages than NAS to NAS:
  • Both real time and scheduled modes
  • Two way synchronization (I'm not a fan and don't know how well it handles conflicts but it's available for the many users asking for it)
  • Choice of protocols; RTRR, FTP, CIFS/Samba
  • Versioning (this alone make it a killer app!)
  • Best performance for bulk copying on fast, LAN-like, connections (disable; Check file contents, Compress files during transmission, Delete extra files)
...
Can you elaborate on the 'versioning' ... I must have missed that one!

Also - what's the general opinion of error correction / detection / reliability of RTRR? I did the following:
  1. copy a large directory structure comprised mainly of images (jpegs, TIFF, and raw) from laptop hard drive to NAS-1
  2. use RTRR to replicate the directory structure to an identical structure on NAS-2
  3. compare the laptop directory structure on the laptop (source in step 1 above) to the directory structure on NAS-2 using 'beyond compare' (something I've used for over 10 years and trust).
One jpg image file on NAS-2 failed the compare, and it failed to open properly in a jpeg viewer. Unfortunately, I corrected the problem (copied from laptop to NAS-2) before realizing I should have saved it for further analysis. Anyway, it occurred to me that corruption could occur during the RTRR process, and got me thinking about what error detection and correction may be built into the process.

I'll do a bunch more testing and see what I can learn.
P3R
Guru
Posts: 13192
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: NAS to NAS or RTRR

Post by P3R »

Steerpike wrote:Can you elaborate on the 'versioning' ...
Versioning is only relevant for backups, not when synchronizing two sites that I believe is your application.

It's simply that in addition to the latest backup you also keep one or more versions of all the files that over time have changed or been deleted. It's like having historical "snapshots" of your data structure.

Think about the case where a ransomware encrypt data and before you realize that you your backup, without versioning, runs. Your only healthy files in the backup will be overwritten by all the encrypted files...

Or

When you realize that you once again need a file you deleted two months ago. Without versioning you're lost but with it you just go to the backup version that's slightly older than two months.

Backup versioning is possible to do manually with rotating backup destinations.

The beauty of having versioning integrated in the backup application is however that (if it's done properly) it's easy to manage the number and age of available versions and that old versions doesn't take up any more storage space than only the changed/deleted files themselves.
I must have missed that one!
I don't blame you. It's been a pretty well kept secret for 18 months or so.

Versioning in RTRR is for some odd reason only available by installing the virtually undocumented Backup versioning app.

The Backup versioning app is officially still in beta but I've been using it from the beginning and on my actual production data for about 51 weeks now. It's been steady as a rock.

The reason RTRR versioning isn't out of beta, and most likely never will be, is that in QTS 4.3 there's a new backup solution that will replace NAS to NAS and RTRR. QTS 4.3 is also in beta and that I wouldn't use on my production unit until it's being released as stable software.
Also - what's the general opinion of error correction / detection / reliability of RTRR?
I have no opinion as I've never seen any documentation from Qnap on how it really works under the hood. Therefore I can only judge by my experience of the only case I've been using it extensively in. That's for backups to external disks connected locally to the NAS and I've noticed no issues with it. But I also haven't checked the data deeply, only random samples here and there when verifying my backups.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
Steerpike
Starting out
Posts: 43
Joined: Tue Nov 22, 2016 11:39 pm
Location: Bay Area, CA

Re: NAS to NAS or RTRR

Post by Steerpike »

I re-copied (re-sync'd) the entire directory in question (this time, over the WAN taking several hours rather than minutes) and could not replicate the problem; so it would appear that one jpeg file got corrupted during the initial RTRR sync between my two NAS boxes. I will continue to validate my NAS backups (syncs) by comparing to the original files periodically, and see how it goes over time. If only 1 file out of thousands gets corrupted, and only occasionally, I can accept that.
User avatar
Moogle Stiltzkin
Guru
Posts: 11445
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: NAS to NAS or RTRR

Post by Moogle Stiltzkin »

Steerpike wrote:I re-copied (re-sync'd) the entire directory in question (this time, over the WAN taking several hours rather than minutes) and could not replicate the problem; so it would appear that one jpeg file got corrupted during the initial RTRR sync between my two NAS boxes. I will continue to validate my NAS backups (syncs) by comparing to the original files periodically, and see how it goes over time. If only 1 file out of thousands gets corrupted, and only occasionally, I can accept that.
huh? how did it get corrupted :S what were your settings.

as far as i know it could only get corrupted if it the source file already was corrupted to begin with, or you had a powercut or shutdown mid rtrr task perhaps ?

But didn't expect there to be any errors from just performing the rtrr task :'


and how did you manage to figure out from those thousands of files which got corrupted? :shock:
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
Steerpike
Starting out
Posts: 43
Joined: Tue Nov 22, 2016 11:39 pm
Location: Bay Area, CA

Re: NAS to NAS or RTRR

Post by Steerpike »

Moogle Stiltzkin wrote:
Steerpike wrote:I re-copied (re-sync'd) the entire directory in question (this time, over the WAN taking several hours rather than minutes) and could not replicate the problem; so it would appear that one jpeg file got corrupted during the initial RTRR sync between my two NAS boxes. I will continue to validate my NAS backups (syncs) by comparing to the original files periodically, and see how it goes over time. If only 1 file out of thousands gets corrupted, and only occasionally, I can accept that.
huh? how did it get corrupted :S what were your settings.

as far as i know it could only get corrupted if it the source file already was corrupted to begin with, or you had a powercut or shutdown mid rtrr task perhaps ?

But didn't expect there to be any errors from just performing the rtrr task :'

and how did you manage to figure out from those thousands of files which got corrupted? :shock:
Well, that's the big question - how did the corruption occur? Hence, my asking about error detection and correction.

As for my settings - they are all defaults for the RTRR job. I have since disabled the syncing of 'temp' files / recycle folder, but that should not have any bearing.

As I mentioned, I copied (windows file copy from laptop to SMB share on NAS) all the files to NAS-1; I then used RTRR to sync those files to NAS-2 over my LAN; I then relocated NAS-2 to my office in CA, and reconfigured the RTRR job to work with this new address (but no actual files were transferred after this since everything was 'seeded' already); then used 'Beyond compare' to validate the files on NAS-2 against the original files on the laptop. Beyond Compare compares two directory structures and highlights differences. Only 1 file out the the thousands involved was flagged as different.

I deleted the directory containing the corrupt file on NAS-2, and re-sync'd it from NAS-1 (this time over the WAN); all files were good at this point. So that tells me the 'source' of the original RTRR - NAS-1 - was good.

So something must have happened during the original RTRR process to cause that one file to be corrupted. This is not terribly shocking to me, and is the reason I'm doing these tests - to determine if I'm really willing to commit to this as my backup strategy.
User avatar
Moogle Stiltzkin
Guru
Posts: 11445
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: NAS to NAS or RTRR

Post by Moogle Stiltzkin »

you mean this?

http://www.scootersoftware.com/features.php


but if thats true i hope they fix it :S
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
P3R
Guru
Posts: 13192
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: NAS to NAS or RTRR

Post by P3R »

Steerpike wrote:I then relocated NAS-2 to my office in CA, and reconfigured the RTRR job to work with this new address (but no actual files were transferred after this since everything was 'seeded' already);...
Which tells us that you didn't have the option Check file contents ticked. Because had you used that, RTRR should have found the difference and automatically updated the file for you.

Please note that the above isn't criticism, just an observation that Check file contents would automatically have corrected the corrupt file. However depending on the available bandwidth, time and volume of data, Check file contents may not be practically usable in all applications.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
User avatar
Moogle Stiltzkin
Guru
Posts: 11445
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: NAS to NAS or RTRR

Post by Moogle Stiltzkin »

P3R wrote:
Steerpike wrote:I then relocated NAS-2 to my office in CA, and reconfigured the RTRR job to work with this new address (but no actual files were transferred after this since everything was 'seeded' already);...
Which tells us that you didn't have the option Check file contents ticked. Because had you used that, RTRR should have found the difference and automatically updated the file for you.

Please note that the above isn't criticism, just an observation that Check file contents would automatically have corrected the corrupt file. However depending on the available bandwidth, time and volume of data, Check file contents may not be practically usable in all applications.
i was actually planning to do a factory reset sometime soon once a more stable firmware comes.

so if source had already previous backed up to destination, but later i accumulates just a few new files i wanted to rtrr. Would i need to enable check file contents? Or would leaving that disabled be fine for my purposes ?

Because previously i did in this order

1. NAS to NAS, source to destination
2. RTRR, source to destination, with verify file enabled (i'm assuming all it did was check if file the same, if it is, don't transfer? that said it still took a long time to complete :S )

So if source had a bunch of new files (not replacements for existing ones that are already on destination), would it be okay to leave verify files disabled ? :S
rtrr p30.jpg

on side note, you can't just simply rtrr now, without setting it into scheduled or real time mode. In disabled it's completely disabled. Even the icon for run task refuses to run. But there is a run now toggle from the pull down :/ Looking further apparently have to schedule and run now. guess they err on the side of caution by not simply allowing you to click run task if it's set to disabled in here.
rtrr p31.jpg
rtrr p32.jpg

but anyway what i'm trying to get at is, since the files i had prior added already was verified, i rather avoid verifying them again, as their file name that it's already at destination would already suffice if only so it knows not to delete them or add a copy of the file at destination based on folder/file name alone. Then only the new file transferred would verify it's contents to ensure it transfers over without issues. Or is the transfer process already solid in not coughing up a corrupt file on the other end aka destination with verify setting disabled? :shock:




by the way got rtrr running now with verify disabled. I also disabled ssh seeing as it's done locally and it's not port forwarded over to the internet anyway. Anyway performance is excellent at 100 MB/s on average. It was probably either encrypted transfer or the verify contents which tanked the performance in prior backups.

but what concerns me is the roughly 1000+ backed up files. Because the only new additional files i added were not that many :S ... Also is this a bug that time left and time remaining to match ? :? synchronizing also not the same
rtrr p33.jpg

Would it be possible for the job log for after rtrr is completed to show just which files were backed up (as in not skipped) ? cause right all i know is it skipped some, but backed up a 1000+ things of which i'm not sure what :S gonna be hard to self verify seeing as my contents have so many sub folders and pathings like a maze :shock:


*update

RTRR completed in 8 hours by not verifying content and encryption transfer disabled. Also data already previously already been RTRR to the destination.

anyway it says 2tb roughly was updated, but the total size for the data rtrr was 6tb. I was only expecting the updated to be less than 10gb, not 2tb... as there was only a few additional files added recently. Trying to check logs did nothing because all it told me during the rtrr was number of files skipped and files backed up, and the total amount of the whole dataset being processed. When the whole RTRR completed the logs was very basic and doesn't tell you indepth which folders/sub folders/files exactly were newly successfully backed up (and not skipped due to an existing file already at destination). So i'll need to go filestation for destination server, to delete and then run rtrr again but using "verify contents". i did a rough check and did not see any duplicates or errors, but i need some certainty that it copied exactly... hence why i need to rtrr again from scratch this time :(
rtrr p35.jpg

For future subsequent rtrr i'll have to use verify contents :ashamed:



*update

doing rtrr from scratch with verify contents enabled. but encryption disabled (since doing over local network so doubt i need it). 101 MB/S ETA 16 or so hours to backup 6.6 TB worth of data, very nice :)
rtrr p36.jpg
Last edited by Moogle Stiltzkin on Sat Dec 10, 2016 2:03 am, edited 1 time in total.
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
Steerpike
Starting out
Posts: 43
Joined: Tue Nov 22, 2016 11:39 pm
Location: Bay Area, CA

Re: NAS to NAS or RTRR

Post by Steerpike »

Moogle Stiltzkin wrote:....
Would it be possible for the job log for after rtrr is completed to show just which files were backed up (as in not skipped) ? cause right all i know is it skipped some, but backed up a 1000+ things of which i'm not sure what :S gonna be hard to self verify seeing as my contents have so many sub folders and pathings like a maze :shock:

...
There is a check box in the RTRR job 'options' screen to enable detailed logs. On the RTRR screen, at the very top (not specific to any one job), click on the options button and then choose to enable detailed logs. I set that when I first started working with the QNAP so I can't say what a 'non-detailed' job log looks like, but with this checked, you get a log entry for every single file sync'd by the job. I also found the log file at /etc/logs/qsync/detail. Sadly it gets overwritten each time the job runs, so I have a script to rename it and move it to another location.

In my case, I was getting a lot more files sync'd than I expected but the log revealed it was due to the contents of @recycle being included; I've subsequently disabled that setting.
Steerpike
Starting out
Posts: 43
Joined: Tue Nov 22, 2016 11:39 pm
Location: Bay Area, CA

Re: NAS to NAS or RTRR

Post by Steerpike »

P3R wrote:
Steerpike wrote:I then relocated NAS-2 to my office in CA, and reconfigured the RTRR job to work with this new address (but no actual files were transferred after this since everything was 'seeded' already);...
Which tells us that you didn't have the option Check file contents ticked. Because had you used that, RTRR should have found the difference and automatically updated the file for you.

Please note that the above isn't criticism, just an observation that Check file contents would automatically have corrected the corrupt file. However depending on the available bandwidth, time and volume of data, Check file contents may not be practically usable in all applications.
I have yet to explore the impact of 'check file contents' on the performance of the sync job. I am expecting it to render the sync unusable in my situation, as I only have ~5Mbps upload speed and I have hundreds of thousands of files, but - I will certainly give it a try.

I agree that 'check contents' would have caught this corruption, but ironically it would have been caught even without it (I believe), since the corruption rendered the file smaller (I understand that, without 'check contents', it will rely on name/date/size).

Let's hope they implement 'check contents' in some optimized fashion; simplistically, if they just do a compare, then they would have to first copy the file to the target side, then compare, and discard if not different - which means every sync would be a full copy of all files. I can imagine some efficient checksum calculations though ... I guess I need to try it and observe!
Locked

Return to “Remote Replication/ Disaster Recovery”