NAS to NAS or RTRR
- Moogle Stiltzkin
- Guru
- Posts: 11445
- Joined: Thu Dec 04, 2008 12:21 am
- Location: Around the world....
- Contact:
Re: NAS to NAS or RTRR
okay tested.
I want to backup storage2 source to storage2 destination.
But the result in destination was storage2 > the task name > storage2 > latest > then all the contents from storage 2....
Whereas with NAS to NAS using the same source and destination settings, i get storage 2 > then all the contents from storage 2.
How do i get the backup pathing same like NAS to NAS >->; because some of my files were already quite long pathing so adding 2-3 extra sub directories is not gonna work out for me
bad... (pathing as result of rtrr)
good (pathing as result of nas to nas)
any ideas :[
I want to backup storage2 source to storage2 destination.
But the result in destination was storage2 > the task name > storage2 > latest > then all the contents from storage 2....
Whereas with NAS to NAS using the same source and destination settings, i get storage 2 > then all the contents from storage 2.
How do i get the backup pathing same like NAS to NAS >->; because some of my files were already quite long pathing so adding 2-3 extra sub directories is not gonna work out for me
bad... (pathing as result of rtrr)
good (pathing as result of nas to nas)
any ideas :[
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1
Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)
Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1
Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)
Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
-
- Guru
- Posts: 13192
- Joined: Sat Dec 29, 2007 1:39 am
- Location: Stockholm, Sweden (UTC+01:00)
Re: NAS to NAS or RTRR
Use Synchronize instead of Backup in RTRR.Moogle Stiltzkin wrote:How do i get the backup pathing same like NAS to NAS >->; because some of my files were already quite long pathing so adding 2-3 extra sub directories is not gonna work out for me
There is absolutely no need to obfuscate your RFC1918 private address information. It's useless to everyone outside of your network as they can't be reached over the internet and impossible for you to hide for anyone inside your network.
It's your public ip address (usually on the WAN interface of your router/firewall) that you shouldn't reveal to anyone.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
- Moogle Stiltzkin
- Guru
- Posts: 11445
- Joined: Thu Dec 04, 2008 12:21 am
- Location: Around the world....
- Contact:
Re: NAS to NAS or RTRR
You mean this?P3R wrote:Use Synchronize instead of Backup in RTRR.
local folder to remote folder same like backup, just that now the main header if syncronization mode?
Anyhow i did as you suggested and maybe it's working as i wanted it to. Here are the steps i followed and to verify it's working
Few things though. Because i had already previously done the backup using NAS to NAS, the destination already has the files. So i'm just running RRsync to test the feature works.
But based on configuration it will check if data is same or not, and to delete extra files. Meaning it ought to be a 1:1 copy from destination to folder. Anything extra in destination not found on source will be deleted. I assumed this did not include deleting the exact file already there on destination, seeing there is already a policy to first check if it's same or not ?
Cause why is it still take 35 hours just to complete this task if all is needed to do is first is to verify if file exist, if it does and no changes is needed, then there should be no need to replace the file. Or is my settings wrong ?
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1
Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)
Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1
Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)
Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
-
- Guru
- Posts: 13192
- Joined: Sat Dec 29, 2007 1:39 am
- Location: Stockholm, Sweden (UTC+01:00)
Re: NAS to NAS or RTRR
The option Check file contents will do exactly that. It will compare the data in the file on the source with the one on the destination to determine if there are any differences. Checking the contents of files in both source and destination will take a long time if it's much data.
If you disable the Check file contents option RTRR will still check size, date and time to determine if the files are the same and that will be much faster.
If you disable the Check file contents option RTRR will still check size, date and time to determine if the files are the same and that will be much faster.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
- Moogle Stiltzkin
- Guru
- Posts: 11445
- Joined: Thu Dec 04, 2008 12:21 am
- Location: Around the world....
- Contact:
Re: NAS to NAS or RTRR
thx. yeah i did suspect it takes longer (cauz i do hash checks all the time), but i didn't think it would be this long... oh well.. since it's a beta test on my part, i'll do it using check file contents first. Then i'll test the other method later with it disabled.P3R wrote:The option Check file contents will do exactly that. It will compare the data in the file on the source with the one on the destination to determine if there are any differences. Checking the contents of files in both source and destination will take a long time if it's much data.
If you disable the Check file contents option RTRR will still check size, date and time to determine if the files are the same and that will be much faster.
The qts help was very brief was rtrr was. Had to use the online resource to find the detailed guide for 4.3.x manual
https://www.qnap.com/en/support/con_show.php?cid=11
http://files.qnap.com/manualpdf/4.3/cat ... SMB_en.zip
ctrl+f to find "rtrr"
Didn't see mention about difference between backup and sync mode for the same source folder to destination had a different output guess they need to add that into the manual kek. Or was the first bullet point explaining exactly that? Anyway problem solved for me.
Just a recap for other readers testing rtrr
Solution
source share: storage2
then create destination a share also storage2.
RTRR, Sync mode, local folder to remote folder, storage2 (source) > storage2 (destination)
result = contents of storage 2 is placed in destination storage2, so you don't get storage2/storage2. For anyone else who needs to do this but did not know
alternate output
But what happens if use backup mode, but also local folder to remote folder, storage2 (source) > storage2 (destination) ?
apparently the destination output will be storage2/task name/latest/ (the contents of storage 2)
The manual does not tell you this
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1
Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)
Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1
Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)
Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
-
- Starting out
- Posts: 43
- Joined: Tue Nov 22, 2016 11:39 pm
- Location: Bay Area, CA
Re: NAS to NAS or RTRR
Can you elaborate on the 'versioning' ... I must have missed that one!P3R wrote: ...
RTRR have been constantly developed and have more features and advantages than NAS to NAS:...
- Both real time and scheduled modes
- Two way synchronization (I'm not a fan and don't know how well it handles conflicts but it's available for the many users asking for it)
- Choice of protocols; RTRR, FTP, CIFS/Samba
- Versioning (this alone make it a killer app!)
- Best performance for bulk copying on fast, LAN-like, connections (disable; Check file contents, Compress files during transmission, Delete extra files)
Also - what's the general opinion of error correction / detection / reliability of RTRR? I did the following:
- copy a large directory structure comprised mainly of images (jpegs, TIFF, and raw) from laptop hard drive to NAS-1
- use RTRR to replicate the directory structure to an identical structure on NAS-2
- compare the laptop directory structure on the laptop (source in step 1 above) to the directory structure on NAS-2 using 'beyond compare' (something I've used for over 10 years and trust).
I'll do a bunch more testing and see what I can learn.
-
- Guru
- Posts: 13192
- Joined: Sat Dec 29, 2007 1:39 am
- Location: Stockholm, Sweden (UTC+01:00)
Re: NAS to NAS or RTRR
Versioning is only relevant for backups, not when synchronizing two sites that I believe is your application.Steerpike wrote:Can you elaborate on the 'versioning' ...
It's simply that in addition to the latest backup you also keep one or more versions of all the files that over time have changed or been deleted. It's like having historical "snapshots" of your data structure.
Think about the case where a ransomware encrypt data and before you realize that you your backup, without versioning, runs. Your only healthy files in the backup will be overwritten by all the encrypted files...
Or
When you realize that you once again need a file you deleted two months ago. Without versioning you're lost but with it you just go to the backup version that's slightly older than two months.
Backup versioning is possible to do manually with rotating backup destinations.
The beauty of having versioning integrated in the backup application is however that (if it's done properly) it's easy to manage the number and age of available versions and that old versions doesn't take up any more storage space than only the changed/deleted files themselves.
I don't blame you. It's been a pretty well kept secret for 18 months or so.I must have missed that one!
Versioning in RTRR is for some odd reason only available by installing the virtually undocumented Backup versioning app.
The Backup versioning app is officially still in beta but I've been using it from the beginning and on my actual production data for about 51 weeks now. It's been steady as a rock.
The reason RTRR versioning isn't out of beta, and most likely never will be, is that in QTS 4.3 there's a new backup solution that will replace NAS to NAS and RTRR. QTS 4.3 is also in beta and that I wouldn't use on my production unit until it's being released as stable software.
I have no opinion as I've never seen any documentation from Qnap on how it really works under the hood. Therefore I can only judge by my experience of the only case I've been using it extensively in. That's for backups to external disks connected locally to the NAS and I've noticed no issues with it. But I also haven't checked the data deeply, only random samples here and there when verifying my backups.Also - what's the general opinion of error correction / detection / reliability of RTRR?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
-
- Starting out
- Posts: 43
- Joined: Tue Nov 22, 2016 11:39 pm
- Location: Bay Area, CA
Re: NAS to NAS or RTRR
I re-copied (re-sync'd) the entire directory in question (this time, over the WAN taking several hours rather than minutes) and could not replicate the problem; so it would appear that one jpeg file got corrupted during the initial RTRR sync between my two NAS boxes. I will continue to validate my NAS backups (syncs) by comparing to the original files periodically, and see how it goes over time. If only 1 file out of thousands gets corrupted, and only occasionally, I can accept that.
- Moogle Stiltzkin
- Guru
- Posts: 11445
- Joined: Thu Dec 04, 2008 12:21 am
- Location: Around the world....
- Contact:
Re: NAS to NAS or RTRR
huh? how did it get corrupted what were your settings.Steerpike wrote:I re-copied (re-sync'd) the entire directory in question (this time, over the WAN taking several hours rather than minutes) and could not replicate the problem; so it would appear that one jpeg file got corrupted during the initial RTRR sync between my two NAS boxes. I will continue to validate my NAS backups (syncs) by comparing to the original files periodically, and see how it goes over time. If only 1 file out of thousands gets corrupted, and only occasionally, I can accept that.
as far as i know it could only get corrupted if it the source file already was corrupted to begin with, or you had a powercut or shutdown mid rtrr task perhaps ?
But didn't expect there to be any errors from just performing the rtrr task
and how did you manage to figure out from those thousands of files which got corrupted?
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1
Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)
Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1
Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)
Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
-
- Starting out
- Posts: 43
- Joined: Tue Nov 22, 2016 11:39 pm
- Location: Bay Area, CA
Re: NAS to NAS or RTRR
Well, that's the big question - how did the corruption occur? Hence, my asking about error detection and correction.Moogle Stiltzkin wrote:huh? how did it get corrupted what were your settings.Steerpike wrote:I re-copied (re-sync'd) the entire directory in question (this time, over the WAN taking several hours rather than minutes) and could not replicate the problem; so it would appear that one jpeg file got corrupted during the initial RTRR sync between my two NAS boxes. I will continue to validate my NAS backups (syncs) by comparing to the original files periodically, and see how it goes over time. If only 1 file out of thousands gets corrupted, and only occasionally, I can accept that.
as far as i know it could only get corrupted if it the source file already was corrupted to begin with, or you had a powercut or shutdown mid rtrr task perhaps ?
But didn't expect there to be any errors from just performing the rtrr task
and how did you manage to figure out from those thousands of files which got corrupted?
As for my settings - they are all defaults for the RTRR job. I have since disabled the syncing of 'temp' files / recycle folder, but that should not have any bearing.
As I mentioned, I copied (windows file copy from laptop to SMB share on NAS) all the files to NAS-1; I then used RTRR to sync those files to NAS-2 over my LAN; I then relocated NAS-2 to my office in CA, and reconfigured the RTRR job to work with this new address (but no actual files were transferred after this since everything was 'seeded' already); then used 'Beyond compare' to validate the files on NAS-2 against the original files on the laptop. Beyond Compare compares two directory structures and highlights differences. Only 1 file out the the thousands involved was flagged as different.
I deleted the directory containing the corrupt file on NAS-2, and re-sync'd it from NAS-1 (this time over the WAN); all files were good at this point. So that tells me the 'source' of the original RTRR - NAS-1 - was good.
So something must have happened during the original RTRR process to cause that one file to be corrupted. This is not terribly shocking to me, and is the reason I'm doing these tests - to determine if I'm really willing to commit to this as my backup strategy.
- Moogle Stiltzkin
- Guru
- Posts: 11445
- Joined: Thu Dec 04, 2008 12:21 am
- Location: Around the world....
- Contact:
Re: NAS to NAS or RTRR
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1
Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)
Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1
Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)
Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
-
- Guru
- Posts: 13192
- Joined: Sat Dec 29, 2007 1:39 am
- Location: Stockholm, Sweden (UTC+01:00)
Re: NAS to NAS or RTRR
Which tells us that you didn't have the option Check file contents ticked. Because had you used that, RTRR should have found the difference and automatically updated the file for you.Steerpike wrote:I then relocated NAS-2 to my office in CA, and reconfigured the RTRR job to work with this new address (but no actual files were transferred after this since everything was 'seeded' already);...
Please note that the above isn't criticism, just an observation that Check file contents would automatically have corrected the corrupt file. However depending on the available bandwidth, time and volume of data, Check file contents may not be practically usable in all applications.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
- Moogle Stiltzkin
- Guru
- Posts: 11445
- Joined: Thu Dec 04, 2008 12:21 am
- Location: Around the world....
- Contact:
Re: NAS to NAS or RTRR
i was actually planning to do a factory reset sometime soon once a more stable firmware comes.P3R wrote:Which tells us that you didn't have the option Check file contents ticked. Because had you used that, RTRR should have found the difference and automatically updated the file for you.Steerpike wrote:I then relocated NAS-2 to my office in CA, and reconfigured the RTRR job to work with this new address (but no actual files were transferred after this since everything was 'seeded' already);...
Please note that the above isn't criticism, just an observation that Check file contents would automatically have corrected the corrupt file. However depending on the available bandwidth, time and volume of data, Check file contents may not be practically usable in all applications.
so if source had already previous backed up to destination, but later i accumulates just a few new files i wanted to rtrr. Would i need to enable check file contents? Or would leaving that disabled be fine for my purposes ?
Because previously i did in this order
1. NAS to NAS, source to destination
2. RTRR, source to destination, with verify file enabled (i'm assuming all it did was check if file the same, if it is, don't transfer? that said it still took a long time to complete )
So if source had a bunch of new files (not replacements for existing ones that are already on destination), would it be okay to leave verify files disabled ?
on side note, you can't just simply rtrr now, without setting it into scheduled or real time mode. In disabled it's completely disabled. Even the icon for run task refuses to run. But there is a run now toggle from the pull down :/ Looking further apparently have to schedule and run now. guess they err on the side of caution by not simply allowing you to click run task if it's set to disabled in here.
but anyway what i'm trying to get at is, since the files i had prior added already was verified, i rather avoid verifying them again, as their file name that it's already at destination would already suffice if only so it knows not to delete them or add a copy of the file at destination based on folder/file name alone. Then only the new file transferred would verify it's contents to ensure it transfers over without issues. Or is the transfer process already solid in not coughing up a corrupt file on the other end aka destination with verify setting disabled?
by the way got rtrr running now with verify disabled. I also disabled ssh seeing as it's done locally and it's not port forwarded over to the internet anyway. Anyway performance is excellent at 100 MB/s on average. It was probably either encrypted transfer or the verify contents which tanked the performance in prior backups.
but what concerns me is the roughly 1000+ backed up files. Because the only new additional files i added were not that many ... Also is this a bug that time left and time remaining to match ? synchronizing also not the same
Would it be possible for the job log for after rtrr is completed to show just which files were backed up (as in not skipped) ? cause right all i know is it skipped some, but backed up a 1000+ things of which i'm not sure what gonna be hard to self verify seeing as my contents have so many sub folders and pathings like a maze
*update
RTRR completed in 8 hours by not verifying content and encryption transfer disabled. Also data already previously already been RTRR to the destination.
anyway it says 2tb roughly was updated, but the total size for the data rtrr was 6tb. I was only expecting the updated to be less than 10gb, not 2tb... as there was only a few additional files added recently. Trying to check logs did nothing because all it told me during the rtrr was number of files skipped and files backed up, and the total amount of the whole dataset being processed. When the whole RTRR completed the logs was very basic and doesn't tell you indepth which folders/sub folders/files exactly were newly successfully backed up (and not skipped due to an existing file already at destination). So i'll need to go filestation for destination server, to delete and then run rtrr again but using "verify contents". i did a rough check and did not see any duplicates or errors, but i need some certainty that it copied exactly... hence why i need to rtrr again from scratch this time
For future subsequent rtrr i'll have to use verify contents
*update
doing rtrr from scratch with verify contents enabled. but encryption disabled (since doing over local network so doubt i need it). 101 MB/S ETA 16 or so hours to backup 6.6 TB worth of data, very nice
Last edited by Moogle Stiltzkin on Sat Dec 10, 2016 2:03 am, edited 1 time in total.
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1
Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)
Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1
Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)
Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
-
- Starting out
- Posts: 43
- Joined: Tue Nov 22, 2016 11:39 pm
- Location: Bay Area, CA
Re: NAS to NAS or RTRR
There is a check box in the RTRR job 'options' screen to enable detailed logs. On the RTRR screen, at the very top (not specific to any one job), click on the options button and then choose to enable detailed logs. I set that when I first started working with the QNAP so I can't say what a 'non-detailed' job log looks like, but with this checked, you get a log entry for every single file sync'd by the job. I also found the log file at /etc/logs/qsync/detail. Sadly it gets overwritten each time the job runs, so I have a script to rename it and move it to another location.Moogle Stiltzkin wrote:....
Would it be possible for the job log for after rtrr is completed to show just which files were backed up (as in not skipped) ? cause right all i know is it skipped some, but backed up a 1000+ things of which i'm not sure what gonna be hard to self verify seeing as my contents have so many sub folders and pathings like a maze
...
In my case, I was getting a lot more files sync'd than I expected but the log revealed it was due to the contents of @recycle being included; I've subsequently disabled that setting.
-
- Starting out
- Posts: 43
- Joined: Tue Nov 22, 2016 11:39 pm
- Location: Bay Area, CA
Re: NAS to NAS or RTRR
I have yet to explore the impact of 'check file contents' on the performance of the sync job. I am expecting it to render the sync unusable in my situation, as I only have ~5Mbps upload speed and I have hundreds of thousands of files, but - I will certainly give it a try.P3R wrote:Which tells us that you didn't have the option Check file contents ticked. Because had you used that, RTRR should have found the difference and automatically updated the file for you.Steerpike wrote:I then relocated NAS-2 to my office in CA, and reconfigured the RTRR job to work with this new address (but no actual files were transferred after this since everything was 'seeded' already);...
Please note that the above isn't criticism, just an observation that Check file contents would automatically have corrected the corrupt file. However depending on the available bandwidth, time and volume of data, Check file contents may not be practically usable in all applications.
I agree that 'check contents' would have caught this corruption, but ironically it would have been caught even without it (I believe), since the corruption rendered the file smaller (I understand that, without 'check contents', it will rely on name/date/size).
Let's hope they implement 'check contents' in some optimized fashion; simplistically, if they just do a compare, then they would have to first copy the file to the target side, then compare, and discard if not different - which means every sync would be a full copy of all files. I can imagine some efficient checksum calculations though ... I guess I need to try it and observe!