How to diagnose rsync failure with new setup?

Discussion on remote replication.
Post Reply
Jim DeLaHunt
Starting out
Posts: 35
Joined: Tue Apr 06, 2010 1:41 pm

How to diagnose rsync failure with new setup?

Post by Jim DeLaHunt » Thu Jun 04, 2020 8:49 am

Hello,

I am trying to get my new FreeNAS file server to back up files to my old QNAP TS-219P NAS using rsync. I did what I thought was reasonable to turn on Rsync Server in the Backup Station application on the TS-219P. I did what I thought was reasonable to schedule an Rsync task in FreeNAS. I launched the job. Then… nothing.

I would like to look at logs on the TS-219P to get some idea of what the rsync Pull-side is hearing. Where do I find such logs? I have turned on QTS Control Panel -- System Logs -- System Connection Logs. I have logged into the TS-219P as admin and looked at `/var/log/messages` from the shell. I'm not seeing the sort of granular logs I would expect. Where should I look? How can I diagnose a failing new rsync setup?

My TS-219P is running 4.3.3.1252 Build 20200409.
Jim DeLaHunt, multilingual websites consultant, Vancouver, Canada. User Jim_DeLaHunt on wiki.qnap.com.
My QNAP device: TS-219P, firmware 4.3.3.1252 Build 20200409, 3TB + 1TB Western Digital disks as a backup container.
Also: a FreeNAS Mini E with 4 x 6TB Western Digital Red disks and a ZFS filesystem.

User avatar
Moogle Stiltzkin
Ask me anything
Posts: 8910
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: How to diagnose rsync failure with new setup?

Post by Moogle Stiltzkin » Thu Jun 04, 2020 9:59 am

depends what app you are using.

If you are login into qts, you can use HBS and do a rsync.

Your freenas would have a rsync server for the qnap to login to.

If you are doing the backup from freenas, you use their os for rsync. Then on the qnap you would have a rsync server enabled.


To check that rsync worked, you check from the device and it's own app you are using to perform the rsync operation from. there should be a "progress" or logs about how the rsync operation worked.
NAS
[Main Server] QNAP TS-877 w. 4tb [ 3x HGST Deskstar NAS (HDN724040ALE640) & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A w. 5x 2TB Samsung F3 (HD203WI) EXT4 Raid5
[Backup] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) single disks.
[^] QNAP TS-659 Pro II
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-228
[^] QNAP TS-128
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Asus AC68U Router|100dl/50ul MBPS FTTH Internet | Win10, WC PC-Intel i7 920 Ivy bridge desktop (1x 512gb Samsung 850 Pro SSD + 1x 4tb HGST Ultrastar 7K4000)


Guides & articles
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin

Jim DeLaHunt
Starting out
Posts: 35
Joined: Tue Apr 06, 2010 1:41 pm

Re: How to diagnose rsync failure with new setup?

Post by Jim DeLaHunt » Fri Jun 05, 2020 8:39 am

Thank you for your reply.
Moogle Stiltzkin wrote:
Thu Jun 04, 2020 9:59 am
If you are login into qts, you can use HBS and do a rsync.
By "HBS" you mean the QNAP application "Hybrid Backup Sync", correct?

Since I have already said I am backing up data from the FreeNAS server to the QNAP NAS, if I start with an app on the QNAP side, it needs to be an app that can initiate an rsync "pull" operation. It looks to me like the "Rsync Server" in QNAP's Backup Station app can only wait for the other system to initiate an Rsync operation, and it is not clear if that operation can go in either direction, or only from remote system to local system. By contrast, the "Rsync (File-level backup)" in QNAP's Backup Station app can initiate an operation, but only in the direction from the local system to the remote system, which is not what I want.

Can the Hybrid Backup Sync (HBS) application initiate an Rsync "pull" operation? Does it work better than the "Rsync Server" in QNAP's Backup Station?
Moogle Stiltzkin wrote:
Thu Jun 04, 2020 9:59 am
To check that rsync worked, you check from the device and it's own app you are using to perform the rsync operation from. there should be a "progress" or logs about how the rsync operation worked.
So does that say that the "Rsync Server" in QNAP's Backup Station app does not leave any logs about how the operation goes from its side?

If so, that's a pity. I now have the FreeNAS server initiating rsync "push" operations from the FreeNAS server to the QNAP server. Those jobs are failing after a few hours and tens of GB. All I see in the FreeNAS logs is a message like, "rsync: connection unexpectedly closed", as if the QNAP side closed the connection. I would like to see logs on the QNAP side giving its explanation for the problem.

Is there a way to enable such logs? And does the HBS application provide good logs when problems occur?
Jim DeLaHunt, multilingual websites consultant, Vancouver, Canada. User Jim_DeLaHunt on wiki.qnap.com.
My QNAP device: TS-219P, firmware 4.3.3.1252 Build 20200409, 3TB + 1TB Western Digital disks as a backup container.
Also: a FreeNAS Mini E with 4 x 6TB Western Digital Red disks and a ZFS filesystem.

User avatar
Moogle Stiltzkin
Ask me anything
Posts: 8910
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: How to diagnose rsync failure with new setup?

Post by Moogle Stiltzkin » Fri Jun 05, 2020 8:55 am

Jim DeLaHunt wrote:
Fri Jun 05, 2020 8:39 am
By "HBS" you mean the QNAP application "Hybrid Backup Sync", correct?
correct. most of us just say HBS so we don't have to type out the full thing. it gets tiring after a while, hence the abbreviation HBS :mrgreen:

Jim DeLaHunt wrote:
Fri Jun 05, 2020 8:39 am
Since I have already said I am backing up data from the FreeNAS server to the QNAP NAS, if I start with an app on the QNAP side, it needs to be an app that can initiate an rsync "pull" operation. It looks to me like the "Rsync Server" in QNAP's Backup Station app can only wait for the other system to initiate an Rsync operation, and it is not clear if that operation can go in either direction, or only from remote system to local system. By contrast, the "Rsync (File-level backup)" in QNAP's Backup Station app can initiate an operation, but only in the direction from the local system to the remote system, which is not what I want.
to my understanding, one of the NAS has to have a RSYNC server running for the other to connect to. You can either do that on your freenas, or on your QNAP. QNAP has RTRR also but that is exclusive to QNAP devices. So you have to use RSYNC instead.

Well if you don't want to use HBS you don't have to. Use the rsync app in freenas, to connect to the QNAP and run your backup from there.

lawrence explains how rsync works on freenas
https://www.youtube.com/watch?v=Qhlp18QTUTo

@10:20
"How to sync it with another NON freenas" :wink:


Your freenas backup app should retain logs somewhere? :S
FreeNAS Experienced
posted Aug 20, 2015

1.The execution of the rsync task is shown in /var/log/cron.
2. Any messages are put into /var/log/messages.
3. By default, rsync will produce no output when no files need to be transferred.
4. Therefore, if all is well, no messages go into /var/log/messages.
https://www.ixsystems.com/community/thr ... ask.36457/


and for HBS

Hybrid Backup Sync: Where are detailed logs?
viewtopic.php?f=15&t=148937


if you use HBS it will definitely generate logs (when creating and using backup/rsync jobs from HBS app). But if your freenas is pushing rsync to the QNAP, not quite sure, but i suspect it may because it has a rsync server running on the QNAP :'

but if it doesn't you can still check the logs from the freenas side, for what it succesfully rsync over etc.


maybe you can ask P3R, i heard he also uses freenas or something else? so he might know more about this :'
NAS
[Main Server] QNAP TS-877 w. 4tb [ 3x HGST Deskstar NAS (HDN724040ALE640) & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A w. 5x 2TB Samsung F3 (HD203WI) EXT4 Raid5
[Backup] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) single disks.
[^] QNAP TS-659 Pro II
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-228
[^] QNAP TS-128
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Asus AC68U Router|100dl/50ul MBPS FTTH Internet | Win10, WC PC-Intel i7 920 Ivy bridge desktop (1x 512gb Samsung 850 Pro SSD + 1x 4tb HGST Ultrastar 7K4000)


Guides & articles
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin

Post Reply

Return to “Remote Replication/ Disaster Recovery”