Rsync never completes

Discussion on remote replication.
Locked
saboteur
Starting out
Posts: 29
Joined: Thu Sep 23, 2010 2:15 am

Rsync never completes

Post by saboteur »

Hi,

I have set up a Remote Replication job to another NAS running rsyncd on the same network. The job appears to run fine (shows the percentage change in status) but never successfully completes.

The two devices are:

xx-nas01
QNAP TS-259 Pro+
Version 4.0.7 (2014/04/12)

xx-nas02
HP N54L
FreeNAS-9.2.1.4.1-RELEASE-x64 (fec915c)

Here are the error messages on the QNAP side:

Code: Select all

Warning : [Remote Replication] esx failed: Connection reset by peer. Remote backup service might be restarted/stopped.. Begin 1st retry.
Warning : [Remote Replication] esx failed: Connection reset by peer. Remote backup service might be restarted/stopped.. Begin 2nd retry.
Warning : [Remote Replication] esx failed: Connection reset by peer. Remote backup service might be restarted/stopped.. Begin 3rd retry.
Error : [Remote Replication] esx failed: Connection reset by peer. Remote backup service might be restarted/stopped..
'Rsync log'

Code: Select all

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Connection reset by peer (104)
rsync: connection unexpectedly closed (2621470 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.7]
These are the options enabled in the rsync job:

Perform incremental replication
Handle sparse files efficiently

Here are the errors on the FreeNAS side:

Code: Select all

Apr 25 02:02:22 xx-nas02 rsyncd[42418]: rsync error: timeout in data send/receive (code 30) at io.c(137) [receiver=3.0.9]                                     
Apr 25 02:02:22 xx-nas02 rsyncd[42418]: rsync: connection unexpectedly closed (210 bytes received so far) [generator]                                         
Apr 25 02:02:22 xx-nas02 rsyncd[42418]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [generator=3.0.9]
Here is the rsyncd.conf on the FreeNAS side:

Code: Select all

# cat /usr/local/etc/rsyncd.conf                            
use chroot = yes                                                                
max connections = 4                                                             
pid file = /var/run/rsyncd.pid                                                  
                                                                                
[esx]                                                                           
    path = /mnt/esx                                                             
    max connections = 0                                                         
    uid = esx                                                                   
    gid = esx                                                                   
    read only = false                                                           
    write only = false                                                          
    hosts allow = 192.168.1.11
Any clues?

The two devices are connected via a Cisco gigabit switch and we haven't experienced any other network issues.

I was running a 3.x version of QNAP firmware and had the same issue. I was also running an earlier version of FreeNAS but upgraded both ends hoping to resolve the issue.

Cheers
TS-212, TS-251+ both running Debian Bookworm
saboteur
Starting out
Posts: 29
Joined: Thu Sep 23, 2010 2:15 am

Re: Rsync never completes

Post by saboteur »

I managed to solve this by setting the timeout to 0 on the QNAP. The web interface only accepts values between 1 and 600, so I had to use SSH to do it. I made the change "the QNAP way" rather than editing the file directly, so you shouldn't have any nasty surprises.

The Remote Replication job gets started from crontab:

Code: Select all

# grep rsyncRR.sh /etc/config/crontab
0 1 * * 6 /etc/init.d/rsyncRR.sh Schedule0 2>/dev/null
Within rsyncRR.sh, rsync supplies the timeout based on the value of xsTimeout:

Code: Select all

# grep ^xsTimeout /etc/init.d/rsyncRR.sh
xsTimeout=`/sbin/getcfg "Global" "Timeout" -f "${RR_CONF}" -d "600"`
RR_CONF contains settings for Remote Replication:

Code: Select all

# grep ^RR_CONF /etc/init.d/rsyncRR.sh
RR_CONF="/etc/config/rsync_schedule.conf"
My fix was to use setcfg to the desired value:

Before the change:

Code: Select all

# /sbin/getcfg "Global" "Timeout" -f "/etc/config/rsync_schedule.conf" -d "600"
600
Making the change:

Code: Select all

# /sbin/setcfg "Global" "Timeout" "0" -f "/etc/config/rsync_schedule.conf"
After the change:

Code: Select all

# /sbin/getcfg "Global" "Timeout" -f "/etc/config/rsync_schedule.conf" -d "600"
0
Hope this helps someone out there!

Cheers
TS-212, TS-251+ both running Debian Bookworm
User avatar
orcinus
Starting out
Posts: 13
Joined: Fri Jul 13, 2012 7:26 am

Re: Rsync never completes

Post by orcinus »

THANK YOU.
This has been driving me up the wall for months.

There's a bug in the old rsync versions that causes the protocol error.
It happens because the old versions send the keepalive message (0:0) while sending the initial file list, and rsync doesn't expect it during that (initial) part of the sync.
And if you're rsyncing a really big file tree, the file list is so long that the keepalive gets sent in the middle of it, causing rsync to die with an exception.

Disabling the timeout altogether is a workaround (because it turns out to disable sending the keepalive altogether), but i couldn't figure out how to do it "the QNAP way".
jhnmdahl
Starting out
Posts: 13
Joined: Fri Oct 30, 2009 10:18 pm

Re: Rsync never completes

Post by jhnmdahl »

This appears to have solved my issues backing up a TVS-671 to a TVs-659 as well, using HBS or Hybrid Backup and Sync, receiving a "connection reset by peer" error. Wish QNAP would fix the firmware on device such as the 659, which is older but still perfectly capable, to work better.

It's especially annoying that backing up works with any third party solution, but QNAP's own software fails.

John
User avatar
OneCD
Guru
Posts: 12010
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: Rsync never completes

Post by OneCD »

* topic locked to prevent further necroposting *

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage
Locked

Return to “Remote Replication/ Disaster Recovery”