[HOWTO] Daily incremental backup with rsnapshot

Discussion on remote replication.
micke
Experience counts
Posts: 1363
Joined: Sat Feb 06, 2010 2:42 pm
Location: Taipei, Taiwan

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by micke » Wed Jun 01, 2011 11:58 pm

ectospasm wrote:You may also be able to redirect rsync output to a file, but I'm not sure where to set this.

In /etc/init.d/rsyncRR.sh (set DEBUG to yes). In debug mode the log files in /etc/logs/rsync/ should include more information. You can also modify other rsync options in the rsyncRR.sh file, e.g. increase the verbosity.

/Mike

stepir
Getting the hang of things
Posts: 71
Joined: Mon Mar 28, 2011 4:02 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by stepir » Tue Jun 07, 2011 5:48 am

micke wrote:
ectospasm wrote:You may also be able to redirect rsync output to a file, but I'm not sure where to set this.

In /etc/init.d/rsyncRR.sh (set DEBUG to yes). In debug mode the log files in /etc/logs/rsync/ should include more information. You can also modify other rsync options in the rsyncRR.sh file, e.g. increase the verbosity.

/Mike


Thanks for the suggestion but it did not produced any log in /etc/logs/rsync/

I have two issue with rsnapshot:

first is during backup

Code: Select all

ERROR: /opt/bin/rsync returned 20 while processing /share/MD0_DATA/Backup/
WARNING: /share/MD0_DATA/Multimedia/ skipped due to rollback plan
WARNING: /share/MD0_DATA/SWRepo/ skipped due to rollback plan
WARNING: /share/MD0_DATA/StepirData/ skipped due to rollback plan
WARNING: Rolling back "localhost/"
/opt/bin/rm -rf /share/external/sds2/qnapbackup/hourly.0/localhost/
/opt/bin/cp -al /share/external/sds2/qnapbackup/hourly.1/localhost \
    /share/external/sds2/qnapbackup/hourly.0/localhost
touch /share/external/sds2/qnapbackup/hourly.0/
rm -f /opt/var/run/rsnapshot.pid


second is while trying to remove older backup

Code: Select all

rm: unable to remove `/share/external/sds2/qnapbackup/hourly.5/localhost/share/MD0_DATA/Multimedia/Mp3/Compilations/Italian Pop Music/.@__thumb': Directory  not empty
rm: unable to remove `/share/external/sds2/qnapbackup/hourly.5/localhost/share/MD0_DATA/Multimedia/Mp3/Compilations/Italian Pop Music/.AppleDouble': Directory  not empty


Thanks

micke
Experience counts
Posts: 1363
Joined: Sat Feb 06, 2010 2:42 pm
Location: Taipei, Taiwan

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by micke » Tue Jun 07, 2011 11:35 am

stepir wrote:Thanks for the suggestion but it did not produced any log in /etc/logs/rsync/

Ignore what I wrote; for some reason I missed that this was about rsnapshot's usage of rsync. The changes to /etc/init.d/rsyncRR.sh is for remote replication.

Have you tried to increase the verbosity and log level in the rsnapshot.conf file?

/Mike

micke
Experience counts
Posts: 1363
Joined: Sat Feb 06, 2010 2:42 pm
Location: Taipei, Taiwan

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by micke » Tue Jun 07, 2011 3:22 pm

stepir wrote:first is during backup

Code: Select all

ERROR: /opt/bin/rsync returned 20 while processing /share/MD0_DATA/Backup/


How have you configured rsnapshot, i.e. is the user running the rsnapshot process (admin?) allowed to write to the /share/MD0_DATA/Backup/ share? Do you run it over a secure connection (i.e. using the default or optware version of ssh); if so how have you configured that setup?

/Mike

stepir
Getting the hang of things
Posts: 71
Joined: Mon Mar 28, 2011 4:02 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by stepir » Tue Jun 07, 2011 6:36 pm

Thanks Mike

I have raised rsnapshot log level in rsnapshot.conf but still not able to see which file it is processing at the moment in the log (it is just shown in the console if I manually run rsnapshot hourly).

This is a local backup (the destination is a local attached USB drive) and I'm using admin... who is correctly able to read the source data and write in the destination drive: as a double check the error 20 is reported after a few initial directories are copied!). Any way I could access rsync log to discover the actual reason of error 20?

thanks!

micke
Experience counts
Posts: 1363
Joined: Sat Feb 06, 2010 2:42 pm
Location: Taipei, Taiwan

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by micke » Wed Jun 08, 2011 3:18 am

stepir wrote:This is a local backup (the destination is a local attached USB drive)

Is the drive still mounted after the error 20 message? What file system is used for the drive?
stepir wrote:Any way I could access rsync log to discover the actual reason of error 20?

Have you tried to use these settings (I don't know if the settings show enough information from rsync at an error, though),

verbose 5
loglevel 5

Or try to add a few -v options to the rsync command (rsync_short_args in rsnapshot.conf)

/Mike

stepir
Getting the hang of things
Posts: 71
Joined: Mon Mar 28, 2011 4:02 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by stepir » Wed Jun 08, 2011 1:42 pm

micke wrote:Is the drive still mounted after the error 20 message? What file system is used for the drive?


Yes w/o issues, it is HFS+, it's a Drobo so no easy way to format it in EXT3/4

micke wrote:Have you tried to use these settings (I don't know if the settings show enough information from rsync at an error, though),

verbose 5
loglevel 5

Or try to add a few -v options to the rsync command (rsync_short_args in rsnapshot.conf)

/Mike


Will try it and let you know! thanks!

stepir
Getting the hang of things
Posts: 71
Joined: Mon Mar 28, 2011 4:02 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by stepir » Thu Jun 09, 2011 7:40 am

Code: Select all

[08/Jun/2011:20:22:03] /opt/bin/rsnapshot hourly: started
[08/Jun/2011:20:22:03] Setting locale to POSIX "C"
[08/Jun/2011:20:22:03] echo 22909 > /opt/var/run/rsnapshot.pid
[08/Jun/2011:20:22:03] mkdir -m 0755 -p /share/external/sds2/qnapbackup/hourly.0/
[08/Jun/2011:20:22:03] /opt/bin/rsync -av --delete --numeric-ids --relative --delete-excluded --exclude='/:2eTemporaryItems' --exclude='/Network Trash Folder' --exclude='/Temporary Items' --exclude='/.AppleDB' --exclude='/.AppleDesktop' --exclude='/.AppleDouble' --exclude='/.@__thumb' /share/MD0_DATA/Backup /share/external/sds2/qnapbackup/hourly.0/localhost/
[08/Jun/2011:20:22:28] /opt/bin/rsnapshot hourly: ERROR: /opt/bin/rsync returned 20 while processing /share/MD0_DATA/Backup/
[08/Jun/2011:20:22:28] /share/MD0_DATA/Multimedia/ skipped due to rollback plan
[08/Jun/2011:20:22:28] /share/MD0_DATA/SWRepo/ skipped due to rollback plan
[08/Jun/2011:20:22:28] /share/MD0_DATA/StepirData/ skipped due to rollback plan
[08/Jun/2011:20:22:28] touch /share/external/sds2/qnapbackup/hourly.0/
[08/Jun/2011:20:22:28] No directory to delete: /share/external/sds2/qnapbackup/_delete.22909
[08/Jun/2011:20:22:28] rm -f /opt/var/run/rsnapshot.pid
[08/Jun/2011:20:22:28] /opt/bin/rsnapshot hourly: ERROR: /opt/bin/rsnapshot hourly: completed, but with some errors
[08/Jun/2011:23:30:03] require Lchown
[08/Jun/2011:23:30:03] Lchown module not found


this is the log level 5, which complain about lchown missing. Still same error (20) w/o any more details about the reason... using -v in rsync_short_arg do not work (rsync won't even start). It seems that my first source data has something strange in it... maybe the path dept or folder naming... but w/o an rsync log I'm not able to figure out!

micke
Experience counts
Posts: 1363
Joined: Sat Feb 06, 2010 2:42 pm
Location: Taipei, Taiwan

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by micke » Thu Jun 09, 2011 11:27 am

stepir wrote:it is HFS+

And you are sure it works with rsnapshot? Does HFS+ support hard-links?
stepir wrote:using -v in rsync_short_arg do not work (rsync won't even start)

Note that the value must be specified as one single argument, e.g. '-avv' and not '-a -v -v'.

Anyway, you have the command rsnapshot is running in the log,

Code: Select all

/opt/bin/rsync -av --delete --numeric-ids --relative --delete-excluded --exclude='/:2eTemporaryItems' --exclude='/Network Trash Folder' --exclude='/Temporary Items' --exclude='/.AppleDB' --exclude='/.AppleDesktop' --exclude='/.AppleDouble' --exclude='/.@__thumb' /share/MD0_DATA/Backup /share/external/sds2/qnapbackup/hourly.0/localhost/

Run it manually from the command line (to a different destination on the HFS+ formatted drive).

/Mike

User avatar
Eduardo
Easy as a breeze
Posts: 350
Joined: Tue Jun 30, 2009 2:08 am
Location: Sevilla (Spain)

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by Eduardo » Thu Jun 09, 2011 11:33 pm

Hi Ceco,

Nice hint. I improved my NAS with it. But,

Ceco wrote:Hi.

Here is a hint for anyone who wants to have "Postpone the restart/shutdown schedule when a replication job is in progress" option
to work with rsnapshot. This works for my TS-219 and probably other systems. Just replace HDA_DATA with your device's data dir.

1) Create "scripts" directory on HDA_DATA and copy original /etc/init.d/wait_RR into it

Code: Select all

# cd /share/HDA_DATA
# mkdir scripts
# cp /etc/init.d/wait_RR /share/HDA_DATA/scripts


2) Now edit /share/HDA_DATA/scripts/wait_RR.
Replace

Code: Select all

/bin/ps | /bin/grep /usr/bin | /bin/grep "rsync "

With

Code: Select all

/bin/ps | /bin/grep "rsync "



I think it should be better to replace with:

Code: Select all

/bin/ps | /bin/grep "rsync " | /bin/grep -v "grep "


Ceco wrote:3) Rename original /etc/init.d/wait_RR and symlink to new one

Code: Select all

# cd /etc/init.d
# mv wait_RR wait_RR.orig
# ln -s /share/HDA_DATA/scripts/wait_RR wait_RR


4) Add Step 3 into your autorun script to preserve your new /etc/init.d/wait_RR after reboot.
Refer to http://wiki.qnap.com/wiki/Running_Your_ ... at_Startup for more info
TS-259Pro+,TS-239 Pro. Firm 3.7.3 Build0801
2xSeagate ST32000542AS CC34 Raid 1 (+1 ST32000542AS Raid 1 ESATA/USB Backup)
Qpkg: MLDonkey, JRE, Micke OpenSSH, Micke Nasreport, Python, CrashPlan, Rsnap.
Ipkg: Rsnapshot, logrotate, heyu
================================================================================================
Looking forward to a (Prolific 2303 or ftdi) USB to serial driver!

stepir
Getting the hang of things
Posts: 71
Joined: Mon Mar 28, 2011 4:02 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by stepir » Fri Jun 10, 2011 1:19 am

micke wrote:And you are sure it works with rsnapshot? Does HFS+ support hard-links?


Im not a linux expert but yes I would say (rsync now finish...):

Code: Select all

[/share/external/sds2/qnapbackup/hourly.0/localhost/share/MD0_DATA/Backup] # ls -la
drwxr-xr-x    1 admin    administ     4096 Jun  8 08:12 .AppleDB/
drwxrwxrwx    1 stepir   everyone     4096 Jun  8 08:12 .AppleDesktop/
drwxrwxrwx    1 stepir   everyone     4096 Jun  8 08:12 .AppleDouble/
drwxrwxrwx    1 admin    administ     4096 Apr  4 22:40 Apple/
drwxrwxrwx    1 admin    administ     4096 Apr  4 22:40 Applications/
drwxrwxrwx    1 admin    administ     4096 Mar 13 18:47 Chiara/
drwxrwxrwx    1 admin    administ     4096 Jan 31 22:39 Ebook/
drwxrwxrwx    1 admin    administ     4096 Apr  5 12:22 H264Webcam/
drwxrwxrwx    1 stepir   everyone     4096 Apr  6 19:31 Network Trash Folder/
drwxrwxrwx    1 admin    administ     4096 Mar 13 18:51 Phones/
drwxrwxrwx    1 stepir   everyone     4096 Apr  8 16:01 QNap/
drwxrwxrwx    1 admin    administ     4096 Apr  4 22:43 Router Belkin/
drwxrwxrwx    1 admin    administ     4096 Mar 13 18:59 SecureDigitals/
drwxrwxrwx    1 admin    administ     4096 May  5 20:48 SteQuad/
drwxrwxrwx    1 admin    administ     4096 Apr  4 23:20 SteVaio/
drwxrwxrwx    1 stepir   everyone     4096 Apr  6 19:31 Temporary Items/
drwxrwxrwx    1 admin    administ        0 Mar 15 13:37 Work/
drwx------    1 admin    administ     4096 Jun  9 16:00 Xbox360/
drwx------    1 admin    administ     4096 Jun  9 16:00 XboxLulla/


By the way, why these folders are still backed up even if I have the exclusion?:

Code: Select all

drwxr-xr-x    1 admin    administ     4096 Jun  8 08:12 .AppleDB/
drwxrwxrwx    1 stepir   everyone     4096 Jun  8 08:12 .AppleDesktop/
drwxrwxrwx    1 stepir   everyone     4096 Jun  8 08:12 .AppleDouble/


micke wrote:Note that the value must be specified as one single argument, e.g. '-avv' and not '-a -v -v'.
/Mike


That's what I did, I just specified -v since it was the only additional argument. How about using 'rsync_short_args >> rsynclog.txt' ?

micke
Experience counts
Posts: 1363
Joined: Sat Feb 06, 2010 2:42 pm
Location: Taipei, Taiwan

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by micke » Sat Jun 11, 2011 7:40 pm

stepir wrote:Im not a linux expert but yes I would say (rsync now finish...):

So everything is working now?
stepir wrote:By the way, why these folders are still backed up even if I have the exclusion?:

Incorrect exclusion? For example, to exclude the .AppleDouble directory you could use '--exclude='.AppleDouble/'.

/Mike

stepir
Getting the hang of things
Posts: 71
Joined: Mon Mar 28, 2011 4:02 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by stepir » Sun Jun 12, 2011 2:30 pm

On the backup side yes... still have doubts if the symb link are working or not with this drive (HFS+)

Will try your suggestion with the exclusion, could you please confirm you mean I should add a quotation (') at the beginning and at the end of the path and it should end with /. ??

Unfortunately I still have the issues in deleting older snapshot (still getting the Directory not empty error). I have to manually delete the content with multiple delete command since the files in those subfolders are shown only after the rm command.....

micke
Experience counts
Posts: 1363
Joined: Sat Feb 06, 2010 2:42 pm
Location: Taipei, Taiwan

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by micke » Sun Jun 12, 2011 7:59 pm

stepir wrote:could you please confirm you mean I should add a quotation (') at the beginning and at the end of the path and it should end with /. ??

Instead of / at the beginning I would suggest you add a / at the end for all exclude patterns that are directories, e.g. --exclude='.AppleDouble/'.

/Mike

Ceco
Starting out
Posts: 32
Joined: Thu Jan 20, 2011 3:08 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by Ceco » Mon Jun 13, 2011 9:01 pm

Eduardo wrote:Hi Ceco,

Nice hint. I improved my NAS with it. But,


I think it should be better to replace with:

Code: Select all

/bin/ps | /bin/grep "rsync " | /bin/grep -v "grep "



Ahh. I see your point. Thanx for the fix :)
TS-419U @ Work (4.3.3)
TS-431U @ Work (4.3.3)
TS-219P+ @ Home (4.3.3)

Post Reply

Return to “Remote Replication/ Disaster Recovery”