[HOWTO] Daily incremental backup with rsnapshot

Discussion on remote replication.
Post Reply
User avatar
AndyChuo
Experience counts
Posts: 2396
Joined: Thu Sep 13, 2007 11:56 am
Location: Taipei, Taiwan

Re: Daily incremental backup: rsync + snapshots via shell script

Post by AndyChuo » Wed Jun 03, 2009 11:49 am

Stickied. Thanks for sharing and we'd like to see how it goes after running for a certain period of time and will consider adding to the current firmware.

Andy
=============================================================>>>
TS-659-Pro [RAID6] rtorrent+SABnzbdplus+SickBeard+Couchpotato [Best PVR] Plex+PMS [Ultimate Streamer]
Apple iPad [Best Tablet] HTC One M8 [Mobile Phone] Samsung UA46ES6100 [My Screen] KRK Rokit 6 [Audio Speakers]
Chrome Cast [Screen Casting] Philips Hue [Personal Lighing]
Buffalo WZR-1750DHP [My Wifi Hub] D-Link DGS-1005D [Gbit Network]
=============================================================>>>

Adrian
Starting out
Posts: 40
Joined: Fri May 08, 2009 5:27 am

Re: Daily incremental backup: rsync + snapshots via shell script

Post by Adrian » Fri Jun 05, 2009 4:32 pm

this looks like a great solution for my Small Office (4 users on a server) however it is missing 2 things to make this an ideal solution for me.
1) a web GUI
2) a daily backup log of changed files (viewable in HTML)

it would be great if the Qnap team could package this for easy install.

thanks,
Adrian

crushdepth
Getting the hang of things
Posts: 76
Joined: Thu Mar 26, 2009 8:56 pm

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by crushdepth » Sun Jun 14, 2009 1:07 pm

Ok I have this in production at work now using two TS-639s, one of which is being used as a file server, and the other as a backup server running rsnapshot over SSH with key-based authentication. So far it is working well, but I'll post back after our users have pounded on the system for a month or so.

Also attached: Slightly revised and corrected rsnapshot HOWTO guide. Please use this version as the last one had an error in the command to relaunch the crontab (required to make changes persistent). It should have read:

Code: Select all

crontab /etc/config/crontab


EDIT: Please see post #22 below for the final version of the installation guide, I have deleted the old version.
Last edited by crushdepth on Sun Jul 19, 2009 1:49 pm, edited 1 time in total.

kl-NAS
Starting out
Posts: 18
Joined: Tue Nov 25, 2008 3:39 pm

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by kl-NAS » Mon Jun 15, 2009 3:46 pm

Hi Crushdepth,

excellent paper :D - I just implemented this on a TS-509 locally and it worked great. Next step will be to incorporate some remote PCs with key-based authentication.

Again, many thanks for this great contribution to this wonderful forum.

kenhag
Starting out
Posts: 18
Joined: Wed Jan 28, 2009 11:40 pm

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by kenhag » Thu Jun 25, 2009 9:29 pm

Great! Just what I was looking for.
Of course, with a GUI it would be even better. :D

/Kent
------------------------------------------------------------------
QNAP TS-509Pro, Firmware version: 4.1.0
5 x Samsung Spinpoint F1 1TB

User avatar
Spag
Know my way around
Posts: 227
Joined: Tue Nov 18, 2008 7:42 pm
Location: Denmark

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by Spag » Thu Jul 09, 2009 9:15 pm

Can anyone tell me how the folder structure or backup image looks like on the backup server?
TS-569 Pro, Firmware 4.1.0 Build 1101 (Beta)
TS-469 Pro, Firmware 4.1.0 Build 1101 (Beta)
TS-239 Pro, Firmware 4.0.4 Build 20131001
TS-209 Pro II, Firmware 3.3.2 Build 0918

crushdepth
Getting the hang of things
Posts: 76
Joined: Thu Mar 26, 2009 8:56 pm

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by crushdepth » Thu Jul 09, 2009 11:06 pm

You designate a backup directory on the backup server. Inside this, you will get a collection of snapshots building up over time, eg:

hourly.0
hourly.1
hourly.2
hourly.3
daily.0
daily.1
daily.2
daily.3
weekly.0....etc
monthly.0 ...etc

Each one of these will contain a file structure identical to the target file system at the time the snapshot was taken.

crushdepth
Getting the hang of things
Posts: 76
Joined: Thu Mar 26, 2009 8:56 pm

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by crushdepth » Fri Jul 17, 2009 12:05 pm

Ok I've been running rsnapshot in our office for a month now between our two TS-639s. We use one as a file server and the other as a backup device. I've also set it to backup our remote webserver/website as well. I'm happy to say it is working very well and no problems at all. I have added a small section to the install guide on monitoring the rsnapshot log, this will be the final version.

By the way, if anyone has any feedback on how well (or not) this is working for them, I would appreciate it.
You do not have the required permissions to view the files attached to this post.

kaiynne
New here
Posts: 4
Joined: Wed Jan 28, 2009 3:21 pm

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by kaiynne » Fri Aug 07, 2009 12:45 pm

Thanks for the how to was excellent!

GeoffC
Starting out
Posts: 16
Joined: Wed Aug 26, 2009 7:51 pm

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by GeoffC » Mon Sep 14, 2009 1:37 pm

I have been meaning to add some comments here for a while.

Firstly - Thanks Crushdepth - a great job with the "How To". For what this can do, you made it all quite straight forward.

A couple of things I found;
When you download the Optware-IPKG file, you need to unzip it before you install it - I know it's obvious, and it was to me when it didn't work the first time. It may even be IE changing the extension, as I have seen it do funning things like that before.

I've installed this on two TS-239 units - on one (was running 2.1.7 firmware if that matters, while the other was 3.1.1), I had to reboot the NAS before I could run "ipkg update". Maybe it was just a one off or something else I did.

As for editors, I like to use WinSCP, as it helps me locate the files - seasoned command line users please yourself.

When configuring rsnapshot.conf, I found "interval" very confusing, as it may or may not have anything to do with how often you take snapshots.
Reading the man page at rsnapshot.org, "interval" is now a depreciated alias for the keyword "retain" which makes much more sense with its use.

crikeyjp
New here
Posts: 2
Joined: Thu Sep 17, 2009 8:41 pm

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by crikeyjp » Thu Sep 17, 2009 9:15 pm

Hi,

I've been running rsnapshot on a 219P for a while now and had automated brilliantly. Thanks for the guide and help!!

However, I'm having problems running rsnapshot-diff. It seems to fall over as soon as it hits a large file (greater than 2GB I think). Is anyone else seeing this problem???

Output looks like this:

Code: Select all

snapshot-diff daily.1/localhost/share/dir2 daily.0/localhost/share/dir2
Comparing daily.1/localhost/share/dir2 to daily.0/localhost/share/dir2
Use of uninitialized value in numeric eq (==) at /opt/bin/rsnapshot-diff line 220.
Can't compare across devices.
(looking at daily.1/localhost/share/dir2/file1)


BTW, dir2 contains hundreds of files but file1 is 3114364980 bytes big.

Cheers,
JP

jimguo
Getting the hang of things
Posts: 54
Joined: Sun Jul 26, 2009 12:52 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by jimguo » Sun Nov 15, 2009 11:47 pm

Hi crushdepth,

Re: rsnapshot.conf

I can only find only 2 places needs to be amended?

1. #snapshot_root /opt/var/rsnapshot/
- should I change the /opt/var/rsnapshot/ to /share/myownfolder ?

2. if I want to backup /Qusb on the source, however I got errors with the below:
backup admin@myowndomain:/share/Qusb / localhost/
don't know where the problem is.

appreciate your help.

regards
Jim
TS 239 + TS 119 + 469U

crushdepth
Getting the hang of things
Posts: 76
Joined: Thu Mar 26, 2009 8:56 pm

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by crushdepth » Mon Nov 16, 2009 11:31 am

I'm not sure if it would matter, but it is probably a good idea to set up a different folder as your snapshot root as you say. My root path looks like this (RAID 6, yours might look slightly different if you are using a different kind of RAID):

Code: Select all

snapshot_root   /share/MD0_DATA/snapshots/


Your backup point looks ok, but try using the IP address of the source machine rather than a domain name. eg:

backup admin@192.168.10.42:/share/MD0_DATA/ orca/ [this is another NAS on my local network, I backup the whole share folder]
backup root@89.24.126.98:/home/penguins/public_html/ server.penguins.com/ [this is a remote webserver]

Have you got any other backup points working? What does the error message say?

jimguo
Getting the hang of things
Posts: 54
Joined: Sun Jul 26, 2009 12:52 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by jimguo » Mon Nov 16, 2009 8:56 pm

thanks!
it's still not working. i got below when input 'rsnapshot configtest'

Code: Select all

rsnapshot encountered an error! The program was invoked with these options:
/opt/bin/rsnapshot configtest
----------------------------------------------------------------------------
ERROR: /opt/etc/rsnapshot.conf on line 162:
ERROR: backup admin@125.107.82.10:/share/HDA_DATA/Qusb/     localhost/
ERROR: ---------------------------------------------------------------------
ERROR: Errors were found in /opt/etc/rsnapshot.conf,
ERROR: rsnapshot can not continue. If you think an entry looks right, make
ERROR: sure you don't have spaces where only tabs should be.


don't know where the problem is?

haven't tested others yet.
TS 239 + TS 119 + 469U

crushdepth
Getting the hang of things
Posts: 76
Joined: Thu Mar 26, 2009 8:56 pm

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by crushdepth » Tue Nov 17, 2009 11:49 am

Ok if there's an error in the config file it usually means you have a space character in there somewhere. Everything has to be separated with tab characters, spaces are illegal. Is there any spaces in that line?

Post Reply

Return to “Remote Replication/ Disaster Recovery”