Page 1 of 14

[HOWTO] Daily incremental backup with rsnapshot

Posted: Sun Apr 05, 2009 7:02 pm
by crushdepth
I'm looking at buying a TS-439 as a file server for work, and a TS-639 as a backup server (for the 439). Our office needs a decent level of backup protection, so I would like to retain:

1. Daily backups, for the last seven days.
2. Weekly backups, for the last 4 weeks.
3. Monthly backups, for the last 3 months.

I was thinking of implementing this using rsync and snapshots. Since the QNAP admin interface doesn't support snapshots, could I run the job manually via a shell script?

Thanks.

Re: Daily incremental backup: rsync + snapshots via shell script

Posted: Sun Apr 05, 2009 7:43 pm
by steckel
There is an IPKG-Package called rSnapshot http://www.rsnapshot.org. You can run it as cronjob periodically. It automatically removes/merges superfluous backups when backup frequency changes.

On the rsnapshot homepage they say
rsnapshot was originally based on an article called Easy Automated Snapshot-Style Backups with Linux and Rsync, by Mike Rubel.
so I would say it is more advanced than your initial proposal.

I personally have used it some years ago and it worked very well.

Re: Daily incremental backup: rsync + snapshots via shell script

Posted: Sun Apr 05, 2009 11:40 pm
by crushdepth
Thanks, rSnapshot would be perfect, if its possible. I have the Optware IPKG installed now, looking at the list of feeds, it looks they are model (or processor) specific.

Anyone know if there are TS-439 specific feeds available?

Re: Daily incremental backup: rsync + snapshots via shell script

Posted: Fri Apr 10, 2009 1:17 pm
by crushdepth
Ok I've just about got this working now - having a few problems getting my cron jobs to persist after reboot. rsnapshot is *excellent*. Will write up a detailed guide on how to implement this shortly.

Edit: It works!!! The first run of rsnapshot (rsync) is horribly slow and CPU intensive - took over an hour. But the subsequent runs only take a few seconds. Cool!!!!

Re: Daily incremental backup: rsync + snapshots via shell script

Posted: Mon Apr 20, 2009 12:09 pm
by adrian.mallett
When is the detailed write up due? :D

Cheers,

Adrian

Re: Daily incremental backup: rsync + snapshots via shell script

Posted: Tue Apr 21, 2009 12:52 am
by crushdepth
Ok the tutorial on installation of rsnapshot on the TS-439 is available for download. You will also need to download the official rsnapshot HOWTO which provides a tutorial on configuration. rsnapshot really is great if you want to create an automatically rotating time-series of backups, or if you want an easy way to automate rsync.

Summary

rsnapshot is a file system backup utility based on rsync. Using rsnapshot is possible to take snapshots of your filesystems at different points in time. Using hard links, rsnapshot creates the illusion of multiple full backups, while only taking up the space of one full backup plus changes. When coupled with ssh, it is possible to take snapshots of remote file systems as well. This document describes how to set up rsnapshot on the QNAP TS-439 (but I imagine it would work on all TS-x39 series) of NAS boxes with a view to schedule a nested and rotating series of incremental backups, eg:
• Hourly incremental backups, for the past 24 hours.
• Daily incremental backups, for the past 7 days.
• Weekly incremental backups, for the past 4 weeks.
• Monthly incremental backups, for the past 3 months.

Re: Daily incremental backup: rsync + snapshots via shell script

Posted: Fri Apr 24, 2009 3:48 pm
by filiphw
..the domain seems to be down... cannot download it =(

Re: Daily incremental backup: rsync + snapshots via shell script

Posted: Fri Apr 24, 2009 3:49 pm
by Q
that's really great, thanks for the tutorial! :) would love to see such functionality integrated by default with a nice gui because versioning and snapshots is highly demanded by enterprise customers as well as home users these days... HINT @QNAP :mrgreen:

@filiphw: download worked for me, maybe just try again

Re: Daily incremental backup: rsync + snapshots via shell script

Posted: Fri Apr 24, 2009 4:03 pm
by filiphw
Tried it again without success... can anyone attach the pdf to this forum thread?

Re: Daily incremental backup: rsync + snapshots via shell script

Posted: Fri Apr 24, 2009 4:10 pm
by Q
sure

Re: Daily incremental backup: rsync + snapshots via shell script

Posted: Fri Apr 24, 2009 4:13 pm
by filiphw
Tnx

Re: Daily incremental backup: rsync + snapshots via shell script

Posted: Fri Apr 24, 2009 5:40 pm
by crushdepth
Yeah sorry it looks like my website is having some DNS problems. Thanks for posting the zip file!
would love to see such functionality integrated by default with a nice gui because versioning and snapshots is highly demanded by enterprise customers as well as home users these days... HINT @QNAP
I agree. Simple replication just isn't enough for business users. Having some versioning / history in your backup is essential. rsnapshot is a mature and well-known tool which is easy to use. A killer feature that would differentiate QNAP from other NAS! Please integrate it into the web interface!

Re: Daily incremental backup: rsync + snapshots via shell script

Posted: Fri May 01, 2009 8:39 pm
by dsp
this is great thanks , i have just posted to find out if i can replicate multiple boxes to 1 main box, did you have any need for that as a function ? :)

Re: Daily incremental backup: rsync + snapshots via shell script

Posted: Sat May 02, 2009 12:30 pm
by crushdepth
I think you could. You can define what directories you want backed up in the rsnapshot config file, and according to the howto that includes remote file systems. So you could probably s_uck* in several NAS to a bigger one (but I haven't tried it).

*Note to mods: s_uck is not a swear word, it does have legitimate uses. Please don't filter it :-)

rsnapshot: howto guide for TS-x39

Posted: Thu May 28, 2009 11:31 pm
by crushdepth
I've updated the rsnapshot installation howto (attached) with a section on how to set up key-based authentication. This enables rsnapshot jobs to run over SSH without requiring password login to remote servers.

It's now a fully-functioning solution for backup with versioning on your QNAP. Enjoy!

[Edit: Please see the revised howto in post below, I have removed the old version from this post as it contained an error]