[HOWTO] Daily incremental backup with rsnapshot

Discussion on remote replication.
Post Reply
crushdepth
Getting the hang of things
Posts: 75
Joined: Thu Mar 26, 2009 8:56 pm

[HOWTO] Daily incremental backup with rsnapshot

Post by crushdepth » Sun Apr 05, 2009 7:02 pm

I'm looking at buying a TS-439 as a file server for work, and a TS-639 as a backup server (for the 439). Our office needs a decent level of backup protection, so I would like to retain:

1. Daily backups, for the last seven days.
2. Weekly backups, for the last 4 weeks.
3. Monthly backups, for the last 3 months.

I was thinking of implementing this using rsync and snapshots. Since the QNAP admin interface doesn't support snapshots, could I run the job manually via a shell script?

Thanks.
Last edited by crushdepth on Tue Jun 09, 2009 4:28 pm, edited 1 time in total.

steckel
Starting out
Posts: 15
Joined: Mon Mar 30, 2009 4:00 am

Re: Daily incremental backup: rsync + snapshots via shell script

Post by steckel » Sun Apr 05, 2009 7:43 pm

There is an IPKG-Package called rSnapshot http://www.rsnapshot.org. You can run it as cronjob periodically. It automatically removes/merges superfluous backups when backup frequency changes.

On the rsnapshot homepage they say
rsnapshot was originally based on an article called Easy Automated Snapshot-Style Backups with Linux and Rsync, by Mike Rubel.

so I would say it is more advanced than your initial proposal.

I personally have used it some years ago and it worked very well.

crushdepth
Getting the hang of things
Posts: 75
Joined: Thu Mar 26, 2009 8:56 pm

Re: Daily incremental backup: rsync + snapshots via shell script

Post by crushdepth » Sun Apr 05, 2009 11:40 pm

Thanks, rSnapshot would be perfect, if its possible. I have the Optware IPKG installed now, looking at the list of feeds, it looks they are model (or processor) specific.

Anyone know if there are TS-439 specific feeds available?

crushdepth
Getting the hang of things
Posts: 75
Joined: Thu Mar 26, 2009 8:56 pm

Re: Daily incremental backup: rsync + snapshots via shell script

Post by crushdepth » Fri Apr 10, 2009 1:17 pm

Ok I've just about got this working now - having a few problems getting my cron jobs to persist after reboot. rsnapshot is *excellent*. Will write up a detailed guide on how to implement this shortly.

Edit: It works!!! The first run of rsnapshot (rsync) is horribly slow and CPU intensive - took over an hour. But the subsequent runs only take a few seconds. Cool!!!!

adrian.mallett
Starting out
Posts: 44
Joined: Tue Mar 31, 2009 8:26 am

Re: Daily incremental backup: rsync + snapshots via shell script

Post by adrian.mallett » Mon Apr 20, 2009 12:09 pm

When is the detailed write up due? :D

Cheers,

Adrian

crushdepth
Getting the hang of things
Posts: 75
Joined: Thu Mar 26, 2009 8:56 pm

Re: Daily incremental backup: rsync + snapshots via shell script

Post by crushdepth » Tue Apr 21, 2009 12:52 am

Ok the tutorial on installation of rsnapshot on the TS-439 is available for download. You will also need to download the official rsnapshot HOWTO which provides a tutorial on configuration. rsnapshot really is great if you want to create an automatically rotating time-series of backups, or if you want an easy way to automate rsync.

Summary

rsnapshot is a file system backup utility based on rsync. Using rsnapshot is possible to take snapshots of your filesystems at different points in time. Using hard links, rsnapshot creates the illusion of multiple full backups, while only taking up the space of one full backup plus changes. When coupled with ssh, it is possible to take snapshots of remote file systems as well. This document describes how to set up rsnapshot on the QNAP TS-439 (but I imagine it would work on all TS-x39 series) of NAS boxes with a view to schedule a nested and rotating series of incremental backups, eg:
• Hourly incremental backups, for the past 24 hours.
• Daily incremental backups, for the past 7 days.
• Weekly incremental backups, for the past 4 weeks.
• Monthly incremental backups, for the past 3 months.

filiphw
Starting out
Posts: 12
Joined: Thu Apr 16, 2009 8:04 pm

Re: Daily incremental backup: rsync + snapshots via shell script

Post by filiphw » Fri Apr 24, 2009 3:48 pm

..the domain seems to be down... cannot download it =(
// Filiph
Ts-119
Ts-109 II

User avatar
Q
Experience counts
Posts: 1441
Joined: Wed Sep 05, 2007 4:18 pm
Location: Switzerland
Contact:

Re: Daily incremental backup: rsync + snapshots via shell script

Post by Q » Fri Apr 24, 2009 3:49 pm

that's really great, thanks for the tutorial! :) would love to see such functionality integrated by default with a nice gui because versioning and snapshots is highly demanded by enterprise customers as well as home users these days... HINT @QNAP :mrgreen:

@filiphw: download worked for me, maybe just try again
I am Q
www.qnap.ch

I don't work at QNAP.
And RAID is really NO backup!

filiphw
Starting out
Posts: 12
Joined: Thu Apr 16, 2009 8:04 pm

Re: Daily incremental backup: rsync + snapshots via shell script

Post by filiphw » Fri Apr 24, 2009 4:03 pm

Tried it again without success... can anyone attach the pdf to this forum thread?
// Filiph
Ts-119
Ts-109 II

User avatar
Q
Experience counts
Posts: 1441
Joined: Wed Sep 05, 2007 4:18 pm
Location: Switzerland
Contact:

Re: Daily incremental backup: rsync + snapshots via shell script

Post by Q » Fri Apr 24, 2009 4:10 pm

sure
You do not have the required permissions to view the files attached to this post.
I am Q
www.qnap.ch

I don't work at QNAP.
And RAID is really NO backup!

filiphw
Starting out
Posts: 12
Joined: Thu Apr 16, 2009 8:04 pm

Re: Daily incremental backup: rsync + snapshots via shell script

Post by filiphw » Fri Apr 24, 2009 4:13 pm

Tnx
// Filiph
Ts-119
Ts-109 II

crushdepth
Getting the hang of things
Posts: 75
Joined: Thu Mar 26, 2009 8:56 pm

Re: Daily incremental backup: rsync + snapshots via shell script

Post by crushdepth » Fri Apr 24, 2009 5:40 pm

Yeah sorry it looks like my website is having some DNS problems. Thanks for posting the zip file!

would love to see such functionality integrated by default with a nice gui because versioning and snapshots is highly demanded by enterprise customers as well as home users these days... HINT @QNAP


I agree. Simple replication just isn't enough for business users. Having some versioning / history in your backup is essential. rsnapshot is a mature and well-known tool which is easy to use. A killer feature that would differentiate QNAP from other NAS! Please integrate it into the web interface!

dsp
New here
Posts: 4
Joined: Fri May 01, 2009 7:16 pm

Re: Daily incremental backup: rsync + snapshots via shell script

Post by dsp » Fri May 01, 2009 8:39 pm

this is great thanks , i have just posted to find out if i can replicate multiple boxes to 1 main box, did you have any need for that as a function ? :)

crushdepth
Getting the hang of things
Posts: 75
Joined: Thu Mar 26, 2009 8:56 pm

Re: Daily incremental backup: rsync + snapshots via shell script

Post by crushdepth » Sat May 02, 2009 12:30 pm

I think you could. You can define what directories you want backed up in the rsnapshot config file, and according to the howto that includes remote file systems. So you could probably s_uck* in several NAS to a bigger one (but I haven't tried it).

*Note to mods: s_uck is not a swear word, it does have legitimate uses. Please don't filter it :-)

crushdepth
Getting the hang of things
Posts: 75
Joined: Thu Mar 26, 2009 8:56 pm

rsnapshot: howto guide for TS-x39

Post by crushdepth » Thu May 28, 2009 11:31 pm

I've updated the rsnapshot installation howto (attached) with a section on how to set up key-based authentication. This enables rsnapshot jobs to run over SSH without requiring password login to remote servers.

It's now a fully-functioning solution for backup with versioning on your QNAP. Enjoy!

[Edit: Please see the revised howto in post below, I have removed the old version from this post as it contained an error]
Last edited by crushdepth on Sun Jun 14, 2009 1:09 pm, edited 2 times in total.

Post Reply

Return to “Remote Replication/ Disaster Recovery”