[HOWTO] Daily incremental backup with rsnapshot

Discussion on remote replication.
Post Reply
micke
Experience counts
Posts: 1363
Joined: Sat Feb 06, 2010 2:42 pm
Location: Taipei, Taiwan

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by micke » Thu May 26, 2011 2:38 am

virtualdj wrote:But if I access through Samba (on a Windows machine) the Public folder and go into the subfolders Backup/daily.0/localhost/share/Public/script, even if I'm logged with the user 'web' (and not admin!) I can remove the files.

That doesn't make sense. If the file system is read-only then it shouldn't matter how you access it, but I don't use samba, so I have no idea how it works on QNAP devices...

/Mike

virtualdj
Experience counts
Posts: 1895
Joined: Wed May 26, 2010 2:44 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by virtualdj » Thu May 26, 2011 2:50 am

Yeah, you're right. That doesn't make sense.
If you don't use Samba I suppose that you use NFS: can you briefly explain me how to export the /opt/var/rsnapshot folder to "localhost" using NFS on QNAP? That should do the trick.

micke
Experience counts
Posts: 1363
Joined: Sat Feb 06, 2010 2:42 pm
Location: Taipei, Taiwan

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by micke » Thu May 26, 2011 3:34 am

virtualdj wrote:Yeah, you're right. That doesn't make sense.

I activated samba on my NAS and connected to the read-only file system from a linux client. Not possible to remove files...
virtualdj wrote:can you briefly explain me how to export the /opt/var/rsnapshot folder to "localhost" using NFS on QNAP?

I guess you would have to make the kind of manual modifications that were described in the link you referred to, but as soon as you start making manual modifications to these kinds of files you take the risk of them being overwritten by the system at any time...

Would probably be a better idea to figure out why the read-only file system + samba combination doesn't work as expected on your NAS.

/Mike

virtualdj
Experience counts
Posts: 1895
Joined: Wed May 26, 2010 2:44 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by virtualdj » Fri May 27, 2011 1:17 am

Pardon my insistence (and my long post), but I'm trying to understand.
I'm doing some tests with rsnapshot, so I configured the /opt/etc/rsnapshot.conf like this (removing unnecessary lines):

Code: Select all

# All snapshots will be stored under this root directory.
snapshot_root   /opt/var/rsnapshot/

# LOCALHOST
backup  /share/Public/script/   localhost/

So the only data that I'm backing up is the folder script under the Public share. I put 3 text files (File1.txt, File2.txt and File3.txt) into that folder.
Then I run 2 times rsnapshot:

Code: Select all

[/] # rsnapshot hourly
[/] # rsnapshot hourly
[/] # ls -l /opt/var/rsnapshot/
drwxr-xr-x    3 admin    administ     4096 May 26 19:07 hourly.0/
drwxr-xr-x    3 admin    administ     4096 May 26 19:08 hourly.1/

OK, that's right. Now I want to mount this folder in read-only mode under the Backup dir of the Public folder (another time I've tried also not saving it under Public, but the results are the same!). So I follow your tips:

Code: Select all

[/] # mkdir /share/Public/Backup
[/] # ls -l /share/Public/
drwxr-xr-x    2 admin    administ     4096 May 26 19:11 Backup/
[/] # /bin/mount --bind /opt/var/rsnapshot /share/Public/Backup
[/] # /bin/mount -o remount,ro /share/Public/Backup
[/] # mount | grep Backup
/share/HDA_DATA/.qpkg/Optware/var/rsnapshot on /share/HDA_DATA/Public/Backup type none (ro,bind)

Right, the folder is mounted read-only. Now I can try to delete any file under SSH also by root and it (correctly) doesn't work:

Code: Select all

[/] # rm -f /share/Public/Backup/hourly.0/localhost/share/Public/script/File1.txt
rm: unable to remove `/share/Public/Backup/hourly.0/localhost/share/Public/script/File2.txt': Read-only file system

OK, now I go to the Windows computer (but I think that a Linux with Samba will be the same) and browse the Public/Backup share.
This is the folder tree:

Code: Select all

\\QNAP\Public\Backup.
├───hourly.0
│   └───localhost
│       └───share
│           └───Public
│               └───script
│                   └───File1.txt
│                   └───File2.txt
│                   └───File3.txt
└───hourly.1
    └───localhost
        └───share
            └───Public
                └───script
                    └───File1.txt
                    └───File2.txt
                    └───File3.txt

Now the problem raises:
  • If I try to delete the hourly.0 folder, the system won't (OK)
  • If I try to delete the localhost folder inside hourly.0, the system won't (OK)
  • If I try to delete the share folder inside hourly.0/localhost, the system won't (OK)
  • If I try to delete the Public folder inside hourly.0/localhost/share, the system don't delete the folder (OK) but it removes all the files and the script subfolder (NOOO!)
  • If I try to delete, for example, File2.txt inside hourly.1/localhost/share/Public/script the file is erased (NOOO!)

I know that I bother you so much, but if you could try to reproduce this behaviour on your TS-239 I will thank you a lot.
I don't think I'm missing anything... maybe it's a bug on Samba?

virtualdj
Experience counts
Posts: 1895
Joined: Wed May 26, 2010 2:44 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by virtualdj » Sat May 28, 2011 2:44 am

I've done further investigations. At least I know now why I can delete the files inside hourly.0/localhost/share and the subfolders: they were owned by the "web" user and I was logged in with the "web" account from Windows.
So, if I log-in from Windows using the admin account, I actually can delete ALL the files and the folders of the read-only mounted dir.

To summarize, this is the behaviour: using a remounted read-only folder, you CANNOT add NOR edit/rename any of the file/folders inside the read-only folder, but the USER that OWNs a file/folder CAN DELETE them from within Samba (but cannot do the same under SSH).
It's like a "delete permission"...

Once again, I think that this is a bug; I'm just surprised that nobody has discovered that before. :|

micke
Experience counts
Posts: 1363
Joined: Sat Feb 06, 2010 2:42 pm
Location: Taipei, Taiwan

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by micke » Sat May 28, 2011 3:17 am

virtualdj wrote:but the USER that OWNs a file/folder CAN DELETE them from within Samba

Not on my system with samba (using a Linux client).

/Mike

virtualdj
Experience counts
Posts: 1895
Joined: Wed May 26, 2010 2:44 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by virtualdj » Sat May 28, 2011 5:05 pm

micke wrote:Not on my system with samba (using a Linux client).

Thank you again Mike, but it's not Windows' fault.
I installed Ubuntu 10.010 on a virtual machine, accessed through "Windows Network" (i.e. Samba) the QNAP share with the read-only mount using the admin user, and as on Windows, I cannot create files but I can delete all the files!

So if you don't report this behaviour it should be something different on my QNAP Samba configuration?
Definitely it's not on the shared folder, because I've tried to put the read-only mount on another share (/share/SambaTest) with same results; the only way to prevent this behaviour is setting the whole shared folder as read-only also to the admin user, but that isn't what I want.

I have "Enable Advanced Folder Permissions" disabled and "Enable Folder Aggregation" enabled but with the list of folders empty.
This is my smb.conf (cut down for brevity):

Code: Select all

[global]
workgroup = WORKGROUP
security = USER
server string = NAS Server
        encrypt passwords = Yes
username level = 0
        map to guest = Bad User
null passwords = yes
        max log size = 10
socket options = TCP_NODELAY SO_KEEPALIVE SO_SNDBUF=262144 SO_RCVBUF=131072
os level = 20
preferred master = no
        dns proxy = No
        config file = /etc/config/smb.conf
        smb passwd file=/etc/config/smbpasswd
        username map = /etc/config/smbusers
        guest account = guest
        directory mask = 0777
        create mask = 0777
oplocks = yes
        locking = yes
        disable spoolss = yes
        load printers = no
        dos charset = UTF8
display charset = UTF8
force directory security mode = 0000
        template shell = /bin/sh
veto files = /.AppleDB/.AppleDouble/.AppleDesktop/:2eDS_Store/Network Trash Folder/Temporary Items/TheVolumeSettingsFolder/.@__thumb/.@__desc/:2e*/
        delete veto files = yes
map archive = no
map system = no
map hidden = no
map read only = no
deadtime = 10
use sendfile = yes
case sensitive = auto
unix extensions = no
min receivefile size = 4096
wins support = no
passdb backend = smbpasswd
store dos attributes = yes
client ntlmv2 auth = yes
dos filetime resolution = yes
domain master = auto
local master = yes
inherit acls = yes
wide links = yes

<... removed ...>

[Public]
comment = System default share
path = /share/HDA_DATA/Public
browsable = yes
oplocks = yes
ftp write only = no
public = yes
invalid users =
read list =
write list = "admin",@"everyone","guest","web"
valid users = "root","admin",@"everyone","guest","web"
inherit permissions = yes

[SambaTest]
comment =
path = /share/HDA_DATA/SambaTest
browsable = yes
oplocks = yes
ftp write only = no
public = yes
invalid users =
read list =
write list = "guest","admin",@"everyone"
valid users = "root","admin",@"everyone","guest"
inherit permissions = yes

I'm on 3.4.2 build 0331T firmware. Have you got any other clue? There must be something that I'm missing...

ectospasm
Starting out
Posts: 20
Joined: Thu Oct 07, 2010 5:21 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by ectospasm » Sat May 28, 2011 5:56 pm

virtualdj wrote:

Code: Select all

[/opt/var] # ls -l rsnapshot/hourly.0/localhost/share/Public/script/
-rwxrwxrwx    2 admin    administ      735 Nov 24 21:04 alreadyrunning.sh*
drwxrwxrwx    2 web      everyone     4096 May 24 21:12 empty/
-rwxrwxrwx    2 admin    administ     8644 Dec  7 21:33 jdn.sh*



Notice the Linux/UNIX permissions of these directories. They are world writable, which means any user that can read the parent directory can delete the files. This is not a bug, but a mishandling of directory permissions. Note that the rsnapshot manual recommends that directory permissions of the snapshot root be 0700, which equates to directory permissions of drwx------, not drwxrwxrwx like you've got it. Please look elsewhere for a discussion of directory permissions in UNIX/Linux, there are many, many guides.

virtualdj
Experience counts
Posts: 1895
Joined: Wed May 26, 2010 2:44 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by virtualdj » Sat May 28, 2011 9:54 pm

ectospasm wrote:They are world writable, which means any user that can read the parent directory can delete the files.

ectospasm wrote:Note that the rsnapshot manual recommends that directory permissions of the snapshot root be 0700, which equates to directory permissions of drwx------, not drwxrwxrwx like you've got it.

Yes, it could theoretically be true, but we're not talking about permission, we're talking about a read-only mount.

If what you've said would apply to those folders, that would be the same also under SSH. But, as I already said before, the "delete" works only under Samba (both from Windows and Linux) and not under SSH (with admin/root user):

Code: Select all

[/] # id
uid=0(admin) gid=0(administrators) groups=0(administrators),100(everyone)
[/] # rm -f /share/Public/Backup/hourly.0/localhost/share/Public/script/File1.txt
rm: unable to remove `/share/Public/Backup/hourly.0/localhost/share/Public/script/File2.txt': Read-only file system

Anyway I've tried to apply "chmod 700" to all the folders... with the same results!
If you have further tips, I will be glad to hear them! Thank you.

Magnus8513
Getting the hang of things
Posts: 83
Joined: Wed Nov 24, 2010 8:06 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by Magnus8513 » Sat May 28, 2011 11:40 pm

@virtualdj: I ran into the exact same Problem on my Qnaps some time ago, but didn't have the time to investigate it further. I am pleased to read your posts and hope that this can be resolved.
---
2 x TS-110, WD RE3 WD1002FBYS 1TB // Seagate 7200.11 ST3500320AS 0.5 TB f/w: SD25
2x TS-119P+, WD RE4 WD1003FBYX 1TB
2x TS-119P II, WD RE4 WD1003FBYX 1TB // Seagate 7200.14 ST2000DM001 2 TB f/w: CC4H
f/w: 3.6.1; SABNZB+ 0.6.14; Twonkey 6.0.38; rsnap 3.1.2;

ectospasm
Starting out
Posts: 20
Joined: Thu Oct 07, 2010 5:21 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by ectospasm » Sun May 29, 2011 8:04 am

virtualdj,

This is a separate issue, not one concerning rsnapshot. You should post this as a separate topic, in the appropriate forum. Trying to get help for it here will just muddy the waters further.

ectospasm
Starting out
Posts: 20
Joined: Thu Oct 07, 2010 5:21 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by ectospasm » Sun May 29, 2011 9:25 am

crushdepth,

This howto saved me at least an hour figuring this out by myself. I have experience with setting up rsnapshot on a standard Linux system, the gotchas (i.e., crontab, autorun) saved me a lot of time. The main part that would have killed me is the installation of QPKG/ipkg. That saved me a ton in research!

I would recommend some minor changes to your HOWTO. I did notice a slight error in your autorun.sh configuration. You're missing a space between "/dev/sdx6" and "/tmp/config". Also, I'd recommend listing the SSH configuration first. For that, I had to edit cmd_ssh to be the QNAP built-in (/usr/bin/ssh rather than /opt/bin/ssh). Otherwise, the SSH keys wouldn't work.

virtualdj
Experience counts
Posts: 1895
Joined: Wed May 26, 2010 2:44 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by virtualdj » Wed Jun 01, 2011 12:48 am

ectospasm wrote:This is a separate issue.

Right, I though it was a simpler problem and didn't want to "dirty" this thread, sorry!
I've opened a new thread here, for anyone that may be interested (and anyone that may help me... :D).

stepir
Getting the hang of things
Posts: 71
Joined: Mon Mar 28, 2011 4:02 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by stepir » Wed Jun 01, 2011 6:12 pm

sorry for the silly question.... can anyone point me to the rsync log used by rsnapshot? I'd like to check which files are actually copied as this is not included in the rsnapshot log itself.

The rsync config file in /etc/config/rsyncd.conf doesn't include the option log file = /var/log/rsyncd.log and even after adding it won't produce any output (and by the way I've noticed it get lost at every reboot).

thanks!

ectospasm
Starting out
Posts: 20
Joined: Thu Oct 07, 2010 5:21 am

Re: [HOWTO] Daily incremental backup with rsnapshot

Post by ectospasm » Wed Jun 01, 2011 7:12 pm

stepir wrote:sorry for the silly question.... can anyone point me to the rsync log used by rsnapshot? I'd like to check which files are actually copied as this is not included in the rsnapshot log itself.

The rsync config file in /etc/config/rsyncd.conf doesn't include the option log file = /var/log/rsyncd.log and even after adding it won't produce any output (and by the way I've noticed it get lost at every reboot).

thanks!


You can probably get this by increasing the log level for rsnapshot. This is only a guess, however. You may also be able to redirect rsync output to a file, but I'm not sure where to set this.

Post Reply

Return to “Remote Replication/ Disaster Recovery”