Backup to External free space problems with large files

[csgardner]

This is happening on a TS-459 Pro II on 3.6.1 firmware with a USB 3.0 external drive attached. We use Veeam to grab copies of VMs and put them on a share on the QNAP's array. I then set up a job to backup that folder to the external. Normally it works well.

The problem I've run into is that if the free space on the external is less than the largest VM, the job fails with this error:

[External Drive] Job BackupToExternal to USB failed: Job is aborted because there is no space left on the destination device!

This error was occurring with as much as 250GB of free space on the external drive. The largest VM is just over 250GB. It appears that the changed file is getting written to the external before the old file is deleted, but that's causing problems for this use case. I'm having to waste 1/4 of the drive to account for this overhead. Is there any way to get the backup to delete the old file before writing the changed file to the external drive?

[Mike Goble]

I have this same backup syncing problem also. To get around it I wrote a script to erase the external drive connected to the QNAP before the copy procedure was scheduled to start. This script needs to be run by a user with administration rights on the QNAP as the backup folder is created with admin 755 rights although the parent folder has 777 rights. It would be great if there was an option to erase the target folder of the sync pair in the backup before begining the copying.

[theop1]

I'm having the same issue but would like to try and work around it without the need to run a script from a seperate device. Is there a way to create and run a script on the qnap? Any other ideas?

[steve_nas]

QNAP, how about a fix for this? Or maybe an extra option which deletes extra files *before* copying the new files, if you feel this should not be the standard behavior...

We're using QNAP NASes as backup targets at multiple client sites (most of them TS419U), and are doing offsite backups to external USB drives via the QNAPs build-in backup function. Now that the backup volume is getting larger, we're hitting this problem very often and are forced to manually delete files and rerun the job by hand.

I'm afraid that just running a script that deletes the disk before the backup job would not be the right fix here, because

1) this would considerably increase backup time, as some large unchanged files would have to be copied again
2) the script would have to be installed again each time we do a firmware update

Best Regards,
Steve

[P3R]

Is there a way to create and run a script on the qnap? Any other ideas?

For those that aren't script gurus in Linux a very simple workaround would be to precede each job that suffer from this space issue with a special backup job with an empty folder as source and with the delete extra files option selected. That way the backup target will be cleared before the real backup job runs. Of course this would have all the same negative effects as a simple script deleting all files but it would be:
1. Working today, while you wait for Qnap to revise the external backup feature (it may not be a simple change depending on the underlying technology used).
2. Manageable from the same external backup gui.
3. Fast and easy to deploy, anyone that can setup a backup job can also use the workaround.

Maybe it could also be developed into something more refined by using filtering and other advanced options in the external backup? I haven't looked at it as I don't need the workaround myself.

[ccsjohn]

Hi there
Is there any official word on when the updated firmware with this essential functionality will be released?
We have a number of clients using this feature and would appreciate any official updates.
I am aware that we could do as suggested by P3R and wipe the external drive prior to every sync, but copying >1TB files every single time seems unnecessary and prone to errors.
Thank you in advance

[theop1]

Is there a way to create and run a script on the qnap? Any other ideas?

For those that aren't script gurus in Linux a very simple workaround would be to precede each job that suffer from this space issue with a special backup job with an empty folder as source and with the delete extra files option selected. That way the backup target will be cleared before the real backup job runs. Of course this would have all the same negative effects as a simple script deleting all files but it would be:
1. Working today, while you wait for Qnap to revise the external backup feature (it may not be a simple change depending on the underlying technology used).
2. Manageable from the same external backup gui.
3. Fast and easy to deploy, anyone that can setup a backup job can also use the workaround.

Maybe it could also be developed into something more refined by using filtering and other advanced options in the external backup? I haven't looked at it as I don't need the workaround myself.

Thanks P3R. This is the most elegant solution for me. Backup takes a little longer but at least it works.

[steve_nas]

Thanks P3R, this is certainly less hacky than using a script...

Anyone know if this is scheduled to be fixed in a future firmware release? If not, where would be the right place to post a feature request with QNAP?

[P3R]

Anyone know if this is scheduled to be fixed in a future firmware release?

I can't remember seeing it being mentioned by any Qnap employee as a work in progress or even scheduled for the future so I would keep expectations at a low level.

To call it a fix probably isn't correct as it is a request for changed functionality.

If not, where would be the right place to post a feature request with QNAP?

That would be the Features Wanted section.
'

[pete_c20]

I Have a 2TB TS-110 which backs up a 2TB system HDD and I can no longer back it up due to this problem. I have only backed up to it 4 times. The first time was a major file copy as it was new. The next couple of times were minor file updates. Then the source drive was updated with many files leaving 189GB free on the source drive. During the 4th backup the TS-110 ran out of space. This was due to it not deleting files that had been deleted from the source drive.

I would say that the ability to delete unwanted files from the backup is an essential feature and one I would expect of a backup system. I bought a QNAP thinking it was good and after only 4 backups it's failed. This is very poor.

The manual seems to suggest that the option to 'delete extra files' is available on scheduled backups but I don't want to run scheduled backups. I want to run backups on demand.

Also the current version of replicator (v4.0.5.0524) doesn't display its version number and F1 for help does not work

[P3R]

The manual seems to suggest that the option to 'delete extra files' is available on scheduled backups but I don't want to run scheduled backups. I want to run backups on demand.

The Delete extra files option should as far as I know be available if configured for on demand backup also. It is not clickable (but enabled anyway) when using Real-time mode.

But if selecting Delete extra files is your only issue, this thread is really about a different problem. I have also described a workaround for it above. It's not elegant but it should work.

[pete_c20]

The manual seems to suggest that the option to 'delete extra files' is available on scheduled backups but I don't want to run scheduled backups. I want to run backups on demand.

The Delete extra files option should as far as I know be available if configured for on demand backup also. It is not clickable (but enabled anyway) when using Real-time mode.

But if selecting Delete extra files is your only issue, this thread is really about a different problem. I have also described a workaround for it above. It's not elegant but it should work.

Dear P3R,
Thankyou for the reply. I'd spent several hours trying to fix my problem (of the NAS failing to backup when the NAS free space dipped below the largest source file, even though that file has not been changed and therefore doesn't need to be backed up again) and on the verge of falling asleep wrote to the thread in the hope of help. Now, after some sleep, yes you're correct this thread is about something different. This thread was one of 2 or 3 I could find on here that was similar to my problem and I went for the wrong one! I shall repost on the copy vs synch thread which hopefully be more apt.

On the subject of Delete extra files being auto enabled in real time mode. Using the current Netbak replicator (NetBak-4.0.5.0524) that is not the case. Performing a small backup as a test it still leaves a file on the NAS which has been deleted from the source drive. QNAP technical have just said it is a safety feature. So the backup is copying not synchronizing . I shall pursue this on the other thread more targeted at this.

EDITED to add....
QNAP technical have just said that 'delete extra files' only works in an auto backup job. And version Number can be displayed by clicking icon top left in NetBak.

[P3R]

On the subject of Delete extra files being auto enabled in real time mode. Using the current Netbak replicator (NetBak-4.0.5.0524) that is not the case.

Well I didn't talk about NetBak Replicator at all since that program is totally unrelated to this thread. Here we discuss the Backup to external disk feature of the NAS.

I beleive that almost all discussions here are about the above backup feature and that very few use NetBak Replicator at all, so make sure it is really the topic of any thread you try next. If you don't find a thread that fits, go ahead and open a new one.

[rendude]

Ok, I think i have a similar issue and cannot use the suggested work around for this.
Basically we are using the NAS to capture full and incremental backups of a Windows Server with shadow protect.
The NAS has 2 x 2TB mirrored drives
Attached to the NAS we have an external iTank enclosure that has a 2TB drive that is rotated weekly (There are 3x 2TB drives so there is always one offsite.)
Basically it is set to backup the NAS to the external drive nightly and then once a week the drive is manually ejected and swapped for one of the other offsite drives.
Delete Extra files is selected in the job.
The NAS is currently using around 1.3TB of the drive space, so it reports having 530GB of free storage.
It normally has around 40 SPF files with dynamic file names that are always changing. Normally 4 are around 350GB and the reset are smaller incremental backups.
now the issue is, if I erase the external, and reset up the folder pairs, it will sync fine, until we come to putting in one of the older drives that have some of the older backups on it. When that gets put in, the additional files are not deleted before the backup starts, so it fails with a disk full error, and the only way to get it going again is to manually erase the extra files first.

Now I am not a linux scripting guru, so a non scripting or very very clear instructions solution would be appriciated if anyone has some an idea if how to get around this?

[P3R]

...and cannot use the suggested work around for this.

Why?

Now I am not a linux scripting guru, so a non scripting or very very clear instructions solution would be appriciated if anyone has some an idea if how to get around this?

1. Make a scheduled backup job that syncronizes from an EMPTY source folder (you need to make sure this folder exist and is always empty on the NAS) on the NAS to the external disk backup destination folder (example /USBDisk1/Datafolder). Make sure the Delete extra files option is selected.
2. Make a scheduled (15 minutes later than the job in #1 above) backup job that syncronizes from the source folder (example /Datafolder) on the NAS to the external disk backup destination folder (example /USBDisk1/Datafolder).