HBS3 Backup does not check the storage is still complete

[pbrooks]

I've been running offsite backups to DropBox and to Google cloud.

In the Dropbox website I went into the destination files from the backup, and deleted one folder. I expected that the next time HBS3 on the QNAP ran, it would detect those files were no longer there, and would back them up again to replace them and re-instate the backup cleanly.
It didn't - it ran briefly and finished successfully, without uploading any files.

It appears that when HBS3 does a backup, it does not ever actually check that the backup location still contains the files that its local database says should be there from previous runs.

Now I have no way to re-instate those files in backup storage and repair the backup job, other than to delete the ENTIRE backup store, re-build a new backup job, and re-backup and upload the entire set of files (~15,000 folders) even though its only one folder to be reinstated.

Also, I haven't found any way that a backup job can verify the storage still contains the backup fileset. It appears the program blindly trusts its local database of what has been backed up previously, and there is no way to verify and update the local database in the event some of the destination files become corrupted or missing. Sort of like a Restore job scanning through the backup files, flagging files that it can or cannot restore, but without actually downloading all the files.

This makes a incremental backup highly risky. If something else corrupts or deletes a portion of the data in the backup store, you can never discover this until you try to restore the data and cannot.

Does anyone else know of a way to rebuild the local database based on what is actually still in the backed-up filestore, so that the next backup can fill in the missing or corrupted files?

[P3R]

It appears that when HBS3 does a backup, it does not ever actually check that the backup location still contains the files that its local database says should be there from previous runs.

I don't know how many complaints from upset users I've seen here about HBS doing downloads from the cloud (taking very long and eating up the users limited download budget) when people was doing a backup to the cloud.

Now HBS apparently trust the cloud to be very reliable, build some kind of local database and doesn't check what the destination contain and then you're upset about HBS not checking what's in the backup.

HBS obviously can't check what's in the destination if it's not allowed to download the entire content so it's impossible to satisfy both types of wishes. That's the dilemma when using cheap bulk storage (which is what almost all users want) as a backup destination and not an intelligent client/server type backup service.

Now I have no way to re-instate those files in backup storage and repair the backup job, other than to delete the ENTIRE backup store, re-build a new backup job, and re-backup and upload the entire set of files (~15,000 folders) even though its only one folder to be reinstated.

A simple workaround that probably will work is to remove the same folder locally on the original data, run the backup, add the same folder back and finally run the backup again.

Also, I haven't found any way that a backup job can verify the storage still contains the backup fileset. It appears the program blindly trusts its local database of what has been backed up previously, and there is no way to verify and update the local database in the event some of the destination files become corrupted or missing. Sort of like a Restore job scanning through the backup files, flagging files that it can or cannot restore, but without actually downloading all the files.

Yes a verify feature or simply adding a flag in the restore job to do a dry run and not actually change anything on the source could be a good addition (open a case with Qnap and suggest it as a feature request). I have a question though, how would it be possible for HBS to scan through the backup files, "flagging files that it can or cannot restore, but without actually downloading all the files"? How is it physically possible to check content and verify that a restore works without downloading anything from a simple bulk storage?

This makes a incremental backup highly risky.

So you don't trust the cloud service that you've chosen as your backup destination?

If something else corrupts or deletes a portion of the data in the backup store, you can never discover this until you try to restore the data and cannot.

True.

A very important part of doing backups is to verify that restore work before you actually need your backup but I don't know how to do that without doing a restore so it will most likely not be done with huge amounts of data backed up to relatively slow (at least compared to full gigabit LAN speed) cloud services.

Does anyone else know of a way to rebuild the local database based on what is actually still in the backed-up filestore, so that the next backup can fill in the missing or corrupted files?

A new backup job is the only way I know but I have very little experience with using cloud services so I would recommend that you ask Qnap support that question.

[pbrooks]

It appears that when HBS3 does a backup, it does not ever actually check that the backup location still contains the files that its local database says should be there from previous runs.

I don't know how many complaints from upset users I've seen here about HBS doing downloads from the cloud (taking very long and eating up the users limited download budget) when people was doing a backup to the cloud.

Now HBS apparently trust the cloud to be very reliable, build some kind of local database and doesn't check what the destination contain and then you're upset about HBS not checking what's in the backup.

HBS obviously can't check what's in the destination if it's not allowed to download the entire content so it's impossible to satisfy both types of wishes. That's the dilemma when using cheap bulk storage (which is what almost all users want) as a backup destination and not an intelligent client/server type backup service.

HBS can check filename, file date, filesize and other file-level parameters match, without actually downloading the files. Sure it can't do a content-level CRC to verify the contents aren't corrupted in the way that a backup to a RSYNC target can, but checking directory-level information would identify 90% of the potential issues without any bandwidth-consuming downloads.

Now I have no way to re-instate those files in backup storage and repair the backup job, other than to delete the ENTIRE backup store, re-build a new backup job, and re-backup and upload the entire set of files (~15,000 folders) even though its only one folder to be reinstated.

A simple workaround that probably will work is to remove the same folder locally on the original data, run the backup, add the same folder back and finally run the backup again.

Sure - but that requires one to know which folders or files are missing from the destination - and discovering that is part of the missing functionality.

Yes a verify feature or simply adding a flag in the restore job to do a dry run and not actually change anything on the source could be a good addition (open a case with Qnap and suggest it as a feature request). I have a question though, how would it be possible for HBS to scan through the backup files, "flagging files that it can or cannot restore, but without actually downloading all the files"? How is it physically possible to check content and verify that a restore works without downloading anything from a simple bulk storage?

By checking the destination directory still contains the same filename, date and size.

This makes a incremental backup highly risky.

So you don't trust the cloud service that you've chosen as your backup destination?

I trust the cloud storage not to corrupt the internal contents of the files without any change to the directory (its not an analog tape drive!), but I don't want to trust that no other process (or a manual intervention) hasn't deleted or renamed a file, or caused it to be altered/truncated (and thus the datetime-stamp of the file would have been updated).

If something else corrupts or deletes a portion of the data in the backup store, you can never discover this until you try to restore the data and cannot.

True.

A very important part of doing backups is to verify that restore work before you actually need your backup but I don't know how to do that without doing a restore so it will most likely not be done with huge amounts of data backed up to relatively slow (at least compared to full gigabit LAN speed) cloud services.

I periodically test I can restore a small subset, chosen randomly each time from the universe of the whole backup. testing of 2 random folders out of 100 will be restored successfully gives confidence the remainder will likely restore also if required.

Does anyone else know of a way to rebuild the local database based on what is actually still in the backed-up filestore, so that the next backup can fill in the missing or corrupted files?

A new backup job is the only way I know but I have very little experience with using cloud services so I would recommend that you ask Qnap support that question.

Thanks for your assistance, I will ask QNAP support for a 'validation' function. Even if it has to download everything, if it can verify that only a small subset of data needs to be re-uploaded that will be a win, since uploading is far more bandwidth-constrained than downloading in my case on a residential broadband line. I can download the whole dataset relatively quickly without maxing the line and affecting other users during the process. But constraining the upload to only use 50% of the line capacity to keep some bandwidth for other users makes this worth optimising.

[schopi68]

the same here:

I did a backup to B2 Cloud storage. During the backup the snapshot-storage of the NAS became full, so the drive went to r/o mode and the backup stopped.

After recovering from the r/o mode of the drive, the backup has been manually started again and finished fine.

But: when doing a restore-test, there were files missing:
"Restore jobs "Restore2": Failed to download file/folder from "....xxx". Check the file and try again. Error: the file does not exist in the cloud storage. If you did not manually delete the file, contact the service provider for details."

I could see that the file is marked as stored in the metadata (in the remote backup), but it was not backed up.
A Quick Data Integrity check and also a complete content data integrity check of the data in HBS3 tells me that everything is fine. Especially as according to the description of this check the file existence should be checked.
https://www.qnap.com/en/how-to/tutorial ... k-in-hbs-3

Later i did a complete content check and this told me too that the backup is okay. But during restore there were file missing errors...

This looks to me as if HBS3 is first storing a files metadata into the backup, marks it as uploaded and then the backup is done. And this is not checked correctly in the integrity check.

So the only conclusion of this HBS3 behaviour could be that after every failed (even incremental) backup the whole Backup has to be redone. This is not very funny on a backup with several terabytes.

[OneCD]

* topic locked to prevent further necroposting *