HBS3 Versioning Forces QDFF Format

Post Reply
Jaredlee03
New here
Posts: 2
Joined: Tue Oct 13, 2020 4:52 am

HBS3 Versioning Forces QDFF Format

Post by Jaredlee03 » Wed Oct 14, 2020 2:47 am

I am a new user to the Qnap forums, but a long time user of a Qnap NAS. I am setting up my new TS-251D and testing new back-up jobs to Amazon S3. I have read and understand how Qudedup works, however, I am not interested in that feature and I want to ensure that the back-up data in my S3 bucket remains in a non-QDFF format as I would like to have the option to restore single files (rather than an entire shared folder as required by HBS3 currently) and be able to read my data myself without the Qnap device. I am running into an interesting behavior with HBS3:

If I create a basic back-up test job to Amazon S3 with the following options, the back-up is created in my target S3 bucket in a folder structure which mirrors my source device and in a non-QDFF format:
Versioning: Disabled
Client-side Encryption: Disabled
Compression: Disabled
Qudedup: Disabled

However, if I run a test back-up job to Amazon S3 with the following options (mainly any versioning offered in HBS3), the back-up is created in the QDFF format:
Versioning: Enabled (happens with both Simple and Smart Versioning)
Client-side Encryption: Disabled
Compression: Disabled
Qudedup: Disabled

It seems that HBS3 requires the data to be in QDFF format when using the built in versioning options, even when Qudedup is disabled. Interestingly, if I perform a test back-up job with the same settings as example #2 above (either versioning type enabled) to a USB external drive (instead of Amazon S3), the data is stored in a non-QDFF format with the various versioning folders (the results I expect for a non-Qudedup backup).

Does anyone have insight into this difference in formats between Amazon S3 as the target vs a local target? Again, I'd really prefer not using the QDFF format as I would like independent, non-proprietary access to my data, even when storing in the cloud.

Note: I have a TS-251D running QTS 4.4.3.1439 and HBS3 13.0.0930

P3R
Guru
Posts: 12637
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: HBS3 Versioning Forces QDFF Format

Post by P3R » Wed Oct 14, 2020 3:32 pm

Jaredlee03 wrote:
Wed Oct 14, 2020 2:47 am
I am a new user to the Qnap forums...
Welcome!
Interestingly, if I perform a test back-up job with the same settings as example #2 above (either versioning type enabled) to a USB external drive (instead of Amazon S3), the data is stored in a non-QDFF format with the various versioning folders (the results I expect for a non-Qudedup backup).
Yes becuase the file system you choose for the external drive (Ext4?) happen to support versioning with plain files. Apparently the S3 cloud doesn't.
Again, I'd really prefer not using the QDFF format as I would like independent, non-proprietary access to my data, even when storing in the cloud.
You have the Qnap QuDedup Extract program available on Mac, Ubuntu and Windows platforms that should be able to give you access to your backup data even without a Qnap but no, it's not independent and non-proprietary.

I feel the same and that's one of the reasons I do my versioned backups to external drives and remotely to a Qnap on another location.

Personally I wouldn't use a cloud service for my backups without doing client-side encryption because I want to make reasonably sure that my data isn't read by a third party and then we're stuck with the proprietary backup format anyway... :(
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

Jaredlee03
New here
Posts: 2
Joined: Tue Oct 13, 2020 4:52 am

Re: HBS3 Versioning Forces QDFF Format

Post by Jaredlee03 » Wed Oct 14, 2020 11:51 pm

@P3R, Thank you for the reply.

I thought I read that in the past (prior to the latest version of HBS3) that smart versioning created the multiple version folders, even on Amazon S3. But I guess its now limited to using the QDFF format.

I agree completely with the client-side encryption when storing in the cloud. I do not trust Amazon with their own server side encryption with shared keys and even don't like the idea of using custom S3 keys that are also stored with Amazon. I also plan to store my data outside the U.S., to take advantage of better data privacy laws. I completely plan to use client-side encryption on my actual backup jobs and when testing using the settings of example #1 (no versioning), but with client-side encryption enabled, HBS does not force the QDFF format. So, client-side encryption will be used in my actual backup jobs.

I am in the process of setting up my 3-2-1 back-up plan for my home NAS. I have the NAS in Raid 1 (not part of the back-up plan, but adds redundancy). I will be using the new block-based Qnap Snapshots feature on reserved storage space outside the volume to allow for quick and easy versioning restores (will have many versions available) and will also serve as the first recovery option from ransom-ware (hopefully never need to). I will also back-up using smart versioning to an encrypted external HDD on a weekly basis. This drive will be stored in a safe when at home (for very short periods), but will mostly reside locked in my desk drawer at work (25 miles away). My 3rd copy will reside on Amazon S3 and is my last resort, catastrophe recovery plan. I don't really need versioning on the S3 copy, but I wanted to allow for 2 or 3 versions that roll off (not stay forever) like offered with smart versioning, in the event that ransom-ware were to encrypt files and they get backed-up to the cloud (would allow me to still have a 3rd copy of good versions).

While I have a 2nd NAS that I am pre-emptively decommissioning (due to age), I don't plan to have a remote NAS available at a 2nd location. I prefer not to open my NAS devices to the internet for incoming remote access. All my access is directly from my home LAN, only. Plus, I don't really have a friend or family that I trust to keep their network secured or updated frequently enough for my standards (not to mention updating or even isolating IoT devices from the NAS on the LAN).

My alternate plan would be to utilize Amazon S3's native versioning option (less smart/flexible). In order to limit the potential storage usage (I currently have 1TB on my NAS), I could add a life-cycle rule to set previous versions to expire after 3 days (to limit the number of versions to 3).

I know there is the QuDedup extraction tool, however, it reads the entire QDFF folder, so if I wanted to access certain files in the S3 back-up, I would have to download/restore the entire volume of information, which can exceed $100 if recovering from the S3 Glacier storage class. While this cost is worth it to me in a catastrophic scenario, I'd like a little more flexibility/accessibility in my recovery options from S3.

Thank you for your insights, I appreciate it.

chrisonnas
Starting out
Posts: 32
Joined: Sat Jan 05, 2019 5:27 pm

Re: HBS3 Versioning Forces QDFF Format

Post by chrisonnas » Wed Oct 28, 2020 2:05 am

FWIW, I use a non-smart (ie not space efficient) versioning backup for some of my small-but-regularly-changing folders. Basically it can be set to keep daily/weekly versions of the folders, which each take up storage space.

That seems to be provided by the "Backup Versioning" app which I have installed, which was a QNAP app and now seems to be missing from the app store!

I like it as you get to keep the original file structure, but of course you dont want to backup your whole NAS this way as you would need many times more storage space to hold all the different versions. Not sure if this feature is built into the HBS3 now, but you see it on the "schedule" page during setup, there is an icon on the left called "Version Management".

P3R
Guru
Posts: 12637
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: HBS3 Versioning Forces QDFF Format

Post by P3R » Wed Oct 28, 2020 3:50 am

chrisonnas wrote:
Wed Oct 28, 2020 2:05 am
FWIW, I use a non-smart (ie not space efficient) versioning backup for some of my small-but-regularly-changing folders. Basically it can be set to keep daily/weekly versions of the folders, which each take up storage space.
But you're not using S3 cloud storage are you?

As I'm not aware of versioning working on S3 cloud I will be talking about how it work on regular file systems (on a Qnap or on an external disk connected to a Qnap) below.

Only changed and new files use up additional storage space in the different versions. The versioning use file system links for all unchanged files, which make it appear as if storage is used with several tools (including File Station) but if you look at the capacity used in the actual volume you see that it actually isn't used.

Backup Station versioning is only supported on a few file systems though and at least at some point it was only stable on Ext4.
That seems to be provided by the "Backup Versioning" app which I have installed, which was a QNAP app and now seems to be missing from the app store!
Yes if still using the deprecated Backup Station, you need the versioning app to use versioning.

When updating to QTS 4.5, Backup Station will be replaced by HBS.
Not sure if this feature is built into the HBS3 now...
It is and no versioning app is required for it. In addition to that HBS have it's own backup format, used when enabling QuDedup.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

chrisonnas
Starting out
Posts: 32
Joined: Sat Jan 05, 2019 5:27 pm

Re: HBS3 Versioning Forces QDFF Format

Post by chrisonnas » Wed Oct 28, 2020 7:02 am

P3R wrote:
Wed Oct 28, 2020 3:50 am
But you're not using S3 cloud storage are you?
No, I'm very simply using local network shares and USB disks.
Yes if still using the deprecated Backup Station, you need the versioning app to use versioning.
I found that even the new HBS3 needed this Versioning App to do the simple daily / weekly collection of backups. QuDedup is included in HBS3, but was more resource-intensive (I dont have SSD for the OS, apps or any storage/cache). That might have changed with a newer version of QTS / HBS3, but I dont want to break a working system - that old app can just sit there even if its not needed now.

This might be a little off-topic for the thread, but for some S3 backups, keeping it simple might be worth considering (at the expense of needing more storage space). Needing a proprietary app to even look at your backups isnt exactly a safe place to be. You'd have to balance the inconvenience/risk of being reliant on their app against the features it brings.

P3R
Guru
Posts: 12637
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: HBS3 Versioning Forces QDFF Format

Post by P3R » Wed Oct 28, 2020 7:56 am

chrisonnas wrote:
Wed Oct 28, 2020 7:02 am
I found that even the new HBS3 needed this Versioning App to do the simple daily / weekly collection of backups.
No the versioning app isn't needed with HBS.

I've been using plain versioned backups since HBS was officially released, long before version 3 of HBS introduced QuDedup and the proprietary backup file format. I still do them on both an external drive and on a remote NAS. I don't have the versioning app installed, as it's not needed with HBS.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

Post Reply

Return to “Amazon S3”