Hybrid Backup Sync Smart Versioning Backup Questions

[P3R]

As far as I know snapshots are block based rather than file based so work the way you say. Maybe the person telling you confused the two?

[storageman]

Correct, I spoke to Qnap, seems you are right and I am wrong.
It is not tracking block changes with RTRR versioning.
10 Backups of 10MB files would give 100MB space occupied.

[AlFromCochrane]

I actually spent a great deal of time replying earlier today. I just now came back to the computer only to see a "failed to submit" error. I reviewed what I had written, made one quick edit (changed 'hear' to 'here') then after final preview, I realized an update was needed in my signature. Went to do that, bad mistake! Cam back to white. Nadda for data typed. Ouch. So again from memory:

P3R and storageman, thanks for taking your time to reply to my query. It is truly appreciated. And thanks to damenchoy for his contributions.

Wow!

Although I'm a retired IT professional, I might profess my head is going soft. Just reading through this thread several times leaves me thinking it has been made extremely difficult due to lack of solid documentation and examples. So either soft head or others are also having similar problems determining the anticipated outcome of this versioning implementation.

Hmm, several very sound theories, but no time/resources to scientifically prove and document them. Actually isn't that much like rocket science? lol!

While document versioning in its purest form is straightforward, there has been variety of implementation and when being provided as a tool to backup devices which may already be destination for a particular versioning flavor, it gets fuzzy to me. Unless it is implemented solely to support editing in place on the NAS.

On the destination storage thoughts, To my way of thinking "Smart" versioning suggests an implementation akin to what storageman describes, while perhaps full document version implementations of past decades is akin to "dumb" versioning.

P3R, your bullet form descriptions are very helpful. This whole thread is. Destination storage theories aside.

I don't have any documents being edited. My QNAP implementation is a small Family Cloud operation used to provide mainly photo backup and sharing for family (12 users at this point). There most definitely will be file moves and deletions for this operation. At some point I will be encouraging family members to view the cloud as an offsite backup for all their user data, (not OS).

Additionally,I am backing up 5 devices locally (2 PCs and 3 iOS ). I am using Macrium Reflect to accomplish this and it uses a flavor of versioning where only changes are backed up daily for 31 days then fully monthly backups (retention 2). In a nutshell this describes the data on my front end NAS. I am trying to figure out is HBC versioning is the best way to backup my front end nas to my backup nas.

Just as one example, I would be extremely interested if and how symbolic links would be used by HBS versioning. I periodically create backups to external drives formatted NTFS to be placed in a safety deposit box in case I lose both NAS. To be comfortable, I would need to determine a means to copy the most recent versions of everything, along with its proper location, to the ntfs drives. Hence my desire to learn HBS.

It would be wonderful if the developers would step up and finish the development. As a retired systems programmer, developer, and DeVry Graduate, I have been taught that development is not complete without decent documentation. Both technical and quick start.

The stuff I've found thus far on HBS provides nothing more than just browsing through the app does. perhaps dandy for some simplistic apps but backups are much too important for one to fly without proven and documented outcome.

Even educated guesses provide value given they historically provide the blueprint to finding the required proof. At this point I don't have any knowledge to share as I was just trying to learn whether I should use it or not. And unfortunately. It isn't a task that any of us could accomplish in short order, even if we had the resources (time. money) to do so. It just doesn't lend well to expedited testing.

cheers

[P3R]

On the destination storage thoughts, To my way of thinking "Smart" versioning suggests an implementation akin to what storageman describes, while perhaps full document version implementations of past decades is akin to "dumb" versioning.

Smart versioning is smart in comparision with versioning by multiple complete backup images. That was the only thing available before this versioning. Then again Smart versioning is only a marketing name, who cares what it's called...

If you want what storageman said, it's available through snapshots. It isn't as simple as he described it though. If one byte is changed in a file, it isn't enough to store only that. You need to keep track of where that byte fits into a file so you need to store separate metadata about that as well. This also means that you need to use the specific application to restore anything. One of the really beautiful things about HBS versioning is that it creates a complete file structure from where you can use your favourite file management tool to cherry pick any file from any version available.

Just as one example, I would be extremely interested if and how symbolic links would be used by HBS versioning.

The latest version of a file is always in the "latest" backup. When a backup is run, the old "latest" will become a version.

If the file is unchanged, the backup version then only contain a link (can be seen as a bookmark but it's integrated within the file system and nothing we need to care about) pointing to that actual file. Files that have been changed since the previous backup will be stored as an actual different file within the file structure of the backup version. So no matter how many versions, unchanged files doesn't use any more space than the file itself.

I've been a backup versioning user (first in beta testing and now in HBS as released software) for something like 18 months now and is happy with it.

I periodically create backups to external drives formatted NTFS to be placed in a safety deposit box in case I lose both NAS. To be comfortable, I would need to determine a means to copy the most recent versions of everything, along with its proper location, to the ntfs drives.

I'm not absolutely sure but I think that Ext4 file system is a requirement for using versioning.

[storageman]

Versioning target must be EXT3,EXT4, NTFS or HFS+ and versioning is supported on external USB drive.

[P3R]

Yes, it's on externally USB-connected disk that I've been running it all the time. I wasn't aware that multiple file systems were supported though but that's great.

It's supported from NAS to NAS as well but as far as I know RTRR is a requirement so Qnaps only on both sides. Unfortunately there's a bug so cat2 (modern) Qnap to (legacy) cat1 Qnap doesn't work with versioning.

[AlFromCochrane]

Thanks again to both of you for your comments.

Disaster struck me last night. I had all the equipment on a table in front of the fireplace at our new residence. A temporary thing while completing renovations. Still in pr0cess of moving. Well the whole damn face surrounding the fireplace complete with 100 lbs mantle fell off the wall! Hit the table and sent all my equipment flying all over the place. 5:39 am I hear a big crash and then could hear the TS-877 complaining away in QNAP NAS dialect. lol.

Complaining about having no storage space! Guess that's because all the drives had popped halfway out. lol! My backup nas (ts-431) landed on my lazyboy along with the Nexstar unit. They didn't even hiccup.

I popped them back in and they all tested good, but my datavol1 was in error, and the raid array failed when trying to rebuild. So I rebuilt and reinstalled apps.

My backup Nas ts-431 has 2 folders containing backups. One folder contains a complete backup (pre versioning) but HBS won't let me select it.

Lesson: Do NOT Delete a backup template if you ever wish to someday restore from backups made by it! The backup data exists but it seems to be orphaned by deleting the backup template! Very Very BAD design! This should have been a straight forward no brainer. Not even something one should need to seek documentation for.

Backup/restore software should be able to restore backed up data created by it regardless of whether an original backup template exists or not! Pretty scary! What if you lose your whole NAS, and you are restoring to a totally new NAS sans original backup template?

I suspect I can create a new sync job on one NAS or the other to get me out of this predicament. Or perhaps cherry the folders back, or whatever. I'll recover. I'm not too worried.

The only selection BHS gave me is for an incomplete versioning backup that I cancelled before it completed, as I was only testing the versioning. It's running now, but only contains 2 folders. < 1 TB. It's completed about 10 % in about 6 hours. So I will likely try accessing a copy from my External USB 3.0 after this completes tomorrow sometime.To gain some speed increase over the 1 Gbit lan


But going forward, reliance on BHS isn't leaving me feeling very comfortable at the moment. Has anyone tried any of the other 3rd party backup/restore offerings?

cheers

[P3R]

Backup/restore software should be able to restore backed up data created by it regardless of whether an original backup template exists or not! Pretty scary!

No it's not scary, you're misunderstanding the program and the way it works. HBS can't automatically find it's backup files if you have removed the backup job as there's no specific backup destination or proprietary backup file format to search for. It's just a plain simple file structure in a shared folder, no different from any other file structure in other shared folders on the NAS so how could HBS recognize it? The restore in HBS only swap source and destination of a job around, nothing more.

You have complete freedom to access the backup with whatever favourite file management tools you prefer. You can create the same task again in HBS and select Restore, you can create a sync job from the backup folder to where you want the files, you can use File Manager in Qnap to copy files, you can use Linux commands with SSH, you can use any client computer that can connect to the shared folder where you keep the backup.

With HBS you're not forced to use a restore function in a backup program and whatever limitations that may have. Because of this, there's no lock-in as there is with many other backup programs. After having used (and been let down by) other backup programs, this freedom is in my opinion a huge advantage for HBS and predecessor Backup Station.

[AlFromCochrane]

P3R

I believe you are a big fan of HBS and when i talked it down you acted quickly to defend it. In doing so you misunderstood me. If u look back at what i wrote, i very much understand how it works, or rather how it badly works. I figured out its reliance on the backup template. It does’t tell you that when restoring. Nor is it documented. I would say i gained understanding quickly. And i like most users did not have the time (was it 15 months ?) you have working with the developers to get it where it is today. For most, it is simply a backup restore tool that should be straight forward to use. And should be well documented with usage examples

I told you i was going to get my files back in all the ways that you just told me. And finished that by saying i wasn't worried. Pretty good understanding, not?

I too have been around the block and have used backup software in many different environments before.

When i went to restore, HBS should have allowed me to browse and select my backup folder. Period. Thats how it is done. To rely on a backup template that may or may not exist is Very bad design. No ifs, ands, or buts.

The program loses points for that nonsense bigtime. And yes, that will be a scary experience for most( i figured it out ) when they have a disaster and find they can’t select their backup data for restoration. Perhaps using Horrific is better than using Scary! Lol.

And it loses points for having no documentaion and examples.

That said, i totally understand that you worked hard with them to get the program to where it is today. And as such have earned the right to defend it. That’s an appreciated effort. Hasen’t negative user feedback always carried the most value?

Anyway i trust you in believing BHS is the likely best in the barn at this time, and will likely continue to use it.

Cheers

[AlFromCochrane]

P3R

I am back up an running. So I create a versioning template. I chose 0 hours, 7 days, 5 weeks and 6 months. To see how it works. I start the job running, then realize I never scheduled it to run daily. I stopped the backup job tried to edit the template and couldn't figure out how to add a schedule. So delete and recreate with daily schedule. I started it running,

But since it will take longer than 24 hours to create the initial backup of everything, what will happen when the next daily scheduled backup launches when yesterdays is still running?

So I decide to stop it again and run a full backup without a schedule. So I create another template to do that.

If I create a template without a schedule and let it run for 48 hours or whatever. And then I can't edit the template to add the schedule going forward, So I would need to recreate as I did above to add a schedule to the template. This presents a dilemma if the backup's original template must remain in order for HBS to offer the backup during a restore. Both templates would likely produce undesirable results as they would be restoring 2 separate backup sources even if the target directory was the same. Sorry if I'm complicating things again.

Any advise/direction you can provide would be greatly appreciated.

[P3R]

And i like most users did not have the time (was it 15 months ?) you have working with the developers to get it where it is today.

What I told you is how long I think I've been a user of the Qnap versioning solution. It was available in open beta as an add-on for Backup Station but was never released because they were working on HBS replacing Backup Station. I've had no contact with the developers and have contributed absolutely nothing to the development of versioning, as I never found a single thing not working with it.

I didn't participate at all in the HBS open beta as I was sceptical of having a single application covering all the possible backup scenarios, including the 99 different cloud services that happen to be the cheapest at the moment (which changes every week). I'm still sceptical about HBS being a do-it-all apllication as I think it would have been better to keep Backup Station for local and remote backups and have a separate application/applications for the different cloud backup services, as the requirements are different. But when HBS was released I accepted, because I knew that it was the way forward if I still wanted a Qnap-integrated backup application that offered the freedom of a plain file backup.

It wasn't to brag that I told you how long I had been using versioning, but only to give some confidence to the community that a relatively experienced user have been using it for an extensive time and that it at least have worked well for me.

So to sum up I haven't done any big effort with HBS and feel no need to defend it. I'm just trying to help you by explaining how it's slightly different from the average backup programs and why that isn't necessarily a bad thing.

And should be well documented with usage examples

I have no real disagreement with you about how it should be with documentation in a perfect world. The difference is that I'm a +10 year Qnap user so have learned a few things about how Qnap usually work. Therefore I earlier explained why there's no documentation for HBS presently. Qnap releases these separate add-on apps when the code is of acceptable quality and then start writing the documentation for it (with a main QTS firmware it's different, as that documentation usually is available on release or close after it). It's not great releasing apps without documentation but on the plus side it give us users the possibility to use the applications earlier. You're of course free to use the still available Backup Station. It's documented (not very good though) but have less features, still have bugs that will never be fixed and most of the community aren't using it any more so forum assistance will be even more limited than with HBS.

Here's another disappointment for you, Qnap documentation isn't as extensive as you want it. It's not a classic user guide with lots of background and examples and it's not even a proper command reference that cover all the possible options either. It's a documentation that covers most of the more important options, usually have several step-by-step instructions but very few examples. I'm not defending it, just tell you how it is with Qnap and their documentation. Take it or leave it.

Please note that when I'm explaining how it is with Qnap-documentation, I'm not defending them. I've decided to stay a Qnap user despite of their sometimes limited documentation, not because I love it.

When i went to restore, HBS should have allowed me to browse and select my backup folder. Period. Thats how it is done.

Yes, that's how it must be done in old-school proprietary backup software, where a restore function is absolutely needed. You unfortunately seem to be stuck in the mind-set that a backup application must have a backup and a restore function. Period. I tried to explain why a Restore isn't really important but you apparently don't want to be told things and instead become angry. Period.

Let's remember here that you deleted the backup job but still demand that the backup application should provide a restore feature. No problem, HBS can actually still do what you want. Now I may be asking too much of you here but in return for having deleted the backup job (so that HBS have zero knowledge about your previous backups) you have to stop being obsessed with that it must be called a Restore in the app. If you can only achieve that little thing and instead start thinking of a restore as a backup/sync task in the opposite direction, everything is as you want it, including the browse feature that you request. If it makes you feel any better about the situation you can even name the backup/sync job you create Restore. There it is, your huge problem is solved.

If you want even better browse functionality and if you want to restore only parts of the backup (amybe to different destinations), there's a much better tool available for you in the Qnap and it's called the File Station.

Actually the predessor to HBS, Backup Station didn't even have a Restore-feature as it isn't vital. I can only imagine how upset that would have made you when noticing that.

I am back up an running. So I create a versioning template. I chose 0 hours, 7 days, 5 weeks and 6 months. To see how it works. I start the job running, then realize I never scheduled it to run daily. I stopped the backup job tried to edit the template and couldn't figure out how to add a schedule.

I'm pretty sure that it should be possible to add/change a schedule but don't have access to a Qnap on my current location so can't check and make a guide for you.

But since it will take longer than 24 hours to create the initial backup of everything, what will happen when the next daily scheduled backup launches when yesterdays is still running?

I don't know for sure and can only guess that it won't start a new job unless the previous scheduled job have finished. Subsequent jobs should run faster as unchanged files won't be copied but just linked.

So I decide to stop it again and run a full backup without a schedule. So I create another template to do that.

You're very impatient...

If I create a template without a schedule and let it run for 48 hours or whatever. And then I can't edit the template to add the schedule going forward, So I would need to recreate as I did above to add a schedule to the template.

Did you delete the scheduled versioned job (templates as you call them) that you had already previously created?

Haven't you noticed that HBS can have many different jobs configured? I probably have +20 backup/sync jobs in mine.

This presents a dilemma if the backup's original template must remain in order for HBS to offer the backup during a restore. Both templates would likely produce undesirable results as they would be restoring 2 separate backup sources even if the target directory was the same.

You're still focusing mostly on the totally non-essential restore-function...

If anything, I would rather think that the real problem is that a single full backup probably have a different file structure than the versioned one so your versioned job will most likely not notice your full backup and do a very long first run anyway. I think that full backup will turn out to be a waste of time and more importantly, of backup disk capacity.

Any advise/direction you can provide would be greatly appreciated.

Yes, start the scheduled versioned job. Then try to relax and let it complete. If something turns out to be wrong, then change it afterwards. Learn the app by using it.

My last advice may be the hardest for you, stop focusing on the very unimportant Restore for a while.

[AlFromCochrane]

P3R wrote:
What I told you is how long I think I've been a user of the Qnap versioning solution. It was available in open beta as an add-on for Backup Station but was never released because they were working on HBS replacing Backup Station.

I didn't realize the versioning beta was an add on. I missed that fact somehow. My bad.

It wasn't to brag that I told you how long I had been using versioning

I didn't intend to insinuate you were bagging. I was trying to separate you, and even me for my foreseeable confidence, from what i envision most users expect to happen when they click a a Restore icon in their backup application.

Here's another disappointment for you, Qnap documentation isn't as extensive as you want it

Agreed. Don't have to tell me. Lol. This forum, or Qnap, support is where answers are found. from people like you.

Yes, that's how it must be done in old-school proprietary backup software, where a restore function is absolutely needed. You unfortunately seem to be stuck in the mind-set that a backup application must have a backup and a restore function. Period. I tried to explain why a Restore isn't really important but you apparently don't want to be told things and instead become angry. Period.

Never angry when receiving help. My intention was just to point out I had just said the same things in my post. Knowledge of those same things gave me confidence to not panic or worry.

Let's remember here that you deleted the backup job but still demand that the backup application should provide a restore feature. No problem, HBS can actually still do what you want. Now I may be asking too much of you here but in return for having deleted the backup job (so that HBS have zero knowledge about your previous backups) you have to stop being obsessed with that it must be called a Restore in the app.

Here is where I disagree. So I deleted the backup template. That's not the point. If you NAS walks out the door, or burns, that deletes the template too. What knowledge are you talking about? Everything a restore function needs is in, or should be in, the backup location. All the user should do is chose the desired backup and the target location. I am fine with the way HBS lets you map the folders in the source to target folders. It just needs to let the user chose his backup from whatever backup locations are available. That is what most users will expect. The Restore IS called Restore in the app, not an obsession but a totally valid expectation.

I'm pretty sure that it should be possible to add/change a schedule but don't have access to a Qnap on my current location so can't check and make a guide for you.

I would have thought so to. Even the document link that storageman provided in this thread (thanks for that storageman!) states there is edit buttons that will allow you to edit the templates but for whatever reason, I don't have any edit buttons beside my templates. Perhaps that refers to when versioning was an addon and not part of HBS. Does anyone else have this problem?

If it makes you feel any better about the situation you can even name the backup/sync job you create Restore. There it is, your huge problem is solved.

tool available for you in the Qnap and it's called the File Station.

Now I may be asking too much of you here

If you can only achieve that little thing

and so on...

Come on P3R, these statements are pretty much uncalled for don't you think? :=) I've shown you nothing but respect. And you claim I'm angry...

I hugely value this forum and greatly respect the gurus. Without this forum, what do we have? Screenshots...

[P3R]

I'm sorry for not being able to explain so that you understand.
I'm sorry for becoming frustrated and sarcastic.

[AlFromCochrane]

I do understand. I just have a different opinion is all. I once hunted real hard for a win based backup software that could store the data in its native format. Backup Genie rings a bell. But now i use macrium reflect for its vesioning.

No worries.

Btw, I spent some time yesterday pursuing one of your recommendations. Was looking at using remote snapshots. Still researching and learning there. If i have questions regarding that, i’ll start a new thread. Perhaps i can benefit from your expertise again in the near future

Cheers