HBS3 Alternative

Backup, Restore, Netbak Replicator, Cloud Storage Services
kingbacana
Know my way around
Posts: 131
Joined: Wed Aug 07, 2013 8:40 am

Re: HBS3 Alternative

Post by kingbacana »

edit:
I used SFTP as connection protocol.

type = sftp
ssh
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: HBS3 Alternative

Post by P3R »

kingbacana wrote: Sun Jan 16, 2022 5:37 am P3R

I just tried someting, I added rclone mount to my local nas.
I try my best to help with HBS as I've been a daily user of it's since it was released and hopefully have picked up a few things on the way but I have really have no idea other than the description about rclone. You'll have to ask Qnap_Stephane about that.

Generally speaking, combining features from different apps is a support nightmare and is nothing that I would recommend for backups, that are the most important thing you have to protect your data. You don't want your (or at least I don't want my) data protection hanging on a thread.

I'm sorry but I'm totally backing out of this question before i'm pulled in any deeper...

Good luck with whatever you're trying to achieve!
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
kingbacana
Know my way around
Posts: 131
Joined: Wed Aug 07, 2013 8:40 am

Re: HBS3 Alternative

Post by kingbacana »

P3R thanks for your messaging how can I ping/ mention Stephane about this issue in this topic?

I like to solve this matter, cause I cant get it out my head. its pretty important for me.
Cause I think:
copy moving 12tb of data send it to the cloud.
delete it and do it later again is tearing the drives apart dont u think?

I mean:
the issue is
nas 1 nas2
14tb. 8tb as example

nas1 , has only 9tb left.
More data comes on nas1. so I copy/move 8tb to nas 1 and backup it.
Then delete it again from nas1. to have free space.

Better option is in my opinion make data of nas2 accessible from nas1 and backup to the cloud?

What u think? I think in theory, the data is still copied over in chuncks to nas 1 to upload to cloud. but not the whole 8 tb at once right?

Why hbs 3 not allow us to choose a different source ( only local nas is availalbe as source)

I tried:
Duplicati. but to much database errors after a few backups with more then > 5tb data.
Duplicacy, is terrible in recovering, because you need to leave a window open and give manual paths. the backup works good but recovering is not good.

Rest of option is not good. maybe borgbackup idk..

Also older QTS have no encryption client side for cloud providers.

Otherwise I would do it from nas2. but Nas 2 is a old 4.2 qts. which has old hbs.

so nas 1 is my only option for the backup with encryption.

Im only worried what if Nas1 gets old and the cloud providers change their login/ encryption mechanism. And HBS3 is not able to recover / backup data to the cloud provider?

is all problems im afraid off..
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: HBS3 Alternative

Post by P3R »

I struggle to understand most of this but I'll try with the few things I think are relatively clear.
kingbacana wrote: Mon Jan 17, 2022 6:22 pm P3R thanks for your messaging how can I ping/ mention Stephane about this issue in this topic?
I don't know. If you want help with rclone the best way is probably to reach him in the thread where he made rclone available.
copy moving 12tb of data send it to the cloud.
delete it and do it later again is tearing the drives apart dont u think?
Hard disks are designed and manufactured for one single purpose - to store data. They don't fail from doing what they were intended to do.
Why hbs 3 not allow us to choose a different source ( only local nas is availalbe as source)
The reason is probably that Qnap is in the business of selling new Qnaps. Their main goal isn't to help you keep on using end-of-lifed and unsupported equipment. HBS can pull syncs from servers running several different standard protocols as well as numerous cloud services though if that is of any help.
Im only worried what if Nas1 gets old and the cloud providers change their login/ encryption mechanism. And HBS3 is not able to recover / backup data to the cloud provider?
I don't know how Qnap have designed their client-side encryption but in my world it wouldn't be client-side encryption if it relied on a cloud service encryption mechanism.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
kingbacana
Know my way around
Posts: 131
Joined: Wed Aug 07, 2013 8:40 am

Re: HBS3 Alternative

Post by kingbacana »

P3R

I checked it with a colleague or I asked him he said:

some application on Linux or any OS. Are not following links/ directive links. They show it maybe in the GUI. but when the real job gets done it wont follow a soft link or network link. You need to develop/ code this correctly.
So I think the hbs3 doesn't follow the link correctly and will then notice it does not contain the folder on the real disk location?

as example: X:\folder_one
folder_two = rclone mount. ( link software?)

GUI follows the link with a getFolder() will show or return folder one and two.
HBS starts backup tasks and do a GetFolder() and return folder one or null?
Then it will tell no source folder exists?
Or I missing something in the command of rclone that kills the connection temporarily when I do a intensive backup job and HBS gets a timeout and returns a source folder not exist error because the connection drops?


Its a gamble.

To answer your reply with the client side encryption:
I pronounced it wrong, I mean when the access gets revoked of aging of the HBS3 app, maybe providers will change the keys/ login mechanism and the HBS3 apps or variants are not supported anymore to login?
Ive seen it before on old mac osx systems and or windows.
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: HBS3 Alternative

Post by P3R »

kingbacana wrote: Tue Jan 18, 2022 12:50 am ...in the command of rclone that kills the connection temporarily when I do a intensive backup job and HBS gets a timeout and returns a source folder not exist error because the connection drops?
All of this is a bad idea and I've already told you that I won't take part in that...
I pronounced it wrong, I mean when the access gets revoked of aging of the HBS3 app, maybe providers will change the keys/ login mechanism and the HBS3 apps or variants are not supported anymore to login?
You can always try a desktop OS client system and use any of the apps that you have download links for on the very first page when you start HBS3.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
kingbacana
Know my way around
Posts: 131
Joined: Wed Aug 07, 2013 8:40 am

Re: HBS3 Alternative

Post by kingbacana »

P3R wrote: Tue Jan 18, 2022 1:37 am
kingbacana wrote: Tue Jan 18, 2022 12:50 am ...in the command of rclone that kills the connection temporarily when I do a intensive backup job and HBS gets a timeout and returns a source folder not exist error because the connection drops?
All of this is a bad idea and I've already told you that I won't take part in that...
I pronounced it wrong, I mean when the access gets revoked of aging of the HBS3 app, maybe providers will change the keys/ login mechanism and the HBS3 apps or variants are not supported anymore to login?
You can always try a desktop OS client system and use any of the apps that you have download links for on the very first page when you start HBS3.
I did some testing this weekend, to try make the 509 pro a little more usable.

I notice something weird.

the version used : HBS 1.3 ( cause max supported version of 4.2.6 OS)
I also installed cloud support.

I created a sync one way to gdrive.
I logged in and made a sync task, I clicked the checkedbox named client side encryption, to my suprise, it didnt ask me for a password.

I made a sync, and checked google drive: I seen the test.txt file was encrypted.
I made a one way sync back from cloud to drive, and the test.txt file was received back as un-encrypted.

I started up another 509, made the same task and did now a one way sync from cloud to nas. The file was encryped.

The ** part, somehow it encrypts, but doesnt let me give a option to enter a own password.

Then I went to HBS profiles.
I seen a google drive account, when I delete it and try to create a new one, it doesnt let me choose google drive.
So doing the sync task and add google. somehow it makes it available in the profile tab.
Then I press that account and say edit. Suddenly I have a client side encryption tab. Where I can put a password.
When i type in test test, the checkbox is allow me to press ok. If i type in test test1 it doenst allow me to press ok.

So i type in test test and when I press ok, it says a nonsense message sorry the passwords are not identical.

Weird not?

my theory is:
The moment the google account is added from the sync task, it creates a encryption password based on the nas and the user password?
when I try to work-around change the password for encryption, it compares the new password and the previous password and then says its wrong? tbh this is some weird magic?

I want to make a workaround or fix, to be able to make a backup or sync data to gdrive using encryption.
now im thinking about restic with a gdrive rclone mount, problem is offically I dont want to make scripts/ run things manually.

All I want is now in HBS3 on newer qnap, I have one, but I want to make re-use of the old nas to do the same or equal to.

Also what I hate about HBS3 is, to recover files , you can only download all or nothing? I cannot select a individual file from a encrypted backup.

Whats good about HBS3, the own password. so If my nas dies, I buy a new one, login in google make a recover task and use the old encrypted password and I have my files back, only I have to re-download all and not only 2 files.
So if I want only 1 single file back, its not possible.
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: HBS3 Alternative

Post by P3R »

kingbacana wrote: Mon Jan 24, 2022 4:59 pm ...but I want to make re-use of the old nas...
You can use HBS1 as a destination for backups from HBS3. Then you can do plain file system backups with space-efficient versioning. That's what I've been usning for years and I love it. No QuDedup and no client-side encryption then of course.
Also what I hate about HBS3 is, to recover files , you can only download all or nothing? I cannot select a individual file from a encrypted backup.
I wouldn't know as I don't restore through HBS3 but from the destination directly so don't have any such limitations. But with cloud storage client-side encryption is a must so then it sound like a bad limitation.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: HBS3 Alternative

Post by Moogle Stiltzkin »

kingbacana wrote: Mon Jan 24, 2022 4:59 pm
I did some testing this weekend, to try make the 509 pro a little more usable.
my ts-509 pro is still usable, that said i've moved on. i use the ts-877, ts-653a, and the newest model is a ts-253d. all 3 models have the option of switching truenas if the time ever comes where qts or hbs is no longer maintained.

i tried to do this for the ts-509 pro, but it was a bit of a hassle to do so i gave up. I've had good milleage from it for more than 10 years, so i accepted it was time for an upgrade (you can still use it without upgrading if you want, but for me i wanted something newer and more performant. for example i can't do aes-ni effectively for encrypting shares with this ancient model, due to the performance hit)

anyway i'm interested in your findings on this, so keep us updated how it goes :' all i use on the ts-509 pro is the old no longer maintained hbs.
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
kingbacana
Know my way around
Posts: 131
Joined: Wed Aug 07, 2013 8:40 am

Re: HBS3 Alternative

Post by kingbacana »

Can we run a django web framework / website on qnap

And run commands from a webpage onto the qnap?

Like Example:
1.Go to Https://server-ip:5000 see your own created page:
2. See a textbox type in ls -l /root
3. see a webview list with the /root files displayed?

Is this possible? and note down the website is running on the same device.
kingbacana
Know my way around
Posts: 131
Joined: Wed Aug 07, 2013 8:40 am

Re: HBS3 Alternative

Post by kingbacana »

Ok I think I create a solution for us all, and release it opensource I guess.

Im struggling with a problem that the apps I use , they fill up the tmp folder / root folder.
In qnap this folder are incredible low of space. So that's a little bit a problem.

Because the app I use they output to this place as default and I dont see a override option.

If I however somehow can overrule this and as example make or require u guys to create a folder dedicated for my app solution. as example

/share/MD0_DATA/BACKUP_CONFIG

MD0_DATA is the raid array.
when higher qnap then CACHEDEV_DATA or somehing like that, then this could work. for us.

Symbolic linking did not work, because some apps dont follow the links they want hard dedicated paths. physical paths thats annoying..
lanfox_CN
New here
Posts: 5
Joined: Wed Feb 16, 2022 8:10 pm

Re: HBS3 Alternative

Post by lanfox_CN »

viewtopic.php?f=15&t=165196

I'm not good at English.

When using Rsync, hbs3 seems to synchronize all data, wasting a lot of time.
After trying, I suspect that the script file responsible for generating the Rsync command may have a bug.

In / share / cachedev1_ DATA/. qpkg/HybridBackup/rsyncRR. sh

find

RR_ opt_ inc=`/sbin/getcfg ${RR_sche} "Incremental" -f ${RR_CONF} -d "FALSE" -u`
[ "x${RR_opt_inc}" = "xFALSE" ] && RR_ options=${RR_options}"--whole-file -I "

Change to

RR_ opt_ inc=`/sbin/getcfg ${RR_sche} "Incremental" -f ${RR_CONF} -d "FALSE" -u`
[ "x${RR_opt_inc}" = "xTRUE" ] && RR_ options=${RR_options}"--whole-file -I "

After modification, from the log file, Rsync no longer uses the -- whole file - I parameter.
kingbacana
Know my way around
Posts: 131
Joined: Wed Aug 07, 2013 8:40 am

Re: HBS3 Alternative

Post by kingbacana »

lanfox_CN wrote: Wed Feb 16, 2022 8:35 pm viewtopic.php?f=15&t=165196

I'm not good at English.

When using Rsync, hbs3 seems to synchronize all data, wasting a lot of time.
After trying, I suspect that the script file responsible for generating the Rsync command may have a bug.

In / share / cachedev1_ DATA/. qpkg/HybridBackup/rsyncRR. sh

find

RR_ opt_ inc=`/sbin/getcfg ${RR_sche} "Incremental" -f ${RR_CONF} -d "FALSE" -u`
[ "x${RR_opt_inc}" = "xFALSE" ] && RR_ options=${RR_options}"--whole-file -I "

Change to

RR_ opt_ inc=`/sbin/getcfg ${RR_sche} "Incremental" -f ${RR_CONF} -d "FALSE" -u`
[ "x${RR_opt_inc}" = "xTRUE" ] && RR_ options=${RR_options}"--whole-file -I "

After modification, from the log file, Rsync no longer uses the -- whole file - I parameter.
Awesome, fix I will share it with others,
But my solution will not use rsync I think.

When I succeed I share my way.

Code: Select all

34

It seems like you might want to take a look at the --whole-file or -W switch. This is enabled by default if rsync is doing local filesystem copies, but I think the docs recommend its usage if the LAN speed is high and syncing over the network.

Basically, it disables the rsync delta algorithm and just transfers the whole file if it believes it's different.

You shouldn't have to worry about checksumming unless you force -c. By default, it just looks at the last modified time and the file size, and assumes the files are different if those values are different.
lanfox_CN
New here
Posts: 5
Joined: Wed Feb 16, 2022 8:10 pm

Re: HBS3 Alternative

Post by lanfox_CN »

kingbacana wrote: Thu Feb 17, 2022 12:54 am
lanfox_CN wrote: Wed Feb 16, 2022 8:35 pm viewtopic.php?f=15&t=165196

I'm not good at English.

When using Rsync, hbs3 seems to synchronize all data, wasting a lot of time.
After trying, I suspect that the script file responsible for generating the Rsync command may have a bug.

In / share / cachedev1_ DATA/. qpkg/HybridBackup/rsyncRR. sh

find

RR_ opt_ inc=`/sbin/getcfg ${RR_sche} "Incremental" -f ${RR_CONF} -d "FALSE" -u`
[ "x${RR_opt_inc}" = "xFALSE" ] && RR_ options=${RR_options}"--whole-file -I "

Change to

RR_ opt_ inc=`/sbin/getcfg ${RR_sche} "Incremental" -f ${RR_CONF} -d "FALSE" -u`
[ "x${RR_opt_inc}" = "xTRUE" ] && RR_ options=${RR_options}"--whole-file -I "

After modification, from the log file, Rsync no longer uses the -- whole file - I parameter.
Awesome, fix I will share it with others,
But my solution will not use rsync I think.

When I succeed I share my way.

Code: Select all

34

It seems like you might want to take a look at the --whole-file or -W switch. This is enabled by default if rsync is doing local filesystem copies, but I think the docs recommend its usage if the LAN speed is high and syncing over the network.

Basically, it disables the rsync delta algorithm and just transfers the whole file if it believes it's different.

You shouldn't have to worry about checksumming unless you force -c. By default, it just looks at the last modified time and the file size, and assumes the files are different if those values are different.
Let me add:
There is an option in the configuration page of hbs3 rysnc job: "check file contents“
Before modifying rsyncRR. sh, from the log and actual effect, no matter how you choose, it doesn't actually take effect.
For me, I do remote backup through the Internet. The "-- whole file - I" is too time-consuming and loses the effect of backup.
I think the user should decide whether to check the file or not, rather than hbs3 instead of us.
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: HBS3 Alternative

Post by P3R »

lanfox_CN wrote: Thu Feb 17, 2022 6:41 am I think the user should decide whether to check the file or not, rather than hbs3 instead of us.
It sound like you have found a bug. Have you reported your find to Qnap support? They can't fix issues until they're aware of them...
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
Post Reply

Return to “Backup & Restore”