Replicate BIG files between QNAP NAS

Backup, Restore, Netbak Replicator, Cloud Storage Services
Post Reply
cedimp
New here
Posts: 9
Joined: Fri Feb 24, 2017 9:27 pm

Replicate BIG files between QNAP NAS

Post by cedimp »

Hi,

i'd like to replicate big files between QNAP NAS. But on theses files many blocks does not change.

Let's me give the real example. I'm talking about veeam backup.
You have a big .vbk file containing the initial backup
Some .vib files that are increments.
When you have the right number of increment, the oldest .vib is merged with the .vbk

If you do a "normal" RTRR replication, all the .vbk file will be copied as it was altered. But in fact only a part of it has changed.

Is there a way to copy only modified blocks ?

(i know Veeam is able to do that using its own mecanism, but i'm looking for a solution that is independant)

Any thoughs are welcome

Thanks
cedimp
New here
Posts: 9
Joined: Fri Feb 24, 2017 9:27 pm

Re: Replicate BIG files between QNAP NAS

Post by cedimp »

Hi all,

no idea ? Not even some clues ?

Thanks
cedimp
New here
Posts: 9
Joined: Fri Feb 24, 2017 9:27 pm

Re: Replicate BIG files between QNAP NAS

Post by cedimp »

Hello there,

Is my question in the wrong section ?
Did i i gave the right explanation and info ?
I'm surprised not to have even some ideas

Thanks

Cédric
Last edited by cedimp on Thu Feb 02, 2023 11:18 pm, edited 1 time in total.
User avatar
Toxic17
Ask me anything
Posts: 6469
Joined: Tue Jan 25, 2011 11:41 pm
Location: Planet Earth
Contact:

Re: Replicate BIG files between QNAP NAS

Post by Toxic17 »

you are in the right section, but this is a community forums and users only tend to answer if they are familiar with the scenario given. the fact no one has replied should tell you the likelihood of someone using veeam and backing up modified blocks are slim. QNAP are nearly non existent here in the forums.

if HBS3 wont support your modified backup solution then maybe one of the other 3rd part backup QPKGs might, but I am guessing unless a 3rd party software supports this veeam "modified blocks" protocol it wont know anything about it.

is this the same as VMware's Changed Block Tracking (CBT)? the backup solution would need to support this I guess.

https://www.qnap.com/en/how-to/tutorial ... a-qnap-nas

https://www.myqnap.org/product-category/backup/

sorry I cannot suggest anything else, I dont use this type of backup method.
Regards Simon

Qnap Downloads
MyQNap.Org Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following


NAS: TS-673A QuTS hero h5.1.2.2534 • TS-121 4.3.3.2420 • APC Back-UPS ES 700G
Network: VM Hub3: 500/50 • UniFi UDM Pro: 3.2.9 • UniFi Network Controller: 8.0.28
USW-Aggregation: 6.6.61 • US-16-150W: 6.6.61 • 2x USW Mini Flex 2.0.0 • UniFi AC Pro 6.6.62 • UniFi U6-LR 6.6.62
UniFi Protect: 2.11.21/8TB Skyhawk AI • 3x G3 Instants: 4.69.55 • UniFi G3 Flex: 4.69.55 • UniFi G5 Flex: 4.69.55
cedimp
New here
Posts: 9
Joined: Fri Feb 24, 2017 9:27 pm

Re: Replicate BIG files between QNAP NAS

Post by cedimp »

Hi,

many thanks for repying, you're right i should ask something too precise :)

In fact the backgroup is : how to synchronise big files without coying the entire file when only some part of the file as change...

Maybe i'll have more luck with this
PeterT1959
Know my way around
Posts: 201
Joined: Mon Nov 19, 2012 11:16 am

Re: Replicate BIG files between QNAP NAS

Post by PeterT1959 »

I *think* that using the rsync command AND (ideally) connecting to a rsync server it can just send changed blocks.

The reason you'd want to connect to an resync server is to allow the checksum calculations to be performed on local files.

See the description in Wikipedia especially the section on "Determining which parts of a file have changed" https://en.wikipedia.org/wiki/Rsync

Sent from my Pixel 4a using Tapatalk

PeterT1959
Know my way around
Posts: 201
Joined: Mon Nov 19, 2012 11:16 am

Re: Replicate BIG files between QNAP NAS

Post by PeterT1959 »

I tried an experiment. I made a big file, backed ot up via rsync, made a change and backed it up again.

Code: Select all

petert@dockerhost:~/src/big $ ls -la
total 1048596
drwxr-xr-x 2 petert petert       4096 Feb  2 16:23 .
drwxr-xr-x 9 petert petert       4096 Feb  2 15:53 ..
-rw-r--r-- 1 petert petert 1073741824 Feb  2 16:38 a.big.file
-rwxr-xr-x 1 petert petert        227 Feb  2 16:23 changebig.py
-rwxr-xr-x 1 petert petert        253 Feb  2 16:18 makebig.py

petert@dockerhost:~/src/big $ rsync -v a.big.file rsync@192.168.2.70::rsync/a.big.file
a.big.file

sent 1,074,004,054 bytes  received 39 bytes  65,091,157.15 bytes/sec
total size is 1,073,741,824  speedup is 1.00

petert@dockerhost:~/src/big $ rsync -v a.big.file rsync@192.168.2.70::rsync/a.big.file 
a.big.file

sent 131,170 bytes  received 229,427 bytes  13,607.43 bytes/sec
total size is 1,073,741,824  speedup is 2,977.68

petert@dockerhost:~/src/big $ [b]./changebig.py [/b]

petert@dockerhost:~/src/big $ rsync -v a.big.file rsync@192.168.2.70::rsync/a.big.file
a.big.file

sent 163,938 bytes  received 229,427 bytes  13,802.28 bytes/sec
total size is 1,073,741,824  speedup is 2,729.63
So the initial rsync backup sent 1,074,004,054 bytes and received 39 bytes.

A second backup of an unchanged file sent 131,170 bytes and received 229,427 bytes.

After changing 1024 bytes another backup sent 163,938 bytes and received 229,427 bytes.
cedimp
New here
Posts: 9
Joined: Fri Feb 24, 2017 9:27 pm

Re: Replicate BIG files between QNAP NAS

Post by cedimp »

Waouh, many thanks for that PeterT1959, this is exactly what i needed.
I read somewhere that this is the difference between RTRR and rsync.

Is the rsync used into qnap HBS3 such efficient ?

I'll go ahead and read your wikipedia link, i should learn many thinks

Great !
PeterT1959
Know my way around
Posts: 201
Joined: Mon Nov 19, 2012 11:16 am

Re: Replicate BIG files between QNAP NAS

Post by PeterT1959 »

All this testing was done from a Raspberry Pi (client) to QNAP (server). I enabled rsync on the QNAP.

Sent from my Pixel 4a using Tapatalk

cedimp
New here
Posts: 9
Joined: Fri Feb 24, 2017 9:27 pm

Re: Replicate BIG files between QNAP NAS

Post by cedimp »

Hi and sorry for my late reply.

Many thanks for this clarification, this is greatly appreciated !

Now i have all needed to go forward

Have a nice day !
Post Reply

Return to “Backup & Restore”