Hi,
i'd like to replicate big files between QNAP NAS. But on theses files many blocks does not change.
Let's me give the real example. I'm talking about veeam backup.
You have a big .vbk file containing the initial backup
Some .vib files that are increments.
When you have the right number of increment, the oldest .vib is merged with the .vbk
If you do a "normal" RTRR replication, all the .vbk file will be copied as it was altered. But in fact only a part of it has changed.
Is there a way to copy only modified blocks ?
(i know Veeam is able to do that using its own mecanism, but i'm looking for a solution that is independant)
Any thoughs are welcome
Thanks
Replicate BIG files between QNAP NAS
-
- New here
- Posts: 9
- Joined: Fri Feb 24, 2017 9:27 pm
Re: Replicate BIG files between QNAP NAS
Hi all,
no idea ? Not even some clues ?
Thanks
no idea ? Not even some clues ?
Thanks
-
- New here
- Posts: 9
- Joined: Fri Feb 24, 2017 9:27 pm
Re: Replicate BIG files between QNAP NAS
Hello there,
Is my question in the wrong section ?
Did i i gave the right explanation and info ?
I'm surprised not to have even some ideas
Thanks
Cédric
Is my question in the wrong section ?
Did i i gave the right explanation and info ?
I'm surprised not to have even some ideas
Thanks
Cédric
Last edited by cedimp on Thu Feb 02, 2023 11:18 pm, edited 1 time in total.
- Toxic17
- Ask me anything
- Posts: 6477
- Joined: Tue Jan 25, 2011 11:41 pm
- Location: Planet Earth
- Contact:
Re: Replicate BIG files between QNAP NAS
you are in the right section, but this is a community forums and users only tend to answer if they are familiar with the scenario given. the fact no one has replied should tell you the likelihood of someone using veeam and backing up modified blocks are slim. QNAP are nearly non existent here in the forums.
if HBS3 wont support your modified backup solution then maybe one of the other 3rd part backup QPKGs might, but I am guessing unless a 3rd party software supports this veeam "modified blocks" protocol it wont know anything about it.
is this the same as VMware's Changed Block Tracking (CBT)? the backup solution would need to support this I guess.
https://www.qnap.com/en/how-to/tutorial ... a-qnap-nas
https://www.myqnap.org/product-category/backup/
sorry I cannot suggest anything else, I dont use this type of backup method.
if HBS3 wont support your modified backup solution then maybe one of the other 3rd part backup QPKGs might, but I am guessing unless a 3rd party software supports this veeam "modified blocks" protocol it wont know anything about it.
is this the same as VMware's Changed Block Tracking (CBT)? the backup solution would need to support this I guess.
https://www.qnap.com/en/how-to/tutorial ... a-qnap-nas
https://www.myqnap.org/product-category/backup/
sorry I cannot suggest anything else, I dont use this type of backup method.
Regards Simon
Qnap Downloads
MyQNap.Org Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following
NAS: TS-673A QuTS hero h5.1.2.2534 • TS-121 4.3.3.2420 • APC Back-UPS ES 700G
Network: VM Hub3: 500/50 • UniFi UDM Pro: 3.2.9 • UniFi Network Controller: 8.0.28
USW-Aggregation: 6.6.61 • US-16-150W: 6.6.61 • 2x USW Mini Flex 2.0.0 • UniFi AC Pro 6.6.62 • UniFi U6-LR 6.6.62
UniFi Protect: 2.11.21/8TB Skyhawk AI • 3x G3 Instants: 4.69.55 • UniFi G3 Flex: 4.69.55 • UniFi G5 Flex: 4.69.55
Qnap Downloads
MyQNap.Org Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following
NAS: TS-673A QuTS hero h5.1.2.2534 • TS-121 4.3.3.2420 • APC Back-UPS ES 700G
Network: VM Hub3: 500/50 • UniFi UDM Pro: 3.2.9 • UniFi Network Controller: 8.0.28
USW-Aggregation: 6.6.61 • US-16-150W: 6.6.61 • 2x USW Mini Flex 2.0.0 • UniFi AC Pro 6.6.62 • UniFi U6-LR 6.6.62
UniFi Protect: 2.11.21/8TB Skyhawk AI • 3x G3 Instants: 4.69.55 • UniFi G3 Flex: 4.69.55 • UniFi G5 Flex: 4.69.55
-
- New here
- Posts: 9
- Joined: Fri Feb 24, 2017 9:27 pm
Re: Replicate BIG files between QNAP NAS
Hi,
many thanks for repying, you're right i should ask something too precise
In fact the backgroup is : how to synchronise big files without coying the entire file when only some part of the file as change...
Maybe i'll have more luck with this
many thanks for repying, you're right i should ask something too precise
In fact the backgroup is : how to synchronise big files without coying the entire file when only some part of the file as change...
Maybe i'll have more luck with this
-
- Know my way around
- Posts: 201
- Joined: Mon Nov 19, 2012 11:16 am
Re: Replicate BIG files between QNAP NAS
I *think* that using the rsync command AND (ideally) connecting to a rsync server it can just send changed blocks.
The reason you'd want to connect to an resync server is to allow the checksum calculations to be performed on local files.
See the description in Wikipedia especially the section on "Determining which parts of a file have changed" https://en.wikipedia.org/wiki/Rsync
Sent from my Pixel 4a using Tapatalk
The reason you'd want to connect to an resync server is to allow the checksum calculations to be performed on local files.
See the description in Wikipedia especially the section on "Determining which parts of a file have changed" https://en.wikipedia.org/wiki/Rsync
Sent from my Pixel 4a using Tapatalk
-
- Know my way around
- Posts: 201
- Joined: Mon Nov 19, 2012 11:16 am
Re: Replicate BIG files between QNAP NAS
I tried an experiment. I made a big file, backed ot up via rsync, made a change and backed it up again.
So the initial rsync backup sent 1,074,004,054 bytes and received 39 bytes.
A second backup of an unchanged file sent 131,170 bytes and received 229,427 bytes.
After changing 1024 bytes another backup sent 163,938 bytes and received 229,427 bytes.
Code: Select all
petert@dockerhost:~/src/big $ ls -la
total 1048596
drwxr-xr-x 2 petert petert 4096 Feb 2 16:23 .
drwxr-xr-x 9 petert petert 4096 Feb 2 15:53 ..
-rw-r--r-- 1 petert petert 1073741824 Feb 2 16:38 a.big.file
-rwxr-xr-x 1 petert petert 227 Feb 2 16:23 changebig.py
-rwxr-xr-x 1 petert petert 253 Feb 2 16:18 makebig.py
petert@dockerhost:~/src/big $ rsync -v a.big.file rsync@192.168.2.70::rsync/a.big.file
a.big.file
sent 1,074,004,054 bytes received 39 bytes 65,091,157.15 bytes/sec
total size is 1,073,741,824 speedup is 1.00
petert@dockerhost:~/src/big $ rsync -v a.big.file rsync@192.168.2.70::rsync/a.big.file
a.big.file
sent 131,170 bytes received 229,427 bytes 13,607.43 bytes/sec
total size is 1,073,741,824 speedup is 2,977.68
petert@dockerhost:~/src/big $ [b]./changebig.py [/b]
petert@dockerhost:~/src/big $ rsync -v a.big.file rsync@192.168.2.70::rsync/a.big.file
a.big.file
sent 163,938 bytes received 229,427 bytes 13,802.28 bytes/sec
total size is 1,073,741,824 speedup is 2,729.63
A second backup of an unchanged file sent 131,170 bytes and received 229,427 bytes.
After changing 1024 bytes another backup sent 163,938 bytes and received 229,427 bytes.
-
- New here
- Posts: 9
- Joined: Fri Feb 24, 2017 9:27 pm
Re: Replicate BIG files between QNAP NAS
Waouh, many thanks for that PeterT1959, this is exactly what i needed.
I read somewhere that this is the difference between RTRR and rsync.
Is the rsync used into qnap HBS3 such efficient ?
I'll go ahead and read your wikipedia link, i should learn many thinks
Great !
I read somewhere that this is the difference between RTRR and rsync.
Is the rsync used into qnap HBS3 such efficient ?
I'll go ahead and read your wikipedia link, i should learn many thinks
Great !
-
- Know my way around
- Posts: 201
- Joined: Mon Nov 19, 2012 11:16 am
Re: Replicate BIG files between QNAP NAS
All this testing was done from a Raspberry Pi (client) to QNAP (server). I enabled rsync on the QNAP.
Sent from my Pixel 4a using Tapatalk
Sent from my Pixel 4a using Tapatalk
-
- New here
- Posts: 9
- Joined: Fri Feb 24, 2017 9:27 pm
Re: Replicate BIG files between QNAP NAS
Hi and sorry for my late reply.
Many thanks for this clarification, this is greatly appreciated !
Now i have all needed to go forward
Have a nice day !
Many thanks for this clarification, this is greatly appreciated !
Now i have all needed to go forward
Have a nice day !