Which app use checksum/md5 when copy / synchronise ?

Backup, Restore, Netbak Replicator, Cloud Storage Services
User avatar
storageman
Ask me anything
Posts: 5511
Joined: Thu Sep 22, 2011 10:57 pm

Re: Which app use checksum/md5 when copy / synchronise ?

Post by storageman » Fri Sep 30, 2016 4:43 pm

Some of the newer large Qnap boxes will be offering ZFS file system (but I doubt it will be offered on smaller units).
This has "copy on write" functionality that generally offers greater data integrity than EXT4.
Nevertheless the kind of corruption you're talking about is quite rare and more likely to come from files corrupted elsewhere and copied to the NAS.

jfneuch
Starting out
Posts: 11
Joined: Mon Jun 15, 2015 7:37 am

Re: Which app use checksum/md5 when copy / synchronise ?

Post by jfneuch » Fri Sep 30, 2016 9:19 pm

Thanks you to answer my post. But I have lost very important files when transferring files without checking md5. Even if it is useless, I prefer loose time and check the integrity of the files before deleting the original. By the way, on Unbuty furum, ext4 doesn't have excellent reputation ( https://ubuntuforums.org/archive/index. ... 51603.html )

User avatar
Don
Guru
Posts: 12000
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: Which app use checksum/md5 when copy / synchronise ?

Post by Don » Sat Oct 01, 2016 12:27 am

You are referencing a post from April 2012. That is over 4 years ago. That is a lifetime in regards to computers. I'm sure a lot has changed since then.
So if you have system crashes or the electricity fails, be aware that ext4 by default will corrupt your files.

This can happen with any file system. Computers and file systems do not react well to system crashes and power failures. Power failures can be addressed with a UPS.
Read the Online Manuals and use the forum search feature before posting.

It is a recommended to use RAID and have external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

Submit bugs and feature requests to QNAP via their Helpdesk app.

NAS: TVS-882BR | F/W: 4.3.6.0895 | 40GB | 2 x M.2 SATA RAID 1 (System/VMs) | 4 x M.2 NMVe QM2-4P-384A RAID 5 (Cache) | 5 x 4TB HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-663 | F/W: 4.4.2.1320 | 16GB | 2 x M.2 NMVe QM2-2P RAID 1 (Cache) | 4 x 4TB RAID 5
Apps: Boinc, Squid, DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS, Entware, DLstation, +others

pollardd
Starting out
Posts: 29
Joined: Wed Jan 03, 2018 8:43 am

Re: Which app use checksum/md5 when copy / synchronise ?

Post by pollardd » Tue Sep 04, 2018 7:34 am

I know this post is from a couple of years ago.
I'm looking to perform the same verification of files I have copied to my QNAP drive.
I think what the original poster wants to do is valid.

RAID and another Backup isn't going to help you if the file was corrupted or somehow changed while being transferred to or stored on the QNAP device.
It is just going to copy the coruption.

My backup program creates a MD5 Checksum on a windows server when the backup is created and the pair of files are stored on a local QNAP disk.
I then use Backup Station Nas to Nas Sync to copy my files to an offsite location on another QNAP disk.

Periodically (once a week) I want to run over all my remote files and verify the files still calculate the same MD5 hash.
This way I know all my files are still just the same as when they were created down to the last little "bit". :)

When I find the answer I'll post it here as this post ranked 2nd on a Google search. (The first one was ten years old)

User avatar
storageman
Ask me anything
Posts: 5511
Joined: Thu Sep 22, 2011 10:57 pm

Re: Which app use checksum/md5 when copy / synchronise ?

Post by storageman » Tue Sep 04, 2018 5:55 pm

Use ISCSI if you want to use verification tools

sjohund
New here
Posts: 9
Joined: Tue Feb 05, 2019 2:30 am

Re: Which app use checksum/md5 when copy / synchronise ?

Post by sjohund » Tue Feb 05, 2019 2:33 am

Has anyone solved this issue? I will appreciate if someone did. If yes, how?

ElScOrChO79
New here
Posts: 2
Joined: Tue May 30, 2017 12:20 am

Re: Which app use checksum/md5 when copy / synchronise ?

Post by ElScOrChO79 » Mon Mar 04, 2019 5:00 am

I'm looking for the same thing as you. I've been looking for weeks to find a tool that check the file integrity of the backup and found nothing for QNAP.
I know that I can do that easily on windows but my Qnap was bought mostly for storage and backup easily to an external drive.

I find it weird that a system like QTS doesn't have that kind of tool. It also seems the same with synology

P3R
Guru
Posts: 12139
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Which app use checksum/md5 when copy / synchronise ?

Post by P3R » Mon Mar 04, 2019 5:29 am

Both Qnap Backup Station and Hybrid Backup Sync will do a byte-by-byte verification of source and destination to determine if a file need to be copied or not if specifically configured to do so. So run the job twice and if no files are copied in the second run you have verified that the source and destination are exactly the same.

Be careful though, byte-by-byte verification will of course be slow with large amounts of data.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

ElScOrChO79
New here
Posts: 2
Joined: Tue May 30, 2017 12:20 am

Re: Which app use checksum/md5 when copy / synchronise ?

Post by ElScOrChO79 » Wed Mar 06, 2019 10:31 pm

I found out that Synology has a feature on their Hyper Backup that they can check the files integrity on the backup. That would be a great feature to have in Hybrid Backup Sync
especially that they seem to do the checksum verification hidden. All they need to do is to store the hash of every destination file in a database.

I also found out that SyncBackPro also have that feature, it's sad that I cannot install it on the Qnap server.

P3R
Guru
Posts: 12139
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Which app use checksum/md5 when copy / synchronise ?

Post by P3R » Thu Mar 07, 2019 4:32 am

Checking file integrity can be different things and it can be done in different ways.

The OP specifically asked for a tool to compare files immediately after a local copy (external to internal disk) to make sure the files copied okay and that it's safe to delete the original.

As already explained that's easily solved in the Qnap by using Hybrid Backup Sync:
Select All Jobs, click + Create Job, select Create Sync Job, click the icon for Sync with local NAS, select the Destination folder, select a Source folder, Click Add (so that the folder pair is listed), then click Advanced Settings, click Policy, tick the check box for Check file contents, click Back, tick the check box for Sync Now and lastly click Apply.

The job will run and when it's finished (it says Finished in the Status column). At that point immediately click the Backup Now button to run the the same job a second time. When the second run have completed (it report Finished in the Status column) click the specific Job Name to see the Job Status. If the number of Backed up files report as zero the source and destination are verified to be identical. That's it, mission completed and the source files can be deleted.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

P3R
Guru
Posts: 12139
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Which app use checksum/md5 when copy / synchronise ?

Post by P3R » Thu Mar 07, 2019 4:42 am

ElScOrChO79 wrote:
Wed Mar 06, 2019 10:31 pm
I found out that Synology has a feature on their Hyper Backup that they can check the files integrity on the backup.
..//..
I also found out that SyncBackPro also have that feature, it's sad that I cannot install it on the Qnap server.
If you stop focusing on other products that don't run on a Qnap and instead explain exactly what it is you want to achive (instead of exactly how it should be achived) and why this would be useful for your particular workflow, then we can start to think about if there could be a solution that would work in your situation as well.

Maybe there is or maybe there isn't...

If there isn't maybe someone can suggest an even better workflow or an alternative solution that you haven't considered?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

gleep23
New here
Posts: 3
Joined: Sat May 16, 2020 12:22 pm

Re: Which app use checksum/md5 when copy / synchronise ?

Post by gleep23 » Sat May 16, 2020 12:55 pm

I'm astonished there is no md5 or checksum app onboard my NAS.

It seems insane that anyone in IT relies upon a file system to store files without corruption. Files don't reside on just on file system, they go all over the place, and wherever they land, there should be an independent check. I've never seen work environment that doesn't have checksum that work after transfer.

Even the most basic applications today come with a .md5 .SHA-256 .sfv or gpg to make sure your transfer was completed correctly.

I create checksum on my workstation, then transfer the files + checksum to the NAS.
Later I want to confirm the integrity of the files, and there is no tool to do it on the NAS!!?
So, I have run the checksum on my workstation at the network share files, which means, behind the scenes, I am downloading the entire file, checking it, then deleting it. That is ridiculous.

MikeLagit
Easy as a breeze
Posts: 289
Joined: Fri Mar 22, 2013 11:40 pm

Re: Which app use checksum/md5 when copy / synchronise ?

Post by MikeLagit » Sun May 17, 2020 1:37 pm

Winmerge seems to be a pretty good windows verification tool. But sure, having it do it at the fs level would be best...
Model: TS-877-1700 16GB
(HDD): [RAID-5] 4 x 4TB Seagate Ironwolf

Model:TS-569L 3GB
(HDD): [RAID-5] 4 x 3TB WD Red

User avatar
Moogle Stiltzkin
Ask me anything
Posts: 8962
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: Which app use checksum/md5 when copy / synchronise ?

Post by Moogle Stiltzkin » Sun May 17, 2020 1:59 pm

P3R wrote:
Mon Mar 04, 2019 5:29 am
Both Qnap Backup Station and Hybrid Backup Sync will do a byte-by-byte verification of source and destination to determine if a file need to be copied or not if specifically configured to do so. So run the job twice and if no files are copied in the second run you have verified that the source and destination are exactly the same.

Be careful though, byte-by-byte verification will of course be slow with large amounts of data.
with verification disabled, how does it behave if there already exist the same file name? which may or may not be same size.

Does it skip? or does it write over the file to replace it?

I wasn't sure, so i used the verification ticked, although it is slower with it enabled :S


i can however confirm that with verification, if the file exists at the destination it will not move the file over, according to the logs. it only moves any file that doesn't exist or does not match after verification.
NAS
[Main Server] QNAP TS-877 w. 4tb [ 3x HGST Deskstar NAS (HDN724040ALE640) & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A w. 5x 2TB Samsung F3 (HD203WI) EXT4 Raid5
[Backup] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) single disks.
[^] QNAP TS-659 Pro II
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-228
[^] QNAP TS-128
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Asus AC68U Router|100dl/50ul MBPS FTTH Internet | Win10, WC PC-Intel i7 920 Ivy bridge desktop (1x 512gb Samsung 850 Pro SSD + 1x 4tb HGST Ultrastar 7K4000)


Guides & articles
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin

P3R
Guru
Posts: 12139
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Which app use checksum/md5 when copy / synchronise ?

Post by P3R » Sun May 17, 2020 5:58 pm

Moogle Stiltzkin wrote:
Sun May 17, 2020 1:59 pm
with verification disabled, how does it behave if there already exist the same file name? which may or may not be same size.
If size and date/time match, it's skipped. If either is different, source will overwrite destination.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

Post Reply

Return to “Backup & Restore”