Which app use checksum/md5 when copy / synchronise ?

[storageman]

Some of the newer large Qnap boxes will be offering ZFS file system (but I doubt it will be offered on smaller units).
This has "copy on write" functionality that generally offers greater data integrity than EXT4.
Nevertheless the kind of corruption you're talking about is quite rare and more likely to come from files corrupted elsewhere and copied to the NAS.

[jfneuch]

Thanks you to answer my post. But I have lost very important files when transferring files without checking md5. Even if it is useless, I prefer loose time and check the integrity of the files before deleting the original. By the way, on Unbuty furum, ext4 doesn't have excellent reputation ( https://ubuntuforums.org/archive/index. ... 51603.html )

[Don]

You are referencing a post from April 2012. That is over 4 years ago. That is a lifetime in regards to computers. I'm sure a lot has changed since then.

So if you have system crashes or the electricity fails, be aware that ext4 by default will corrupt your files.

This can happen with any file system. Computers and file systems do not react well to system crashes and power failures. Power failures can be addressed with a UPS.

[pollardd]

I know this post is from a couple of years ago.
I'm looking to perform the same verification of files I have copied to my QNAP drive.
I think what the original poster wants to do is valid.

RAID and another Backup isn't going to help you if the file was corrupted or somehow changed while being transferred to or stored on the QNAP device.
It is just going to copy the coruption.

My backup program creates a MD5 Checksum on a windows server when the backup is created and the pair of files are stored on a local QNAP disk.
I then use Backup Station Nas to Nas Sync to copy my files to an offsite location on another QNAP disk.

Periodically (once a week) I want to run over all my remote files and verify the files still calculate the same MD5 hash.
This way I know all my files are still just the same as when they were created down to the last little "bit".

When I find the answer I'll post it here as this post ranked 2nd on a Google search. (The first one was ten years old)

[storageman]

Use ISCSI if you want to use verification tools

[sjohund]

Has anyone solved this issue? I will appreciate if someone did. If yes, how?

[ElScOrChO79]

I'm looking for the same thing as you. I've been looking for weeks to find a tool that check the file integrity of the backup and found nothing for QNAP.
I know that I can do that easily on windows but my Qnap was bought mostly for storage and backup easily to an external drive.

I find it weird that a system like QTS doesn't have that kind of tool. It also seems the same with synology

[P3R]

Both Qnap Backup Station and Hybrid Backup Sync will do a byte-by-byte verification of source and destination to determine if a file need to be copied or not if specifically configured to do so. So run the job twice and if no files are copied in the second run you have verified that the source and destination are exactly the same.

Be careful though, byte-by-byte verification will of course be slow with large amounts of data.

[ElScOrChO79]

I found out that Synology has a feature on their Hyper Backup that they can check the files integrity on the backup. That would be a great feature to have in Hybrid Backup Sync
especially that they seem to do the checksum verification hidden. All they need to do is to store the hash of every destination file in a database.

I also found out that SyncBackPro also have that feature, it's sad that I cannot install it on the Qnap server.

[P3R]

Checking file integrity can be different things and it can be done in different ways.

The OP specifically asked for a tool to compare files immediately after a local copy (external to internal disk) to make sure the files copied okay and that it's safe to delete the original.

As already explained that's easily solved in the Qnap by using Hybrid Backup Sync:
Select All Jobs, click + Create Job, select Create Sync Job, click the icon for Sync with local NAS, select the Destination folder, select a Source folder, Click Add (so that the folder pair is listed), then click Advanced Settings, click Policy, tick the check box for Check file contents, click Back, tick the check box for Sync Now and lastly click Apply.

The job will run and when it's finished (it says Finished in the Status column). At that point immediately click the Backup Now button to run the the same job a second time. When the second run have completed (it report Finished in the Status column) click the specific Job Name to see the Job Status. If the number of Backed up files report as zero the source and destination are verified to be identical. That's it, mission completed and the source files can be deleted.

[P3R]

I found out that Synology has a feature on their Hyper Backup that they can check the files integrity on the backup.
..//..
I also found out that SyncBackPro also have that feature, it's sad that I cannot install it on the Qnap server.

If you stop focusing on other products that don't run on a Qnap and instead explain exactly what it is you want to achive (instead of exactly how it should be achived) and why this would be useful for your particular workflow, then we can start to think about if there could be a solution that would work in your situation as well.

Maybe there is or maybe there isn't...

If there isn't maybe someone can suggest an even better workflow or an alternative solution that you haven't considered?

[gleep23]

I'm astonished there is no md5 or checksum app onboard my NAS.

It seems insane that anyone in IT relies upon a file system to store files without corruption. Files don't reside on just on file system, they go all over the place, and wherever they land, there should be an independent check. I've never seen work environment that doesn't have checksum that work after transfer.

Even the most basic applications today come with a .md5 .SHA-256 .sfv or gpg to make sure your transfer was completed correctly.

I create checksum on my workstation, then transfer the files + checksum to the NAS.
Later I want to confirm the integrity of the files, and there is no tool to do it on the NAS!!?
So, I have run the checksum on my workstation at the network share files, which means, behind the scenes, I am downloading the entire file, checking it, then deleting it. That is ridiculous.

[MikeLagit]

Winmerge seems to be a pretty good windows verification tool. But sure, having it do it at the fs level would be best...

[Moogle Stiltzkin]

Both Qnap Backup Station and Hybrid Backup Sync will do a byte-by-byte verification of source and destination to determine if a file need to be copied or not if specifically configured to do so. So run the job twice and if no files are copied in the second run you have verified that the source and destination are exactly the same.

Be careful though, byte-by-byte verification will of course be slow with large amounts of data.

with verification disabled, how does it behave if there already exist the same file name? which may or may not be same size.

Does it skip? or does it write over the file to replace it?

I wasn't sure, so i used the verification ticked, although it is slower with it enabled


i can however confirm that with verification, if the file exists at the destination it will not move the file over, according to the logs. it only moves any file that doesn't exist or does not match after verification.

[P3R]

with verification disabled, how does it behave if there already exist the same file name? which may or may not be same size.

If size and date/time match, it's skipped. If either is different, source will overwrite destination.