Which app use checksum/md5 when copy / synchronise ?

Backup, Restore, Netbak Replicator, Cloud Storage Services
User avatar
storageman
Ask me anything
Posts: 5488
Joined: Thu Sep 22, 2011 10:57 pm

Re: Which app use checksum/md5 when copy / synchronise ?

Post by storageman » Fri Sep 30, 2016 4:43 pm

Some of the newer large Qnap boxes will be offering ZFS file system (but I doubt it will be offered on smaller units).
This has "copy on write" functionality that generally offers greater data integrity than EXT4.
Nevertheless the kind of corruption you're talking about is quite rare and more likely to come from files corrupted elsewhere and copied to the NAS.

jfneuch
Starting out
Posts: 11
Joined: Mon Jun 15, 2015 7:37 am

Re: Which app use checksum/md5 when copy / synchronise ?

Post by jfneuch » Fri Sep 30, 2016 9:19 pm

Thanks you to answer my post. But I have lost very important files when transferring files without checking md5. Even if it is useless, I prefer loose time and check the integrity of the files before deleting the original. By the way, on Unbuty furum, ext4 doesn't have excellent reputation ( https://ubuntuforums.org/archive/index. ... 51603.html )

User avatar
Don
Guru
Posts: 11986
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: Which app use checksum/md5 when copy / synchronise ?

Post by Don » Sat Oct 01, 2016 12:27 am

You are referencing a post from April 2012. That is over 4 years ago. That is a lifetime in regards to computers. I'm sure a lot has changed since then.
So if you have system crashes or the electricity fails, be aware that ext4 by default will corrupt your files.

This can happen with any file system. Computers and file systems do not react well to system crashes and power failures. Power failures can be addressed with a UPS.
Read the Online Manuals and use the forum search feature before posting.

It is a recommended to use RAID and have external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

Submit bugs and feature requests to QNAP via their Helpdesk app.

NAS: TVS-882BR | F/W: 4.3.6.0895 | 40GB | 2 x M.2 SATA RAID 1 (System/VMs) | 4 x M.2 NMVe QM2-4P-384A RAID 5 (Cache) | 5 x 4TB HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-663 | F/W: 4.4.1.1086 | 16GB | 2 x M.2 NMVe QM2-2P RAID 1 (Cache) | 4 x 4TB RAID 5
Apps: Boinc, Squid, DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS, Entware, DLstation, +others

pollardd
Starting out
Posts: 29
Joined: Wed Jan 03, 2018 8:43 am

Re: Which app use checksum/md5 when copy / synchronise ?

Post by pollardd » Tue Sep 04, 2018 7:34 am

I know this post is from a couple of years ago.
I'm looking to perform the same verification of files I have copied to my QNAP drive.
I think what the original poster wants to do is valid.

RAID and another Backup isn't going to help you if the file was corrupted or somehow changed while being transferred to or stored on the QNAP device.
It is just going to copy the coruption.

My backup program creates a MD5 Checksum on a windows server when the backup is created and the pair of files are stored on a local QNAP disk.
I then use Backup Station Nas to Nas Sync to copy my files to an offsite location on another QNAP disk.

Periodically (once a week) I want to run over all my remote files and verify the files still calculate the same MD5 hash.
This way I know all my files are still just the same as when they were created down to the last little "bit". :)

When I find the answer I'll post it here as this post ranked 2nd on a Google search. (The first one was ten years old)

User avatar
storageman
Ask me anything
Posts: 5488
Joined: Thu Sep 22, 2011 10:57 pm

Re: Which app use checksum/md5 when copy / synchronise ?

Post by storageman » Tue Sep 04, 2018 5:55 pm

Use ISCSI if you want to use verification tools

sjohund
New here
Posts: 9
Joined: Tue Feb 05, 2019 2:30 am

Re: Which app use checksum/md5 when copy / synchronise ?

Post by sjohund » Tue Feb 05, 2019 2:33 am

Has anyone solved this issue? I will appreciate if someone did. If yes, how?

ElScOrChO79
New here
Posts: 2
Joined: Tue May 30, 2017 12:20 am

Re: Which app use checksum/md5 when copy / synchronise ?

Post by ElScOrChO79 » Mon Mar 04, 2019 5:00 am

I'm looking for the same thing as you. I've been looking for weeks to find a tool that check the file integrity of the backup and found nothing for QNAP.
I know that I can do that easily on windows but my Qnap was bought mostly for storage and backup easily to an external drive.

I find it weird that a system like QTS doesn't have that kind of tool. It also seems the same with synology

P3R
Guru
Posts: 11722
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Which app use checksum/md5 when copy / synchronise ?

Post by P3R » Mon Mar 04, 2019 5:29 am

Both Qnap Backup Station and Hybrid Backup Sync will do a byte-by-byte verification of source and destination to determine if a file need to be copied or not if specifically configured to do so. So run the job twice and if no files are copied in the second run you have verified that the source and destination are exactly the same.

Be careful though, byte-by-byte verification will of course be slow with large amounts of data.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

ElScOrChO79
New here
Posts: 2
Joined: Tue May 30, 2017 12:20 am

Re: Which app use checksum/md5 when copy / synchronise ?

Post by ElScOrChO79 » Wed Mar 06, 2019 10:31 pm

I found out that Synology has a feature on their Hyper Backup that they can check the files integrity on the backup. That would be a great feature to have in Hybrid Backup Sync
especially that they seem to do the checksum verification hidden. All they need to do is to store the hash of every destination file in a database.

I also found out that SyncBackPro also have that feature, it's sad that I cannot install it on the Qnap server.

P3R
Guru
Posts: 11722
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Which app use checksum/md5 when copy / synchronise ?

Post by P3R » Thu Mar 07, 2019 4:32 am

Checking file integrity can be different things and it can be done in different ways.

The OP specifically asked for a tool to compare files immediately after a local copy (external to internal disk) to make sure the files copied okay and that it's safe to delete the original.

As already explained that's easily solved in the Qnap by using Hybrid Backup Sync:
Select All Jobs, click + Create Job, select Create Sync Job, click the icon for Sync with local NAS, select the Destination folder, select a Source folder, Click Add (so that the folder pair is listed), then click Advanced Settings, click Policy, tick the check box for Check file contents, click Back, tick the check box for Sync Now and lastly click Apply.

The job will run and when it's finished (it says Finished in the Status column). At that point immediately click the Backup Now button to run the the same job a second time. When the second run have completed (it report Finished in the Status column) click the specific Job Name to see the Job Status. If the number of Backed up files report as zero the source and destination are verified to be identical. That's it, mission completed and the source files can be deleted.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

P3R
Guru
Posts: 11722
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Which app use checksum/md5 when copy / synchronise ?

Post by P3R » Thu Mar 07, 2019 4:42 am

ElScOrChO79 wrote:
Wed Mar 06, 2019 10:31 pm
I found out that Synology has a feature on their Hyper Backup that they can check the files integrity on the backup.
..//..
I also found out that SyncBackPro also have that feature, it's sad that I cannot install it on the Qnap server.
If you stop focusing on other products that don't run on a Qnap and instead explain exactly what it is you want to achive (instead of exactly how it should be achived) and why this would be useful for your particular workflow, then we can start to think about if there could be a solution that would work in your situation as well.

Maybe there is or maybe there isn't...

If there isn't maybe someone can suggest an even better workflow or an alternative solution that you haven't considered?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

Post Reply

Return to “Backup & Restore”