Local file checksum tool

Discussion on remote replication.
Locked
cybergio1974
Starting out
Posts: 32
Joined: Fri Jan 29, 2010 4:42 am

Local file checksum tool

Post by cybergio1974 »

Dear All !

I have a master backup TS-253 Pro and a slave backup TS-210 which are periodically aligning (Master over Slave) using the NAS 2 NAS backup option.
So far so good and I'm really happy about how it is working.

I only have a doubt which I would kindly ask you to help me with.
What if one or more files on the master are getting corrupted ?
Does it mean that the corrupted files will be recognized as modified and overwritten to the slave due to a change in the checksum after the corruption ?

For such a reason, I would like to know if a qpkg tool (or a tested script) exists which is able to:

1. Generate file checksum within a directory and including sub-folders (I'm interested in file by file checksum and not whole directory hasing as folder content might vary frequently).
2. Compare existing checklum file log with same local folder
3. Compare existing checksum file log with a remote folder (the slave backup target dir).

I proposed already something similar on this forum in 2012 but got no further news.
As I'm an optimist, I hope something might have moved in background while I temporarily dropped the topic ;)

http://forum.qnap.com/viewtopic.php?f=24&t=57443

Thank you very much in advance for your help.
Regards,

Giorgio (Italy)
User avatar
pwilson
Guru
Posts: 22533
Joined: Fri Mar 06, 2009 11:20 am
Location: Victoria, BC, Canada (UTC-08:00)

Re: Local file checksum tool

Post by pwilson »

cybergio1974 wrote:Dear All !

I have a master backup TS-253 Pro and a slave backup TS-210 which are periodically aligning (Master over Slave) using the NAS 2 NAS backup option.
So far so good and I'm really happy about how it is working.

I only have a doubt which I would kindly ask you to help me with.
What if one or more files on the master are getting corrupted ?
Does it mean that the corrupted files will be recognized as modified and overwritten to the slave due to a change in the checksum after the corruption ?

For such a reason, I would like to know if a qpkg tool (or a tested script) exists which is able to:

1. Generate file checksum within a directory and including sub-folders (I'm interested in file by file checksum and not whole directory hasing as folder content might vary frequently).
2. Compare existing checklum file log with same local folder
3. Compare existing checksum file log with a remote folder (the slave backup target dir).

I proposed already something similar on this forum in 2012 but got no further news.
As I'm an optimist, I hope something might have moved in background while I temporarily dropped the topic ;)

http://forum.qnap.com/viewtopic.php?f=24&t=57443

Thank you very much in advance for your help.
Regards,

Giorgio (Italy)
Your NAS already automagically makes configuration backups every Wednesday at 0304hrs (local time). You merely need to backup these backups externally by whatever means you want. Check out the /share/XXXX_DATA/.@config_backup/ directory. On my TS-470 it is at: /share/CACHEDEV1_DATA/.@backup_config/, but depending on model and storage volume choices made the NASAdmin it could also be located at: /share/HDA_DATA/.@backup_config/, or /share/MD0_DATA/.@backup_config/.


From my NAS:

Code: Select all

admin@NASTY2:/share/CACHEDEV1_DATA# ls -alF /share/CACHEDEV1_DATA/.@backup_config/
total 2180
drwxr-xr-x  2 admin administrators   4096 2015-01-14 03:04 ./
drwxrwxrwx 39 admin administrators   4096 2014-12-30 06:54 ../
-rw-r--r--  1 admin administrators 396931 2015-01-14 03:04 0_20150114_0304.tar.gz
-rw-r--r--  1 admin administrators 394304 2015-01-07 03:04 1_20150107_0304.tar.gz
-rw-r--r--  1 admin administrators 386002 2014-12-31 03:04 2_20141231_0304.tar.gz
-rw-r--r--  1 admin administrators 387585 2014-12-24 03:04 3_20141224_0304.tar.gz
-rw-r--r--  1 admin administrators 384478 2014-12-17 03:04 4_20141217_0304.tar.gz
-rw-r--r--  1 admin administrators  51756 2014-11-19 03:04 5_20141119_0304.tar.gz
-rw-r--r--  1 admin administrators  51531 2014-11-12 03:04 6_20141112_0304.tar.gz
-rw-------  1 admin administrators  52280 2014-11-05 03:04 7_20141105_0304.tar.gz
-rw-------  1 admin administrators  52231 2014-10-29 03:04 8_20141029_0304.tar.gz
-rw-r--r--  1 admin administrators  51247 2014-10-22 03:04 9_20141022_0304.tar.gz
admin@NASTY2:/share/CACHEDEV1_DATA# 

Patrick M. Wilson
Victoria, BC Canada
QNAP TS-470 Pro w/ 4 * Western Digital WD30EFRX WD Reds (RAID5) - - Single 8.1TB Storage Pool FW: QTS 4.2.0 Build 20151023 - Kali Linux v1.06 (64bit)
Forums: View My Profile - Search My Posts - View My Photo - View My Location - Top Community Posters
QNAP: Turbo NAS User Manual - QNAP Wiki - QNAP Tutorials - QNAP FAQs

Please review: When you're asking a question, please include the following.
cybergio1974
Starting out
Posts: 32
Joined: Fri Jan 29, 2010 4:42 am

Re: Local file checksum tool

Post by cybergio1974 »

Hello Patrick,

I have the doubt you wanted replying to a different post.
If I misunderstood, sorry but I could not catch your hint...
Regards,

Giorgio
User avatar
pwilson
Guru
Posts: 22533
Joined: Fri Mar 06, 2009 11:20 am
Location: Victoria, BC, Canada (UTC-08:00)

Re: Local file checksum tool

Post by pwilson »

cybergio1974 wrote:I have the doubt you wanted replying to a different post.
If I misunderstood, sorry but I could not catch your hint...
You were asking about CRC's or Checksums of files, in order to verify your backups. I was simply pointing out that the NAS already makes Configuration Backups for you, so you don't need to do this at all. Simply backup the backups, and call it done.

Otherwise try the "cksum" command:

Code: Select all

/sbin/cksum --help
Usage: /sbin/cksum [FILE]...
  or:  /sbin/cksum [OPTION]
Print CRC checksum and byte counts of each FILE.

      --help     display this help and exit
      --version  output version information and exit

Report bugs to <bug-coreutils@gnu.org>.

Patrick M. Wilson
Victoria, BC Canada
QNAP TS-470 Pro w/ 4 * Western Digital WD30EFRX WD Reds (RAID5) - - Single 8.1TB Storage Pool FW: QTS 4.2.0 Build 20151023 - Kali Linux v1.06 (64bit)
Forums: View My Profile - Search My Posts - View My Photo - View My Location - Top Community Posters
QNAP: Turbo NAS User Manual - QNAP Wiki - QNAP Tutorials - QNAP FAQs

Please review: When you're asking a question, please include the following.
cybergio1974
Starting out
Posts: 32
Joined: Fri Jan 29, 2010 4:42 am

Re: Local file checksum tool

Post by cybergio1974 »

Hello Patrick,

Indeed I'm not interested in any configuration backup but only about my personal data.
I already excluded the cksum command as this is not powerful enough for what I need: as I wrote, I'm looking for a recursive subfolder checksumming.

Thank you anyway but my question to the forum is still open.

Regards,

Giorgio
User avatar
pwilson
Guru
Posts: 22533
Joined: Fri Mar 06, 2009 11:20 am
Location: Victoria, BC, Canada (UTC-08:00)

Re: Local file checksum tool

Post by pwilson »

cybergio1974 wrote:Hello Patrick,

Indeed I'm not interested in any configuration backup but only about my personal data.
I already excluded the cksum command as this is not powerful enough for what I need: as I wrote, I'm looking for a recursive subfolder checksumming.

Thank you anyway but my question to the forum is still open.

Regards,

Giorgio
So write a "loop" into your script to do that, writing loops isn't difficult. You might find the "find" and "xargs" commands useful for this task.
Linux scripts are typically build out of many small utilities.

Patrick M. Wilson
Victoria, BC Canada
QNAP TS-470 Pro w/ 4 * Western Digital WD30EFRX WD Reds (RAID5) - - Single 8.1TB Storage Pool FW: QTS 4.2.0 Build 20151023 - Kali Linux v1.06 (64bit)
Forums: View My Profile - Search My Posts - View My Photo - View My Location - Top Community Posters
QNAP: Turbo NAS User Manual - QNAP Wiki - QNAP Tutorials - QNAP FAQs

Please review: When you're asking a question, please include the following.
cybergio1974
Starting out
Posts: 32
Joined: Fri Jan 29, 2010 4:42 am

Re: Local file checksum tool

Post by cybergio1974 »

Hello,

Sounds quite easy from what you write !
Can you show me an example from from your own experience then ?
I need the generation of a checksum for any file in any subfolder from a defined path which needs to be selectable by the user.
All checksum hashes need to be listed in a single file.
I'm looking forward for your resolutive solution proposal.
Indeed if I was able to code it on my own I would not even have posted the question on the forum or asked for a tool to do it (as you fan read in te subject)....

Thank you in advance.

Best Regards,
Giorgio
User avatar
pwilson
Guru
Posts: 22533
Joined: Fri Mar 06, 2009 11:20 am
Location: Victoria, BC, Canada (UTC-08:00)

Re: Local file checksum tool

Post by pwilson »

cybergio1974 wrote:Hello,

Sounds quite easy from what you write !
Can you show me an example from from your own experience then ?
I need the generation of a checksum for any file in any subfolder from a defined path which needs to be selectable by the user.
All checksum hashes need to be listed in a single file.
I'm looking forward for your resolutive solution proposal.
Indeed if I was able to code it on my own I would not even have posted the question on the forum or asked for a tool to do it (as you fan read in te subject)....

Thank you in advance.

Best Regards,
Giorgio
That is more complicated than you originally presented, but it is still "doable". I however am not prepared to write this for you. You have well defined requirements, so I'm sure you can "hire" a programmer to do this for you. The NAS already has all the tools required, so it is merely a matter of writing the script to use the tools required.

Patrick M. Wilson
Victoria, BC Canada
QNAP TS-470 Pro w/ 4 * Western Digital WD30EFRX WD Reds (RAID5) - - Single 8.1TB Storage Pool FW: QTS 4.2.0 Build 20151023 - Kali Linux v1.06 (64bit)
Forums: View My Profile - Search My Posts - View My Photo - View My Location - Top Community Posters
QNAP: Turbo NAS User Manual - QNAP Wiki - QNAP Tutorials - QNAP FAQs

Please review: When you're asking a question, please include the following.
Maddin2
Starting out
Posts: 11
Joined: Tue Oct 21, 2014 4:05 am

Re: Local file checksum tool

Post by Maddin2 »

I am also looking for such an app - and i would expect it to be available from QNAP.
In my opinion it is a quite obvious task to backup a NAS e.g. to an external USB - and then to verify from time to time that file checksums are still intact on both sides.
Only this way a deteriorating file could be detected and replaced by the backup (hopefully intact) - or vice versa.
The calculation of the checksum could and should be done by the NAS itself.

There are tools available which do that on a PC, but then lots of TB have to be transported over the line just for creating checksum.
It should be done at nighttime when the NAS is not busy anyway.
Best regards,
Martin
mibmob
New here
Posts: 2
Joined: Sat May 18, 2013 6:28 pm

Re: Local file checksum tool

Post by mibmob »

There is a version of sha1sum (in busybox) on the qnap that gives the same result as many windows tools.

on windows ...
CRCSHA (part of the full installation of 7zip). After a full 7zip install "CRC SHA" appears on the windows on context menu (right click on a file)

and another

certutil -hashfile (windows native tool?)
MalEbenSo
Starting out
Posts: 34
Joined: Sat Jul 25, 2015 7:45 pm

Re: Local file checksum tool

Post by MalEbenSo »

I am looking for a feature like this, too. Running on the QNAP and in a user-friendly fashion.

Something that will provide the data integrity features (detect issues at least, not necessarily auto-correct them) built into ZFS and btrfs.

A recursive checksum should do this, as has been suggested. This would be relatively light-weight, I assume, and would work with any file system.
2x QNAP TS-431P2, 1x QNAP TS-212P
Firmware current as provided by qnap
Various Seagate 8 TB HDDs (w/ and w/o SMR) configured as single disks
DLNA via Twonky. Replication via RTRR.
MalEbenSo
Starting out
Posts: 34
Joined: Sat Jul 25, 2015 7:45 pm

Re: Local file checksum tool

Post by MalEbenSo »

It appears that on the Linux side dm-integrity was built for this purpose. But it is functionality in the kernel, not user space. So QNAP would have to add it.

https://gitlab.com/cryptsetup/cryptsetu ... MIntegrity
2x QNAP TS-431P2, 1x QNAP TS-212P
Firmware current as provided by qnap
Various Seagate 8 TB HDDs (w/ and w/o SMR) configured as single disks
DLNA via Twonky. Replication via RTRR.
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: Local file checksum tool

Post by Moogle Stiltzkin »

not sure what that does, but like the original user, i wish it could generate a file with the checksum so i could do further verification down the road.

when zfs hero comes out, it ought to auto detect checksum error and fix them.

however, if for some reason that ram begins to fail, and you happen to use the deduplication feature which keeps some of the meta, i would like a separate check for me to do some verification in that scenario.

right now i use an app called den4b renamer and manually generate a md5 crc hash tag into the filename. i can then run a md5 checksum verifier to confirm it's the same file. md5 is useless from a security standpoint, but still sufficient if i'm just checking that the file did not get corrupted. in the past this has helped me.
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
empty.dates
First post
Posts: 1
Joined: Sat Feb 03, 2024 7:46 pm

Re: Local file checksum tool

Post by empty.dates »

I had the same requirements in the past for Windows and was using a self written tool.

Later one, I have come up with a shell script for Linux.

It still under testing and do not produce the same result as Windows tool (formatting issue and file sorting difference between Windows and Linux tools) but if you have only one OS to check - you can try something like script below (run from ssh):

folderContent.sh

Code: Select all

#!/bin/sh

#############################
#  FUNCTIONS                #
#############################

ProcessFile ()
{
  echo "file -$1 -> $folderContent"
  sha1sumOut=$(sha1sum "$1")

  sha1=${sha1sumOut:0:40}			# take sha1 part
  fileName=${sha1sumOut:42}			# take file path part
  fileName=${fileName#"$targetDir/"}	# remove currentDir from path
  fileName=${fileName//[\/]/\\}		# replace linux path slash to windows path slash / -> \

  sha1=$(echo "$sha1" | awk '{print toupper($0)}') # upcase sha1 value
  res="$sha1 - $fileName"			# format like in windows tool

  echo $res
  echo $res >> "$folderContent"
}

ProcessDirectory ()
{
  for dir in "$1"/* ;do
    [ -d "$dir" ] && ProcessDirectory "$dir"
  done

  for file in "$1"/* ;do
    [ -f "$file" ] && ProcessFile "$file"
  done
}

#############################
#  MAIN                     #
#############################

shopt  -s dotglob					# enable hidden files in output

targetDir=$(pwd)
currentTime=$(date "+%Y%m%d%H%M%S")
folderContent="$targetDir/folderContent_$currentTime.txt"

echo "Directory: $targetDir"
echo "Report: $folderContent"

ProcessDirectory "$targetDir"

echo "Done."
It will generate a file in the same folder "folderContent_YYYYmmDDHHMMSS.txt"

with content like:

...
B4914D481A845E52C3FD89FB15B208BFFA7BA2B5 - <relative path>\FileName1
AB4CBEFE2FFA15C33A6456743E3626D28E55C7E6 - <relative path>\FileName2
AE5359C3F84160541D362F9A395C09EE3953C20E - <relative path>\FileName3
...

I am executing this from time to time and compare old result with new.
You can sort file by hash - it will simplify comparison if order of files are changed... this work still in progress.
Locked

Return to “Remote Replication/ Disaster Recovery”