Local file checksum tool

Discussion on remote replication.
Post Reply
cybergio1974
Starting out
Posts: 32
Joined: Fri Jan 29, 2010 4:42 am

Local file checksum tool

Post by cybergio1974 » Tue Jan 20, 2015 7:41 am

Dear All !

I have a master backup TS-253 Pro and a slave backup TS-210 which are periodically aligning (Master over Slave) using the NAS 2 NAS backup option.
So far so good and I'm really happy about how it is working.

I only have a doubt which I would kindly ask you to help me with.
What if one or more files on the master are getting corrupted ?
Does it mean that the corrupted files will be recognized as modified and overwritten to the slave due to a change in the checksum after the corruption ?

For such a reason, I would like to know if a qpkg tool (or a tested script) exists which is able to:

1. Generate file checksum within a directory and including sub-folders (I'm interested in file by file checksum and not whole directory hasing as folder content might vary frequently).
2. Compare existing checklum file log with same local folder
3. Compare existing checksum file log with a remote folder (the slave backup target dir).

I proposed already something similar on this forum in 2012 but got no further news.
As I'm an optimist, I hope something might have moved in background while I temporarily dropped the topic ;)

http://forum.qnap.com/viewtopic.php?f=24&t=57443

Thank you very much in advance for your help.
Regards,

Giorgio (Italy)

User avatar
pwilson
Guru
Posts: 22568
Joined: Fri Mar 06, 2009 11:20 am
Location: Victoria, BC, Canada (UTC-08:00)

Re: Local file checksum tool

Post by pwilson » Tue Jan 20, 2015 11:08 am

cybergio1974 wrote:Dear All !

I have a master backup TS-253 Pro and a slave backup TS-210 which are periodically aligning (Master over Slave) using the NAS 2 NAS backup option.
So far so good and I'm really happy about how it is working.

I only have a doubt which I would kindly ask you to help me with.
What if one or more files on the master are getting corrupted ?
Does it mean that the corrupted files will be recognized as modified and overwritten to the slave due to a change in the checksum after the corruption ?

For such a reason, I would like to know if a qpkg tool (or a tested script) exists which is able to:

1. Generate file checksum within a directory and including sub-folders (I'm interested in file by file checksum and not whole directory hasing as folder content might vary frequently).
2. Compare existing checklum file log with same local folder
3. Compare existing checksum file log with a remote folder (the slave backup target dir).

I proposed already something similar on this forum in 2012 but got no further news.
As I'm an optimist, I hope something might have moved in background while I temporarily dropped the topic ;)

http://forum.qnap.com/viewtopic.php?f=24&t=57443

Thank you very much in advance for your help.
Regards,

Giorgio (Italy)


Your NAS already automagically makes configuration backups every Wednesday at 0304hrs (local time). You merely need to backup these backups externally by whatever means you want. Check out the /share/XXXX_DATA/.@config_backup/ directory. On my TS-470 it is at: /share/CACHEDEV1_DATA/.@backup_config/, but depending on model and storage volume choices made the NASAdmin it could also be located at: /share/HDA_DATA/.@backup_config/, or /share/MD0_DATA/.@backup_config/.


From my NAS:

Code: Select all

admin@NASTY2:/share/CACHEDEV1_DATA# ls -alF /share/CACHEDEV1_DATA/.@backup_config/
total 2180
drwxr-xr-x  2 admin administrators   4096 2015-01-14 03:04 ./
drwxrwxrwx 39 admin administrators   4096 2014-12-30 06:54 ../
-rw-r--r--  1 admin administrators 396931 2015-01-14 03:04 0_20150114_0304.tar.gz
-rw-r--r--  1 admin administrators 394304 2015-01-07 03:04 1_20150107_0304.tar.gz
-rw-r--r--  1 admin administrators 386002 2014-12-31 03:04 2_20141231_0304.tar.gz
-rw-r--r--  1 admin administrators 387585 2014-12-24 03:04 3_20141224_0304.tar.gz
-rw-r--r--  1 admin administrators 384478 2014-12-17 03:04 4_20141217_0304.tar.gz
-rw-r--r--  1 admin administrators  51756 2014-11-19 03:04 5_20141119_0304.tar.gz
-rw-r--r--  1 admin administrators  51531 2014-11-12 03:04 6_20141112_0304.tar.gz
-rw-------  1 admin administrators  52280 2014-11-05 03:04 7_20141105_0304.tar.gz
-rw-------  1 admin administrators  52231 2014-10-29 03:04 8_20141029_0304.tar.gz
-rw-r--r--  1 admin administrators  51247 2014-10-22 03:04 9_20141022_0304.tar.gz
admin@NASTY2:/share/CACHEDEV1_DATA#

Patrick M. Wilson
Victoria, BC Canada
QNAP TS-470 Pro w/ 4 * Western Digital WD30EFRX WD Reds (RAID5) - - Single 8.1TB Storage Pool FW: QTS 4.2.0 Build 20151023 - Kali Linux v1.06 (64bit)
Forums: View My Profile - Search My Posts - View My Photo - View My Location - Top Community Posters
QNAP: Turbo NAS User Manual - QNAP Wiki - QNAP Tutorials - QNAP FAQs

Please review: When you're asking a question, please include the following.

cybergio1974
Starting out
Posts: 32
Joined: Fri Jan 29, 2010 4:42 am

Re: Local file checksum tool

Post by cybergio1974 » Wed Jan 21, 2015 5:23 am

Hello Patrick,

I have the doubt you wanted replying to a different post.
If I misunderstood, sorry but I could not catch your hint...
Regards,

Giorgio

User avatar
pwilson
Guru
Posts: 22568
Joined: Fri Mar 06, 2009 11:20 am
Location: Victoria, BC, Canada (UTC-08:00)

Re: Local file checksum tool

Post by pwilson » Wed Jan 21, 2015 9:52 am

cybergio1974 wrote:I have the doubt you wanted replying to a different post.
If I misunderstood, sorry but I could not catch your hint...


You were asking about CRC's or Checksums of files, in order to verify your backups. I was simply pointing out that the NAS already makes Configuration Backups for you, so you don't need to do this at all. Simply backup the backups, and call it done.

Otherwise try the "cksum" command:

Code: Select all

/sbin/cksum --help
Usage: /sbin/cksum [FILE]...
  or:  /sbin/cksum [OPTION]
Print CRC checksum and byte counts of each FILE.

      --help     display this help and exit
      --version  output version information and exit

Report bugs to <bug-coreutils@gnu.org>.

Patrick M. Wilson
Victoria, BC Canada
QNAP TS-470 Pro w/ 4 * Western Digital WD30EFRX WD Reds (RAID5) - - Single 8.1TB Storage Pool FW: QTS 4.2.0 Build 20151023 - Kali Linux v1.06 (64bit)
Forums: View My Profile - Search My Posts - View My Photo - View My Location - Top Community Posters
QNAP: Turbo NAS User Manual - QNAP Wiki - QNAP Tutorials - QNAP FAQs

Please review: When you're asking a question, please include the following.

cybergio1974
Starting out
Posts: 32
Joined: Fri Jan 29, 2010 4:42 am

Re: Local file checksum tool

Post by cybergio1974 » Thu Jan 22, 2015 7:03 am

Hello Patrick,

Indeed I'm not interested in any configuration backup but only about my personal data.
I already excluded the cksum command as this is not powerful enough for what I need: as I wrote, I'm looking for a recursive subfolder checksumming.

Thank you anyway but my question to the forum is still open.

Regards,

Giorgio

User avatar
pwilson
Guru
Posts: 22568
Joined: Fri Mar 06, 2009 11:20 am
Location: Victoria, BC, Canada (UTC-08:00)

Re: Local file checksum tool

Post by pwilson » Thu Jan 22, 2015 7:21 am

cybergio1974 wrote:Hello Patrick,

Indeed I'm not interested in any configuration backup but only about my personal data.
I already excluded the cksum command as this is not powerful enough for what I need: as I wrote, I'm looking for a recursive subfolder checksumming.

Thank you anyway but my question to the forum is still open.

Regards,

Giorgio


So write a "loop" into your script to do that, writing loops isn't difficult. You might find the "find" and "xargs" commands useful for this task.
Linux scripts are typically build out of many small utilities.

Patrick M. Wilson
Victoria, BC Canada
QNAP TS-470 Pro w/ 4 * Western Digital WD30EFRX WD Reds (RAID5) - - Single 8.1TB Storage Pool FW: QTS 4.2.0 Build 20151023 - Kali Linux v1.06 (64bit)
Forums: View My Profile - Search My Posts - View My Photo - View My Location - Top Community Posters
QNAP: Turbo NAS User Manual - QNAP Wiki - QNAP Tutorials - QNAP FAQs

Please review: When you're asking a question, please include the following.

cybergio1974
Starting out
Posts: 32
Joined: Fri Jan 29, 2010 4:42 am

Re: Local file checksum tool

Post by cybergio1974 » Fri Jan 23, 2015 4:03 pm

Hello,

Sounds quite easy from what you write !
Can you show me an example from from your own experience then ?
I need the generation of a checksum for any file in any subfolder from a defined path which needs to be selectable by the user.
All checksum hashes need to be listed in a single file.
I'm looking forward for your resolutive solution proposal.
Indeed if I was able to code it on my own I would not even have posted the question on the forum or asked for a tool to do it (as you fan read in te subject)....

Thank you in advance.

Best Regards,
Giorgio

User avatar
pwilson
Guru
Posts: 22568
Joined: Fri Mar 06, 2009 11:20 am
Location: Victoria, BC, Canada (UTC-08:00)

Re: Local file checksum tool

Post by pwilson » Sat Jan 24, 2015 4:30 am

cybergio1974 wrote:Hello,

Sounds quite easy from what you write !
Can you show me an example from from your own experience then ?
I need the generation of a checksum for any file in any subfolder from a defined path which needs to be selectable by the user.
All checksum hashes need to be listed in a single file.
I'm looking forward for your resolutive solution proposal.
Indeed if I was able to code it on my own I would not even have posted the question on the forum or asked for a tool to do it (as you fan read in te subject)....

Thank you in advance.

Best Regards,
Giorgio


That is more complicated than you originally presented, but it is still "doable". I however am not prepared to write this for you. You have well defined requirements, so I'm sure you can "hire" a programmer to do this for you. The NAS already has all the tools required, so it is merely a matter of writing the script to use the tools required.

Patrick M. Wilson
Victoria, BC Canada
QNAP TS-470 Pro w/ 4 * Western Digital WD30EFRX WD Reds (RAID5) - - Single 8.1TB Storage Pool FW: QTS 4.2.0 Build 20151023 - Kali Linux v1.06 (64bit)
Forums: View My Profile - Search My Posts - View My Photo - View My Location - Top Community Posters
QNAP: Turbo NAS User Manual - QNAP Wiki - QNAP Tutorials - QNAP FAQs

Please review: When you're asking a question, please include the following.

Maddin2
Starting out
Posts: 10
Joined: Tue Oct 21, 2014 4:05 am

Re: Local file checksum tool

Post by Maddin2 » Sun Mar 20, 2016 7:58 pm

I am also looking for such an app - and i would expect it to be available from QNAP.
In my opinion it is a quite obvious task to backup a NAS e.g. to an external USB - and then to verify from time to time that file checksums are still intact on both sides.
Only this way a deteriorating file could be detected and replaced by the backup (hopefully intact) - or vice versa.
The calculation of the checksum could and should be done by the NAS itself.

There are tools available which do that on a PC, but then lots of TB have to be transported over the line just for creating checksum.
It should be done at nighttime when the NAS is not busy anyway.
Best regards,
Martin

mibmob
New here
Posts: 2
Joined: Sat May 18, 2013 6:28 pm

Re: Local file checksum tool

Post by mibmob » Sat Mar 31, 2018 4:03 pm

There is a version of sha1sum (in busybox) on the qnap that gives the same result as many windows tools.

on windows ...
CRCSHA (part of the full installation of 7zip). After a full 7zip install "CRC SHA" appears on the windows on context menu (right click on a file)

and another

certutil -hashfile (windows native tool?)

MalEbenSo
Starting out
Posts: 34
Joined: Sat Jul 25, 2015 7:45 pm

Re: Local file checksum tool

Post by MalEbenSo » Tue Jan 14, 2020 5:13 pm

I am looking for a feature like this, too. Running on the QNAP and in a user-friendly fashion.

Something that will provide the data integrity features (detect issues at least, not necessarily auto-correct them) built into ZFS and btrfs.

A recursive checksum should do this, as has been suggested. This would be relatively light-weight, I assume, and would work with any file system.
2x QNAP TS-431P2, 1x QNAP TS-212P
Firmware current as provided by qnap
Various Seagate 8 TB HDDs (w/ and w/o SMR) configured as single disks
DLNA via Twonky. Replication via RTRR.

MalEbenSo
Starting out
Posts: 34
Joined: Sat Jul 25, 2015 7:45 pm

Re: Local file checksum tool

Post by MalEbenSo » Wed Jan 15, 2020 11:34 pm

It appears that on the Linux side dm-integrity was built for this purpose. But it is functionality in the kernel, not user space. So QNAP would have to add it.

https://gitlab.com/cryptsetup/cryptsetu ... MIntegrity
2x QNAP TS-431P2, 1x QNAP TS-212P
Firmware current as provided by qnap
Various Seagate 8 TB HDDs (w/ and w/o SMR) configured as single disks
DLNA via Twonky. Replication via RTRR.

User avatar
Moogle Stiltzkin
Ask me anything
Posts: 9026
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: Local file checksum tool

Post by Moogle Stiltzkin » Thu Jan 16, 2020 12:02 am

not sure what that does, but like the original user, i wish it could generate a file with the checksum so i could do further verification down the road.

when zfs hero comes out, it ought to auto detect checksum error and fix them.

however, if for some reason that ram begins to fail, and you happen to use the deduplication feature which keeps some of the meta, i would like a separate check for me to do some verification in that scenario.

right now i use an app called den4b renamer and manually generate a md5 crc hash tag into the filename. i can then run a md5 checksum verifier to confirm it's the same file. md5 is useless from a security standpoint, but still sufficient if i'm just checking that the file did not get corrupted. in the past this has helped me.
NAS
[Main Server] QNAP TS-877 w. 4tb [ 3x HGST Deskstar NAS (HDN724040ALE640) & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A w. 5x 2TB Samsung F3 (HD203WI) EXT4 Raid5
[Backup] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) single disks.
[^] QNAP TS-659 Pro II
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-228
[^] QNAP TS-128
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Asus AC68U Router|100dl/50ul MBPS FTTH Internet | Win10, WC PC-Intel i7 920 Ivy bridge desktop (1x 512gb Samsung 850 Pro SSD + 1x 4tb HGST Ultrastar 7K4000)


Guides & articles
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin

Post Reply

Return to “Remote Replication/ Disaster Recovery”