[ Fdupes ] [ 2.2.1 ] a program for identifying duplicate files residing within specified directories.

This is the best place for community developers to publish their genius work. Your Apps enrich the QNAP Turbo NAS.
User avatar
QNAP_Stephane
Experience counts
Posts: 4802
Joined: Wed Mar 27, 2013 1:00 am

[ Fdupes ] [ 2.2.1 ] a program for identifying duplicate files residing within specified directories.

Post by QNAP_Stephane »

Image
Source : https://github.com/adrianlopezroche/fdupes

Download :

https://www.myqnap.org/product/fdupes-cli/

Note :

delivered with a WebUi, just launch a scan in /share for duplicate files and export in text file in your Public share (/share/Public/fdupes.output)
fdupes command line is automatically added to NAS $PATH for easy to manage over SSH

Warning : Scan process is very long ... and a big output file may crash the webui results display (just check results file directly within your share)

About :

FDUPES is a program for identifying duplicate files residing
within specified directories.

Code: Select all

Usage: fdupes [options] DIRECTORY...

 -r --recurse     	for every directory given follow subdirectories
                  	encountered within
 -R --recurse:    	for each directory given after this option follow
                  	subdirectories encountered within (note the ':' at
                  	the end of the option, manpage for more details)
 -s --symlinks    	follow symlinks
 -H --hardlinks   	normally, when two or more files point to the same
                  	disk area they are treated as non-duplicates; this
                  	option will change this behavior
 -n --noempty     	exclude zero-length files from consideration
 -A --nohidden    	exclude hidden files from consideration
 -f --omitfirst   	omit the first file in each set of matches
 -1 --sameline    	list each set of matches on a single line
 -S --size        	show size of duplicate files
 -m --summarize   	summarize dupe information
 -q --quiet       	hide progress indicator
 -d --delete      	prompt user for files to preserve and delete all
                  	others; important: under particular circumstances,
                  	data may be lost when using this option together
                  	with -s or --symlinks, or when specifying a
                  	particular directory more than once; refer to the
                  	fdupes documentation for additional information
 -N --noprompt    	together with --delete, preserve the first file in
                  	each set of duplicates and delete the rest without
                  	prompting the user
 -I --immediate   	delete duplicates as they are encountered, without
                  	grouping into sets; implies --noprompt
 -p --permissions 	don't consider files with different owner/group or
                  	permission bits as duplicates
 -o --order=BY    	select sort order for output and deleting; by file
                  	modification time (BY='time'; default), status
                  	change time (BY='ctime'), or filename (BY='name')
 -i --reverse     	reverse order while sorting
 -v --version     	display fdupes version
 -h --help        	display this help message

Unless -1 or --sameline is specified, duplicate files are listed 
together in groups, each file displayed on a separate line. The
groups are then separated from each other by blank lines.

When -1 or --sameline is specified, spaces and backslash characters (\) 
appearing in a filename are preceded by a backslash character. For
instance, "with spaces" becomes "with\ spaces".

When using -d or --delete, care should be taken to insure against
accidental data loss. While no information will be immediately
lost, using this option together with -s or --symlink can lead 
to confusing information being presented to the user when prompted
for files to preserve. Specifically, a user could accidentally
preserve a symlink while deleting the file it points to. A similar
problem arises when specifying a particular directory more than 
once. All files within that directory will be listed as their own
duplicates, leading to data loss should a user preserve a file 
without its "duplicate" (the file itself!).
Last edited by QNAP_Stephane on Fri Jan 06, 2023 10:22 pm, edited 2 times in total.
---------------------------------------------------------------------------------------------------------------------------

Find all QPKG in the MyQnap.org repository https://www.myqnap.org
join our discord server for any inquiry related to qpkg - https://discord.gg/4fPxHSWKQW

----------------------------------------------------------------------------------------------------------------------------
giopas
Been there, done that
Posts: 855
Joined: Thu Mar 26, 2015 12:36 am
Location: somewhere in EU

Re: [ Fdupes ] [ 1.61 ] a program for identifying duplicate files residing within specified directories.

Post by giopas »

Thanks Stéphane ! It would have been even better to have it embedded in Qsirch but meanwhile, this is awesome!!!

Sent from my A0001 using Tapatalk
Qnap TS-253Pro 16Gb RAM - Single Storage Pool: 2 WRed 4TB (RAID 1)
Qnap TS-453A 16Gb RAM - Single Storage Pool: 4 WRed 4TB (RAID 5)
blowfish
Starting out
Posts: 19
Joined: Fri Apr 21, 2017 9:53 am

Re: [ Fdupes ] [ 1.61 ] a program for identifying duplicate files residing within specified directories.

Post by blowfish »

Hello Stephane,

(Presuming Stephane will view this:) I take it you're a pretty significant part of the QNAP world. So, thank you in advance for all your efforts.

I have just installed your FDupes 1.61 pkg (downloading the 64bit for my TVS-473 running QTS 4.3.3.0154).

Unfortunately, after installing, when attempting to run, all I get is;

Code: Select all

Error
Page not found or the web server is currently unavailable. Please contact the website administrator for help.
the std error page. I am not running the admin UI on 8080, nor 80. Is this the issue? Does the admin UI need to be running on 80? If I change the port in the <nasname>/fdupes URL (opens in a new tab. Should it be doing this?) to be the same as the admin UI, no difference. I get the same result.

I did originally simply d'load and install whichever qpkg showed up in the QNAP Club repository, but figuring that was probably the 32bit ver. I uninstalled it and manually installed the 64bit pkg directly d'loaded from the link in post 1.

What could possibly be the issue pls?

N.B. whilst not new to Linux, completely new to QNAP.

Assistance from anyone, will be greatly appreciated.
User avatar
QNAP_Stephane
Experience counts
Posts: 4802
Joined: Wed Mar 27, 2013 1:00 am

Re: [ Fdupes ] [ 1.61 ] a program for identifying duplicate files residing within specified directories.

Post by QNAP_Stephane »

It only use the embedded webserver... check to renable it in qts ...it is probably disable then restart qpkg

Envoyé de mon SM-G925F en utilisant Tapatalk
---------------------------------------------------------------------------------------------------------------------------

Find all QPKG in the MyQnap.org repository https://www.myqnap.org
join our discord server for any inquiry related to qpkg - https://discord.gg/4fPxHSWKQW

----------------------------------------------------------------------------------------------------------------------------
blowfish
Starting out
Posts: 19
Joined: Fri Apr 21, 2017 9:53 am

Re: [ Fdupes ] [ 1.61 ] a program for identifying duplicate files residing within specified directories.

Post by blowfish »

Ah! DOH!

Many thanks.
User avatar
phiku
Starting out
Posts: 27
Joined: Thu Jan 08, 2009 5:20 pm
Location: Mainz (Germany)

Re: [ Fdupes ] [ 1.61 ] a program for identifying duplicate files residing within specified directories.

Post by phiku »

I've installing fdupes and run it over the webpage about one Day but the text-file is still empty - 0 Bytes! What have I done wrong or should I run it in SSH-Session? Firmware is 4.3.3.0188.
TS-451A
- LDAP
- Music Station, Plex, Photo Station, Virtualization, etc
- Sonos Music
- LACP with TP-Link SG2008
- RAID 1
QGenie: powerbank and photo storage for shootings
User avatar
QNAP_Stephane
Experience counts
Posts: 4802
Joined: Wed Mar 27, 2013 1:00 am

Re: [ Fdupes ] [ 1.61 ] a program for identifying duplicate files residing within specified directories.

Post by QNAP_Stephane »

you installed the x64 version ?

try over ssh the command line
---------------------------------------------------------------------------------------------------------------------------

Find all QPKG in the MyQnap.org repository https://www.myqnap.org
join our discord server for any inquiry related to qpkg - https://discord.gg/4fPxHSWKQW

----------------------------------------------------------------------------------------------------------------------------
User avatar
mydjtl
Getting the hang of things
Posts: 95
Joined: Wed Jan 11, 2017 7:52 am

Re: [ Fdupes ] [ 1.61 ] a program for identifying duplicate files residing within specified directories.

Post by mydjtl »

once you get the dupes output text how do you make it delete all but one version of file? what ssh commands can you enter? do i have to enter cd first for specific folders? thanks in advance
TS-251+
(2) Seagate IronWolf 12TB 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drives
8gb RAM in one slot. 4.3.3
Firmware as of 3/8/2020: 4.4.1.1216 (2/14/2020)
trastle
New here
Posts: 3
Joined: Fri Oct 09, 2015 11:46 pm

Re: [ Fdupes ] [ 1.61 ] a program for identifying duplicate files residing within specified directories.

Post by trastle »

Hi,

Trying to install this package causes a seg fault.

Code: Select all

# qpkg_cli -m Fdupes_1.61_x86.qpkg
Segmentation fault
Firmware: 4.3.3.0361

Any ideas?

EDIT: NAS Model: TS453 Pro
Last edited by trastle on Sun Sep 23, 2018 5:31 am, edited 1 time in total.
User avatar
Don
Guru
Posts: 12289
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: [ Fdupes ] [ 1.61 ] a program for identifying duplicate files residing within specified directories.

Post by Don »

What model NAS?
Use the forum search feature before posting.

Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
trastle
New here
Posts: 3
Joined: Fri Oct 09, 2015 11:46 pm

Re: [ Fdupes ] [ 1.61 ] a program for identifying duplicate files residing within specified directories.

Post by trastle »

TS-453 Pro
User avatar
Don
Guru
Posts: 12289
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: [ Fdupes ] [ 1.61 ] a program for identifying duplicate files residing within specified directories.

Post by Don »

Try upgrading to the latest released firmware (non beta). Also use the 64 bit QPKG. Isn’t the 453 64bit?
Use the forum search feature before posting.

Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
User avatar
OneCD
Guru
Posts: 12010
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [ Fdupes ] [ 1.61 ] a program for identifying duplicate files residing within specified directories.

Post by OneCD »

trastle wrote:Trying to install this package causes a seg fault.

Code: Select all

# qpkg_cli -m Fdupes_1.61_x86.qpkg
Segmentation fault
QPKGs should be installed like this:

Code: Select all

sh Fdupes_1.61_x86.qpkg

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage
trastle
New here
Posts: 3
Joined: Fri Oct 09, 2015 11:46 pm

Re: [ Fdupes ] [ 1.61 ] a program for identifying duplicate files residing within specified directories.

Post by trastle »

Thanks @onecd and @don

Turns out the UI based install was failing due to /tmp being full and I was not installing correctly from the terminal.
Once I cleared out /tmp the installer worked perfectly.

Thanks for your help.

Code: Select all

# sh Fdupes_1.61_x86_64.qpkg
Install QNAP package on TS-NAS...
./
./built_info
./qinstall.sh
./package_routines
./qpkg.cfg
328+1 records in
328+1 records out
7763599 bytes (7.4MB) copied, 0.066653 seconds, 111.1MB/s
7581+1 records in
7581+1 records out
7763599 bytes (7.4MB) copied, 0.067479 seconds, 109.7MB/s
Fdupes 1.61 is already installed. Setup will now perform package upgrading.
Firmware check is fine.
Link service start/stop script: Fdupes.sh
Set QPKG information in /etc/config/qpkg.conf
[App Center] Fdupes 1.61 has been installed in /share/CACHEDEV2_DATA/.qpkg/Fdupes successfully.
[App Center] Fdupes enabled.
Laffer
Starting out
Posts: 17
Joined: Thu Jan 25, 2018 8:15 pm

Re: [ Fdupes ] [ 1.61 ] a program for identifying duplicate files residing within specified directories.

Post by Laffer »

I tried to install via AppCenter, but it tells me Wrong Architecture: Fdupes is build for arm-x31. I have a QNAP TS 431P2. What did I wrong?
Post Reply

Return to “Community Apps”