Generate Automatic File Listing

Discussion on remote replication.
Post Reply
chikala
New here
Posts: 7
Joined: Thu Jun 18, 2015 10:55 pm

Generate Automatic File Listing

Post by chikala » Mon Mar 04, 2019 7:12 am

Hi
I have a TS212P and would like to know how I can generate a daily list of all files on the NAS in a text file named by today's date e.g. FileListDD-MM-YY.txt
My NAS is pretty old so I don't have a lot of options for installing packages?
I just need some pointers to get started.
Thanks
Richard

User avatar
OneCD
Ask me anything
Posts: 6405
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: Generate Automatic File Listing

Post by OneCD » Mon Mar 04, 2019 7:25 am

A task easily done with BASH shell-scripting and a cron job. :geek:

production NAS: TS-569 Pro with Debian 9.9 'Stretch' (power on/off times are < 1 minute)
backup NAS: TS-559 Pro+ with QTS 4.2.6 #20190921

one.cd.only@gmail.com

Image Image Image Image

chikala
New here
Posts: 7
Joined: Thu Jun 18, 2015 10:55 pm

Re: Generate Automatic File Listing

Post by chikala » Mon Mar 04, 2019 10:42 pm

OneCD wrote:
Mon Mar 04, 2019 7:25 am
A task easily done with BASH shell-scripting and a cron job. :geek:
Ok thanks for the info.
I'm happy to knock up a script, but what is a good environment to test this in?
Directly from the command line of the NAS?
I have put together this:
find ./ | sed -e 's/[^-][^\/]*\//--/g;s/--/ |-/' >/share/File Lists/$( date +"%Y%m%d" ).filelist

Once this works I am happy to setup a cron job.
Thanks

iam@nas
Easy as a breeze
Posts: 267
Joined: Wed Jun 15, 2016 2:49 am

Re: Generate Automatic File Listing

Post by iam@nas » Tue Mar 05, 2019 1:01 am

but what is a good environment to test this in?
Yes, but in a small sub-directory where find runs fast.
With "find . -type f" only files and no directories get into the list. I'd avoid the space in "/File Lists/". Formatting the items with sed like you do will make it harder to copy the line to access the file.

User avatar
OneCD
Ask me anything
Posts: 6405
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: Generate Automatic File Listing

Post by OneCD » Tue Mar 05, 2019 3:35 am

I use this to record the contents of my archive (offline) drives. This makes use of 'tree' and creates a HTML file, as per the below code-snippet:

Code: Select all

tree --dirsfirst --du --nolinks -I '*.tbn|*.nfo|*.jpg' -T "$currentdir contains:" -hnHCp . -o "$filename"
The default QTS doesn't contain 'tree', but it can be installed by first installing the Entware QPKG, then:

Code: Select all

opkg install tree

production NAS: TS-569 Pro with Debian 9.9 'Stretch' (power on/off times are < 1 minute)
backup NAS: TS-559 Pro+ with QTS 4.2.6 #20190921

one.cd.only@gmail.com

Image Image Image Image

chikala
New here
Posts: 7
Joined: Thu Jun 18, 2015 10:55 pm

Re: Generate Automatic File Listing

Post by chikala » Tue Mar 05, 2019 8:39 pm

OneCD wrote:
Tue Mar 05, 2019 3:35 am
I use this to record the contents of my archive (offline) drives. This makes use of 'tree' and creates a HTML file, as per the below code-snippet:

Code: Select all

tree --dirsfirst --du --nolinks -I '*.tbn|*.nfo|*.jpg' -T "$currentdir contains:" -hnHCp . -o "$filename"
The default QTS doesn't contain 'tree', but it can be installed by first installing the Entware QPKG, then:

Code: Select all

opkg install tree
Hi
Thanks for the info.
I have a TS212P
I don't believe I can install Entware on this device?
Can you please confirm?
Thanks

User avatar
OneCD
Ask me anything
Posts: 6405
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: Generate Automatic File Listing

Post by OneCD » Wed Mar 06, 2019 2:49 am

chikala wrote:
Tue Mar 05, 2019 8:39 pm
I have a TS212P
I don't believe I can install Entware on this device?
Can you please confirm?
Yes, it can be installed.

Did you try it? ;)

production NAS: TS-569 Pro with Debian 9.9 'Stretch' (power on/off times are < 1 minute)
backup NAS: TS-559 Pro+ with QTS 4.2.6 #20190921

one.cd.only@gmail.com

Image Image Image Image

chikala
New here
Posts: 7
Joined: Thu Jun 18, 2015 10:55 pm

Re: Generate Automatic File Listing

Post by chikala » Wed Mar 06, 2019 9:33 pm

OneCD wrote:
Wed Mar 06, 2019 2:49 am
chikala wrote:
Tue Mar 05, 2019 8:39 pm
I have a TS212P
I don't believe I can install Entware on this device?
Can you please confirm?
Yes, it can be installed.

Did you try it? ;)
Thanks I got it installed and can play around with this script. :ashamed:
Thanks for the help!

I tried to run your code snippet but get the result:

tree: invalid filename ''

I'm almost there, thanks again.

iam@nas
Easy as a breeze
Posts: 267
Joined: Wed Jun 15, 2016 2:49 am

Re: Generate Automatic File Listing

Post by iam@nas » Wed Mar 06, 2019 9:52 pm

You may want to initialize the variables '$filename' and '$currentdir'.

chikala
New here
Posts: 7
Joined: Thu Jun 18, 2015 10:55 pm

Re: Generate Automatic File Listing

Post by chikala » Wed Mar 06, 2019 11:38 pm

iam@nas wrote:
Wed Mar 06, 2019 9:52 pm
You may want to initialize the variables '$filename' and '$currentdir'.
Sorry I have never written anything in this language. :ashamed:
I'm learning slowly...

I want this to run every day at midnight.
Please can you guide me in roughly the right direction now that I have tree running...
Step 1. I Write and debug a .sh file, E.G. DirectoryListing.Sh to create the daily HTML file in a directory that I can backup.
I'm after a full HTML list of every user file present on my NAS.
My Script so far is:

#!/bin/sh
####################################
#
# List all files NAS TS212-P
# Write these files to a time stamped HTML file.
#
####################################
# Create listing filename.
date_time_stamp=$(date +%d%m%Y_%H%M%S")
hostname=$(hostname -s)
filename="$hostname-$date_time_stamp.html"
tree --dirsfirst --du --nolinks -hnHCp . -o "$filename"

What does $currentdir do as you use it in your script?

Step 2. Make the file executable - I'll do this after I get the script working.

Step 3. Create a cronjob to carry this task out.
I try to navigate to:
/etc/config/crontab
But I get permission denied.
I am logged in as admin.
Is crontab -e and then adding an entry the way to go for this?

4. Manage the HTML files that will be created every day. I guess another scheduled job that deletes the files in the file listing directory if they are older than x days?

iam@nas
Easy as a breeze
Posts: 267
Joined: Wed Jun 15, 2016 2:49 am

Re: Generate Automatic File Listing

Post by iam@nas » Thu Mar 07, 2019 12:26 am

Configure the PATH in your script. When crond calls it you can never be sure how it is set. For example:

Code: Select all

PATH=/opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr/sbin
Use "cd /..." to select the right path, or "cd /" if you really want to scan all files.

Make the script executable: "chmod 755 ./scriptname.sh" - otherwise crond can not execute it.
If you need to debug everything: "bash -x ./scriptname.sh". All debug code is written to STDERR.
If you need to debug a section in a script:

Code: Select all

set -x
a=3
set +x
https://linux.die.net/man/1/tree contains the options you may use with tree.


Make a backup of /etc/config/crontab as you may need it.

Code: Select all

cp /etc/config/crontab /etc/config/crontab.backup.`date -u +%y%m%d` # make the backup of the crontab file, use the ISO 8601 date format. Easier to sort than d.m.y.
crontab -l >/etc/config/crontab2.backup.`date -u +%y%m%d` # backup of the active crontab
diff /etc/config/crontab.backup.`date -u +%y%m%d` /etc/config/crontab2.backup.`date -u +%y%m%d` # These file should be identical otherwise the following commands may remove/add additional commands to the crontab 
printf "%s" "0 0 * * * /path-to-your-file/file.sh" >>/etc/config/crontab # Add one line to the crontab, adjust the path etc.
crontab /etc/config/crontab # Re-create the crontab from this file
/etc/init.d/crond.sh restart # restart the crond
At least for me 'crontab -e' does not update the /etc/config/crontab file.

To restore the current setup (specify the date of the backup):

Code: Select all

cp /etc/config/crontab.backup.201903... /etc/config/crontab
crontab /etc/config/crontab2.backup.201903...
/etc/init.d/crond.sh restart 

Post Reply

Return to “Remote Replication/ Disaster Recovery”