[ RClone ] [ CLI ] [ 1.43 ] RSync for Cloud Storage

This is the best place for community developers to publish their genius work. Your Apps enrich the QNAP Turbo NAS.
treefiddy
New here
Posts: 9
Joined: Sun Jun 26, 2016 6:00 pm

Re: [ RClone ] [ CLI ] [ 1.42 ] RSync for Cloud Storage

Postby treefiddy » Mon Aug 13, 2018 4:01 am

Hello QNAP_Stephane,

first of all i love rclone. Really great, stable and fast updated software. Thanks a lot for porting it to my lovely QNAP 453a. I use it daily!

I'm using rclone copy to upload files from my QNAP Nas but as soon as i upload files from the non installation drive of rclone i'm having issues with the temp folder. I have changed the --cache-tmp-upload-path but i still get the following error message:

2018/08/12 21:54:41 NOTICE: 39c44dsn8c: Removing partially written file on error: write /root/Private/8o2a4knhk42f1rdc/v59ivg9n8bi5nbule1/1g83crd3e61omb1d39c44dsn8c: no space left on device
2018/08/12 21:54:41 ERROR : somefile.file: Failed to copy: write /root/Private/: no space left on device


The problem is that rclone always uses the root folder of the installation drive, furthermore the root folder does not have much space (QNAP standard configuration).

I'm using the following command:

Code: Select all

rclone --config="/drive1/blabla/rclone.conf" copy /share/drive2/something privatecrypt:Backup/something --log-file=/share/drive1/logfile.txt --cache-tmp-upload-path /share/drive2/somethingelse


BUT it does work with the drive where rclone has been installed. In this case i do not get the error message at all (Also big files but i suppose rclone does not need to cache them separately since the files are already on the installation drive)

Do you have any idea what i am doing wrong here? THANKS!

Thanks a lot for your help!

User avatar
QNAP_Stephane
Experience counts
Posts: 2691
Joined: Wed Mar 27, 2013 1:00 am

Re: [ RClone ] [ CLI ] [ 1.42 ] RSync for Cloud Storage

Postby QNAP_Stephane » Mon Aug 20, 2018 5:33 pm

did you try export HOME to another location on user space ?
--------------------------------------------------------------------------
QnapClub AppCenter - https://www.qnapclub.eu
--------------------------------------------------------------------------

User avatar
QNAP_Stephane
Experience counts
Posts: 2691
Joined: Wed Mar 27, 2013 1:00 am

Re: [ RClone ] [ CLI ] [ 1.43 ] RSync for Cloud Storage

Postby QNAP_Stephane » Sun Sep 02, 2018 5:26 pm

updated
--------------------------------------------------------------------------
QnapClub AppCenter - https://www.qnapclub.eu
--------------------------------------------------------------------------

diehardbattery
New here
Posts: 5
Joined: Mon Aug 13, 2018 8:01 pm

Re: [ RClone ] [ CLI ] [ 1.43 ] RSync for Cloud Storage

Postby diehardbattery » Fri Sep 07, 2018 4:48 am

Is there a way to monitor the job? Currently I am doing an initial upload of about 5TB and would like to see the progress or any relevant messages. Also, I have the job setup to run daily in crontab, but the initial upload is going to take several days. How can I make sure that only one instance of rclone runs per job?

salexes
New here
Posts: 2
Joined: Sat Mar 31, 2018 9:45 pm

Re: [ RClone ] [ CLI ] [ 1.43 ] RSync for Cloud Storage

Postby salexes » Fri Sep 07, 2018 9:34 pm

How to change the default location of the rclone.conf file permanently ?

Cadish
Starting out
Posts: 19
Joined: Tue Dec 24, 2013 8:27 pm

Re: [ RClone ] [ CLI ] [ 1.43 ] RSync for Cloud Storage

Postby Cadish » Sat Sep 08, 2018 12:43 am

diehardbattery wrote:Is there a way to monitor the job? Currently I am doing an initial upload of about 5TB and would like to see the progress or any relevant messages. Also, I have the job setup to run daily in crontab, but the initial upload is going to take several days. How can I make sure that only one instance of rclone runs per job?

I'm using this to ensure only running 1 instance. I also can't monitor progress, except for outgoing traffic :?

Code: Select all

#!/bin/sh

LOCKDIR="/tmp/syncDataLock"

#Lock the file so only 1 instance runs
if mkdir "${LOCKDIR}" &>/dev/null; then
   
   #rclone command
   rclone sync XXXXXXXXX

   #remove the lockdir file
   rm -rf "${LOCKDIR}"

else

   exit 1
   
fi

User avatar
QNAP_Stephane
Experience counts
Posts: 2691
Joined: Wed Mar 27, 2013 1:00 am

Re: [ RClone ] [ CLI ] [ 1.43 ] RSync for Cloud Storage

Postby QNAP_Stephane » Sat Sep 08, 2018 4:28 pm

this is an enhancement feature may be to ask to Rclone developper on GitHub, i think
--------------------------------------------------------------------------
QnapClub AppCenter - https://www.qnapclub.eu
--------------------------------------------------------------------------

User avatar
nickiman
Know my way around
Posts: 158
Joined: Thu Nov 13, 2014 11:15 pm
Location: Madrid, Spain

Re: [ RClone ] [ CLI ] [ 1.43 ] RSync for Cloud Storage

Postby nickiman » Mon Oct 15, 2018 4:00 am

Hi all,

Could you show me how do you mount a folder with rclone automatically in every restart of the NAS?

I´ve tried to add in crontab this line:
@reboot /share/CACHEDEV1_DATA/.qpkg/RClone/rclone_mount.sh

But, when I restart the NAS nothing happens and the script is not executed and, when I check crontab with "crontab -l" I can´t see the line previously configured.
It´s like crontab doesn´t like this kind of entries @reboot

I´ve added other lines in my crontab without any problem but with @reboot it´s seems like It doesn´t work.

Can anyone help me?
Thanks.
TS-453 Pro - FW: 4.3.5.0728 build 20181013
6Tb WD Red WD6002FFWX (x3) with RAID 5

User avatar
OneCD
Experience counts
Posts: 4936
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [ RClone ] [ CLI ] [ 1.43 ] RSync for Cloud Storage

Postby OneCD » Mon Oct 15, 2018 4:28 am

nickiman wrote:It´s like crontab doesn´t like this kind of entries @reboot

Yes, @reboot is unavailable with QTS cron. :(

Add the mount command to your [autorun.sh] instead. You'll need a delay to load it after the RClone package starts.

Or, you could hack the RClone init script and add your mount command there.

Debian 'Stretch' on my production NAS (TS-569 Pro), with the backup NAS (TS-559 Pro+) to eventually follow. When that happens, I'll no longer use or support any QNAP firmware. Debian powerup/poweroff times are < 1 minute.

GitHub repos
one.cd.only@gmail.com

User avatar
nickiman
Know my way around
Posts: 158
Joined: Thu Nov 13, 2014 11:15 pm
Location: Madrid, Spain

Re: [ RClone ] [ CLI ] [ 1.43 ] RSync for Cloud Storage

Postby nickiman » Mon Oct 15, 2018 4:43 am

OneCD wrote:Yes, @reboot is unavailable with QTS cron. :(


Thank you @OneCD

OneCD wrote:Add the mount command to your [autorun.sh] instead. You'll need a delay to load it after the RClone package starts.

Or, you could hack the RClone init script and add your mount command there.


Sounds interesting... Could you show me how I could modify the init script to add my mount command when the package is started?

Regards
TS-453 Pro - FW: 4.3.5.0728 build 20181013
6Tb WD Red WD6002FFWX (x3) with RAID 5

User avatar
OneCD
Experience counts
Posts: 4936
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [ RClone ] [ CLI ] [ 1.43 ] RSync for Cloud Storage

Postby OneCD » Mon Oct 15, 2018 5:18 am

nickiman wrote:Sounds interesting... Could you show me how I could modify the init script to add my mount command when the package is started?

Your current init should look like:

Code: Select all

[~] # cat $(getcfg RClone Install_path -f /etc/config/qpkg.conf)/RClone.sh
#!/bin/sh
CONF=/etc/config/qpkg.conf
QPKG_NAME="RClone"
QPKG_ROOT=`/sbin/getcfg $QPKG_NAME Install_Path -f ${CONF}`
APACHE_ROOT=/share/`/sbin/getcfg SHARE_DEF defWeb -d Qweb -f /etc/config/def_share.info`

export QPKG_NAME QPKG_ROOT APACHE_ROOT 


case "$1" in
  start)
    ENABLED=$(/sbin/getcfg $QPKG_NAME Enable -u -d FALSE -f $CONF)
    if [ "$ENABLED" != "TRUE" ]; then
        echo "$QPKG_NAME is disabled."
        exit 1
    fi

/bin/ln -sf $QPKG_ROOT /opt/$QPKG_NAME
/bin/ln -sf $QPKG_ROOT/rclone /usr/bin/rclone


    ;;

  stop)

killall -9 rclone

rm -rf /opt/$QPKG_NAME
rm -rf /usr/bin/rclone

    ;;

  restart)
    $0 stop
    $0 start
    ;;

  *)
    echo "Usage: $0 {start|stop|restart}"
    exit 1
esac

exit 0

Add a call to your script at the end of the 'start' case block so it looks like:

Code: Select all

  start)
    ENABLED=$(/sbin/getcfg $QPKG_NAME Enable -u -d FALSE -f $CONF)
    if [ "$ENABLED" != "TRUE" ]; then
        echo "$QPKG_NAME is disabled."
        exit 1
    fi

/bin/ln -sf $QPKG_ROOT /opt/$QPKG_NAME
/bin/ln -sf $QPKG_ROOT/rclone /usr/bin/rclone

    $QPKG_ROOT/rclone_mount.sh
    ;;

If you also have a umount, you might consider adding it to the beginning of the 'stop' case block.

Debian 'Stretch' on my production NAS (TS-569 Pro), with the backup NAS (TS-559 Pro+) to eventually follow. When that happens, I'll no longer use or support any QNAP firmware. Debian powerup/poweroff times are < 1 minute.

GitHub repos
one.cd.only@gmail.com

User avatar
nickiman
Know my way around
Posts: 158
Joined: Thu Nov 13, 2014 11:15 pm
Location: Madrid, Spain

Re: [ RClone ] [ CLI ] [ 1.43 ] RSync for Cloud Storage

Postby nickiman » Mon Oct 15, 2018 6:10 am

OneCD wrote:Add a call to your script at the end of the 'start' case block so it looks like:

Code: Select all

  start)
    ENABLED=$(/sbin/getcfg $QPKG_NAME Enable -u -d FALSE -f $CONF)
    if [ "$ENABLED" != "TRUE" ]; then
        echo "$QPKG_NAME is disabled."
        exit 1
    fi

/bin/ln -sf $QPKG_ROOT /opt/$QPKG_NAME
/bin/ln -sf $QPKG_ROOT/rclone /usr/bin/rclone

    $QPKG_ROOT/rclone_mount.sh
    ;;

If you also have a umount, you might consider adding it to the beginning of the 'stop' case block.


Understood! Thank you very much OneCD. I´ve been reading also your script in GitHub regarding the autorun.sh... fantastic by the way... it can´t be more easy!
I´ve runned also just to have it prepared in case I needed in the future.

Thanks again for your help.
TS-453 Pro - FW: 4.3.5.0728 build 20181013
6Tb WD Red WD6002FFWX (x3) with RAID 5

User avatar
OneCD
Experience counts
Posts: 4936
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [ RClone ] [ CLI ] [ 1.43 ] RSync for Cloud Storage

Postby OneCD » Mon Oct 15, 2018 6:15 am

nickiman wrote:I´ve runned also just to have it prepared in case I needed in the future.

Thanks again for your help.

No worries.

Yep, forgot to mention the init script will probably be replaced next time you upgrade the RClone QPKG. ;)

Debian 'Stretch' on my production NAS (TS-569 Pro), with the backup NAS (TS-559 Pro+) to eventually follow. When that happens, I'll no longer use or support any QNAP firmware. Debian powerup/poweroff times are < 1 minute.

GitHub repos
one.cd.only@gmail.com

User avatar
nickiman
Know my way around
Posts: 158
Joined: Thu Nov 13, 2014 11:15 pm
Location: Madrid, Spain

Re: [ RClone ] [ CLI ] [ 1.43 ] RSync for Cloud Storage

Postby nickiman » Mon Oct 15, 2018 7:09 am

OneCD wrote:Yep, forgot to mention the init script will probably be replaced next time you upgrade the RClone QPKG. ;)


Yes, it´s true... I´ll take into account and I´ll make a backup of the init script to be safe.
TS-453 Pro - FW: 4.3.5.0728 build 20181013
6Tb WD Red WD6002FFWX (x3) with RAID 5

User avatar
nickiman
Know my way around
Posts: 158
Joined: Thu Nov 13, 2014 11:15 pm
Location: Madrid, Spain

Re: [ RClone ] [ CLI ] [ 1.43 ] RSync for Cloud Storage

Postby nickiman » Mon Oct 22, 2018 6:51 am

nickiman wrote:
OneCD wrote:Yep, forgot to mention the init script will probably be replaced next time you upgrade the RClone QPKG. ;)


Yes, it´s true... I´ll take into account and I´ll make a backup of the init script to be safe.


@OneCD

Thinking about the solution to implement... I´ve started to use the option about the autorun.sh because it should be more clean, and as we talked previously, in case of an upgrade of rclone I will not have to care about restore the init script to include the commands to mount my remotes.

So, I´ve copied my scripts to mount it in the path: /share/CACHEDEV1_DATA/.system/autorun/scripts

And, I´ve restarted the NAS but... I can´t see the remotes mounted.

Looking at the autorun.log I can see the scripts executed...

Code: Select all

Sun Oct 21 23:49:36 CEST 2018 ----- running autorun.sh -----
Sun Oct 21 23:49:36 CEST 2018 - /share/CACHEDEV1_DATA/.system/autorun/scripts/rclone_mount.sh
Sun Oct 21 23:49:36 CEST 2018 - /share/CACHEDEV1_DATA/.system/autorun/scripts/rclone_mount_TeamA.sh

so this makes me ask me if... probably rclone app was not started when the scripts were executed?

I was wondering also what if... if I install your package SortMyQPKGs in order to initialize rclone before to run the startup scripts... but, looking at your GitHub´s page, I´m unable to locate rclone in the ALPHA or OMEGA list.
Is it included?
In case not, could be included in ALPHA list in order to be initialized before to run the scripts from autorun.sh?

Keep waiting your comments. Thanks a lot!
TS-453 Pro - FW: 4.3.5.0728 build 20181013
6Tb WD Red WD6002FFWX (x3) with RAID 5


Return to “Community Apps”

Who is online

Users browsing this forum: No registered users and 2 guests