Backup/Sync App for Amazon Cloud Drive

[rgibson]

+1 Just purchased unlimited storage from Amazon Cloud Drive. Need to sync my Qnap with Amazon now.

[ecke83]

+1 from me for this feature.

https://developer.amazon.com/public/api ... oud-drive/

[cthulhu944]

+1 -- Really need a sync function. I can see that you can connect in file station, but you can't do any syncing in that route. Really need a sync to amazon cloud drive.

usecases I'm interested in:

Sync music folder from amazon cloud to a folder on my NAS
Sync a folder on my NAS to a folder on my Amazon cloud drive
Sync a folder or files on my NAS onto Amazon cloud drive encrypted.

Thanks

[JustRob]

Doesn't QNAP already support Amazon Cloud Drive? I have installed the plugin to File Station, and created the Amazon Cloud Drive link. In the admin shell, I noted a "CloudDrive" mount:
CloudDrive on /mnt/rf/cd type fuse (rw,nosuid,nodev,allow_other,default_permissions)

[/mnt/rf/cd/0/0123456789abcdef0123456789abcdef01234567]

I can see all the files that I have "rsynced" are in my Amazon Cloud Drive (changed the key above), but when I try to do an rsync, using my "rsa" alias as below:
alias rsa="rsync -rlptDv --stats --progress --delete-before --delete-excluded --force --exclude='.@__thumb/' --exclude='@Recycle/'"

I notice that it seems that some of the files are resynced that are already there, and the transfer rate is much faster than my broadband connection should allow (i.e., 30-40MBps - [bytes] whereas my FiOS broadband only allows 75Mbps [best case ~9MBps]). I also don't believe the files physically exist locally but remotely, but I don't understand entirely how this mount is "mounted", and if a checksum (i.e., "rsync -c") is supported remotely or must the file be downloaded and checksum performed locally on both the local and remote file system. The checksum should be performed remotely of the remote files to keep from having to re-download all the files. I guess I could perform the rsync with "-c" and look at the bandwidth indicators to tell.

[/mnt/rf/cd] # df .
Filesystem Size Used Available Use% Mounted on
CloudDrive 2.0M 2.0M 1.0M 67% /mnt/rf/cd

[/mnt/rf/cd] # du -sh .
783G .

I created the Amazon Cloud Drive link from this link.
Can someone explain? Thanks!!
BTW, I was unable to find or download the beta "Cloud Drive Sync" for my TS-451+ even thought it was supposed to be released Q3/2015. I have the following firmwares:
[~] # echo "Firmware: QTS: $(getcfg system version) Build $(getcfg system 'Build Number')"
Firmware: QTS: 4.2.1 Build 20151216
[~] # echo "HybridDesk Station: $(getcfg -f /etc/config/qpkg.conf HD_Station version)"
HybridDesk Station: 3.1.0

**UPDATE** Using the rsync alias above and adding the "-c" parameter of a sample 2GB file that exists on both Amazon Cloud Drive and locally shows that the remote file is actually downloaded and checksum generated locally (undesired) versus the checksum generated remotely (highly desired). Without a checksum, how can we ensure that the files were transferred successfully? Crashplan does the checksum locally of local files and remotely for remote files, and this is preferred for obvious bandwidth savings.

[felizf]

guys try this out...
https://www.youtube.com/watch?v=hqutVc2Tork

[JustRob]

Cool video, using RTRR and a little bit of a kludge. I have tried something similar using "rsync" but I find that after a while it seems that rsync slows to a snail's crawl. That's likely because of the queued-up files. I only have two concerns: 1) after a while, I see a bunch of errors where files show as not having transferred, and most importantly, 2) no checksum comparison of the files (correct me if I'm wrong!). However, the rclone command supports MD5 checksums of the files in the cloud, which is what I want to do (versus a local "rsync -c" which first downloads the cloud files, creates the local file checksum, then compares; if it matches it ignores, otherwise it re-uploads the file).

I think I have this working well now... I have created symlinks to the appropriate local directories I want in my home directory and the rclone config file has a connection to Amazon Cloud Drive as named "ACD:". Here is what I am doing (first line is the location of the binaries - note that on x86_64 platforms, amd64 v1.26 currently DOES NOT WORK - it doesn't correctly authenticate with Amazon):

Create a symbolic link from /usr/bin to your rclone binary like this (first parameter after "-s" is actual location of rclone, second parameter is the symbolic link location):
ln -s /file/location/of/rclone /usr/bin/rclone

Next, using "vi", create a file called rclone-exclude.txt with the contents below:
$RECYCLE.BIN
.\@__thumb/**
\@Recycle/**
Thumbs.db

Now do this:
rclone sync ~/Files/backup ACD:/backup --bwlimit 8M --checkers=16 --contimeout=5m --transfers=8 --exclude-from=/root/rclone-exclude.txt --retries=100 --stats 10s --drive-chunk-size 1024k --delete-excluded -c --dry-run -v

Once it goes through this in verbose mode and you're satisfied that it is doing what it is supposed to do, remove the "-v". Once it plans to do what you think it's supposed to send (i.e., directories are right), get rid of the "--dry-run". Once you've done the first full sync, you can also remove the "-c" (for checksum) to just do date/size comparisons.

Notes:

• The expression:
  --exclude='.@__thumb/'
  works in rsync but the "@" needs to be escaped for rclone
• I am using the two trailing asterisks and slash per the guidance on this page:
  http://rclone.org/filtering/
• The two trailing asterisks are because that specifies any directory depth beneath and any files (see filtering URL above).
• I would also recommend against having "-c" as your daily runner, maybe monthly for data integrity validation though (because of disk thrashing)... I used 8MBps because I have 75Mbps FiOS; adjust to your ISP and provisioned rate so as to not saturate your upstream connection

[felizf]

Cool video, using RTRR and a little bit of a kludge. I have tried something similar using "rsync" but I find that after a while it seems that rsync slows to a snail's crawl. That's likely because of the queued-up files. I only have two concerns: 1) after a while, I see a bunch of errors where files show as not having transferred, and most importantly, 2) no checksum comparison of the files (correct me if I'm wrong!). However, the rclone command supports MD5 checksums of the files in the cloud, which is what I want to do (versus a local "rsync -c" which first downloads the cloud files, creates the local file checksum, then compares; if it matches it ignores, otherwise it re-uploads the file).

I think I have this working well now... I have created symlinks to the appropriate local directories I want in my home directory and the rclone config file has a connection to Amazon Cloud Drive as named "ACD:". Here is what I am doing (first line is the location of the binaries - note that on x86_64 platforms, amd64 v1.26 currently DOES NOT WORK - it doesn't correctly authenticate with Amazon):

Create a symbolic link from /usr/bin to your rclone binary like this (first parameter after "-s" is actual location of rclone, second parameter is the symbolic link location):
ln -s /file/location/of/rclone /usr/bin/rclone

Next, using "vi", create a file called rclone-exclude.txt with the contents below:
$RECYCLE.BIN
.\@__thumb/**
\@Recycle/**
Thumbs.db

Now do this:
rclone sync ~/Files/backup ACD:/backup --bwlimit 8M --checkers=16 --contimeout=5m --transfers=8 --exclude-from=/root/rclone-exclude.txt --retries=100 --stats 10s --drive-chunk-size 1024k --delete-excluded -c --dry-run -v

Once it goes through this in verbose mode and you're satisfied that it is doing what it is supposed to do, remove the "-v". Once it plans to do what you think it's supposed to send (i.e., directories are right), get rid of the "--dry-run". Once you've done the first full sync, you can also remove the "-c" (for checksum) to just do date/size comparisons.

Notes:

• The expression:
  --exclude='.@__thumb/'
  works in rsync but the "@" needs to be escaped for rclone
• I am using the two trailing asterisks and slash per the guidance on this page:
  http://rclone.org/filtering/
• The two trailing asterisks are because that specifies any directory depth beneath and any files (see filtering URL above).
• I would also recommend against having "-c" as your daily runner, maybe monthly for data integrity validation though (because of disk thrashing)... I used 8MBps because I have 75Mbps FiOS; adjust to your ISP and provisioned rate so as to not saturate your upstream connection

hello,
how did you manage to get rclone to authorize against amazon? did you do this by creating an ssh tunnel from your local machine to the qnap? because that's what i did and i can get to the amazon authorization screen and it is says it is successful on the web page but in the shell i get this:

If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
2016/01/04 22:54:11 Failed to configure token: Failed to get token: Post https://api.amazon.com/auth/o2/token: x509: failed to load system roots and no roots provided
[~] #

[JustRob]

I think I have this working well now... I have created symlinks to the appropriate local directories I want in my home directory and the rclone config file has a connection to Amazon Cloud Drive as named "ACD:". Here is what I am doing (first line is the location of the binaries - note that on x86_64 platforms, amd64 v1.26 currently DOES NOT WORK - it doesn't correctly authenticate with Amazon)

how did you manage to get rclone to authorize against amazon? did you do this by creating an ssh tunnel from your local machine to the qnap? because that's what i did and i can get to the amazon authorization screen and it is says it is successful on the web page but in the shell i get this:

If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
2016/01/04 22:54:11 Failed to configure token: Failed to get token: Post https://api.amazon.com/auth/o2/token: x509: failed to load system roots and no roots provided
[~] #

You have to open up that webpage from the QNAP, I couldn't get this to work remotely because there is no VNC client or Xwindows mechanisms installed (to redirect using SSH). I welcome anyone else's suggestions. I installed Firefox and Chrome, and got this to work in HD Station by installing Chrome. This may work with Firefox. Note that you cannot get this to work by changing the IP address of the URL and accessing from an external IP address. I.e., these below WILL NOT WORK.

http://qnap.my.lan:53682/auth
or
http://192.168.1.150:53682/auth
(assuming your DNS resolves or IP address resolves to your QNAP device, respectively).

It seems that the authentication has to occur from localhost. Or at least the above methods from an IP address inside my LAN but external to QNAP did NOT work for me. I assume you did an SSH port redirect like this:
ssh -l user@qnap.my.lan -L 53682:localhost:53682
from the external machine and then tried to open the URL from an external machine on your LAN (assuming qnap.my.lan is your QNAP device). I did not try that, clever idea, but I got frustrated and just did it local to the machine.

[felizf]

I think I have this working well now... I have created symlinks to the appropriate local directories I want in my home directory and the rclone config file has a connection to Amazon Cloud Drive as named "ACD:". Here is what I am doing (first line is the location of the binaries - note that on x86_64 platforms, amd64 v1.26 currently DOES NOT WORK - it doesn't correctly authenticate with Amazon)

how did you manage to get rclone to authorize against amazon? did you do this by creating an ssh tunnel from your local machine to the qnap? because that's what i did and i can get to the amazon authorization screen and it is says it is successful on the web page but in the shell i get this:

If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
2016/01/04 22:54:11 Failed to configure token: Failed to get token: Post https://api.amazon.com/auth/o2/token: x509: failed to load system roots and no roots provided
[~] #

You have to open up that webpage from the QNAP, I couldn't get this to work remotely because there is no VNC client or Xwindows mechanisms installed (to redirect using SSH). I welcome anyone else's suggestions. I installed Firefox and Chrome, and got this to work in HD Station by installing Chrome. This may work with Firefox. Note that you cannot get this to work by changing the IP address of the URL and accessing from an external IP address. I.e., these below WILL NOT WORK.

http://qnap.my.lan:53682/auth
or
http://192.168.1.150:53682/auth
(assuming your DNS resolves or IP address resolves to your QNAP device, respectively).

It seems that the authentication has to occur from localhost. Or at least the above methods from an IP address inside my LAN but external to QNAP did NOT work for me. I assume you did an SSH port redirect like this:
ssh -l user@qnap.my.lan -L 53682:localhost:53682
from the external machine and then tried to open the URL from an external machine on your LAN (assuming qnap.my.lan is your QNAP device). I did not try that, clever idea, but I got frustrated and just did it local to the machine.

that's exactly what i did. i was too lazy to go through HD station lol. ok thanks!

[felizf]

I think I have this working well now... I have created symlinks to the appropriate local directories I want in my home directory and the rclone config file has a connection to Amazon Cloud Drive as named "ACD:". Here is what I am doing (first line is the location of the binaries - note that on x86_64 platforms, amd64 v1.26 currently DOES NOT WORK - it doesn't correctly authenticate with Amazon)

how did you manage to get rclone to authorize against amazon? did you do this by creating an ssh tunnel from your local machine to the qnap? because that's what i did and i can get to the amazon authorization screen and it is says it is successful on the web page but in the shell i get this:

If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
2016/01/04 22:54:11 Failed to configure token: Failed to get token: Post https://api.amazon.com/auth/o2/token: x509: failed to load system roots and no roots provided
[~] #

You have to open up that webpage from the QNAP, I couldn't get this to work remotely because there is no VNC client or Xwindows mechanisms installed (to redirect using SSH). I welcome anyone else's suggestions. I installed Firefox and Chrome, and got this to work in HD Station by installing Chrome. This may work with Firefox. Note that you cannot get this to work by changing the IP address of the URL and accessing from an external IP address. I.e., these below WILL NOT WORK.

http://qnap.my.lan:53682/auth
or
http://192.168.1.150:53682/auth
(assuming your DNS resolves or IP address resolves to your QNAP device, respectively).

It seems that the authentication has to occur from localhost. Or at least the above methods from an IP address inside my LAN but external to QNAP did NOT work for me. I assume you did an SSH port redirect like this:
ssh -l user@qnap.my.lan -L 53682:localhost:53682
from the external machine and then tried to open the URL from an external machine on your LAN (assuming qnap.my.lan is your QNAP device). I did not try that, clever idea, but I got frustrated and just did it local to the machine.

i am still getting the same error message even though i am going through the hybridesk. i have used both chrome and firefox but still get this:
f your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
2016/01/04 22:54:11 Failed to configure token: Failed to get token: Post https://api.amazon.com/auth/o2/token: x509: failed to load system roots and no roots provided
[~] #


i run rclone config from a ssh client on my mac and also from the console from qnap using a monitor and keyboard, both ways i still get error.

[JustRob]

I think I have this working well now... I have created symlinks to the appropriate local directories I want in my home directory and the rclone config file has a connection to Amazon Cloud Drive as named "ACD:". Here is what I am doing (first line is the location of the binaries - note that on x86_64 platforms, amd64 v1.26 currently DOES NOT WORK - it doesn't correctly authenticate with Amazon)

i am still getting the same error message even though i am going through the hybridesk. i have used both chrome and firefox but still get this:
f your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
2016/01/04 22:54:11 Failed to configure token: Failed to get token: Post https://api.amazon.com/auth/o2/token: x509: failed to load system roots and no roots provided
[~] #

i run rclone config from a ssh client on my mac and also from the console from qnap using a monitor and keyboard, both ways i still get error.

Caveat: I did NOT try this on an ARM-based QNAP device, I tried this on a TS-451+, which uses a quad-core Celeron (x86_64). However, the 64-bit version of rclone did not work for me and I believe had the identical error as yours. I had to use the 32-bit version. I suspect this must be some kind of library issue with the QNAP (Debian-based) install tree, despite the fact that I have 16GB (only officially 8GB supported) installed and running fine and it's using a 64-bit kernel. I apologize but I didn't see any information indicating what system you had or if it was ARM or x86-based. I cannot comment on ARM-based since I don't have an ARM-based device.

[RoyIvyIII]

You can configure `rclone` on a local machine and then move the configuration secrets to the QNAP (or the whole ".rclone.conf" file).

The error "... x509: failed to load system roots and no roots provided" is covered in the FAQ (http://rclone.org/faq). `rclone` requires access the viable security certificates. To fix the error, in an admin shell on the QNAP, do:

Code: Select all

mkdir -p /etc/ssl/certs/
curl -o /etc/ssl/certs/ca-certificates.crt https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt

Using the 32-bit linux `rclone` executable (from http://downloads.rclone.org/rclone-curr ... ux-386.zip), a copied config setup from my local machine, and the downloaded certificate store, I've got `rclone` fully working on my two x86-based QNAPs (TS-1679 and TS-879).

[unixvrules]

Looks as if HybridSync was released this week for 4.2. Along with CloudBackup Sync. No Amazon Cloud Drive there yet, though

https://www.qnap.com/i/en/tutorial/inde ... ontent_top
https://www.qnap.com/i/en/tutorial/con_ ... ne&cid=200

[john.bradley.clarke]

+1 to add this to HybridSync

[felizf]

amazon cloud drive was added to the hybrid backup sync app, testing as we speak.