[HOWTO] SpiderOak Online Backup

[datatrack]

Hello Chilly Willy, thanks for sharing this nice post. This is very useful because it supports various OS.

[friday]

Revised and simplified 2015 instructions for SpiderOakONE (yes, that is the official casing) v6
(No Debian or online form required)

Requirements:
X86 CPU (not ARM)
Qnapware: http://apps.qnap.community/11-community/10-qnapware (Update: Qnapware has now been replaced by Entware-ng: https://apps.qnap.community/11-community/10-entware-ng, which I haven't tested myself)

Some parts stolen from the original post by ChillyWilly. Thank you! I've only tested it on my QTS 4.2 TS-251. If you want to adapt it to work with an alternative to Qnapware you may have to change "Apps/opt" to something else. You also need to replace "/share/Download/" if you don't have one.

Follow these commented instructions:

Code: Select all

# Install prerequisites
opkg install python ar

# Create temporary directory for downloading + extracting and cd to it
mkdir /share/Download/spideroak-tmp && cd $_

# Download deb package (32 bit, not 64)
wget --no-check-certificate --output-document=spideroak.deb "https://spideroak.com/getbuild?platform=ubuntu&arch=i386"

# Extract from deb (ar and tar.gz)
ar x spideroak.deb data.tar.gz && tar -zxvf data.tar.gz

# create a conf dir
mkdir opt/SpiderOakONE/conf

# move to /Apps
mv usr/bin/SpiderOakONE /Apps/opt/bin/spideroak
mv opt/SpiderOakONE /Apps/opt/

# delete temp directory
rm -r /share/Download/spideroak-tmp

After that you need to edit the bash script. If you're not comfortable using vi you can "opkg install nano" and use it instead

Code: Select all

vi /Apps/opt/bin/spideroak

Change all references to "/opt" to "/Apps/opt" and add this code after the first line:

Code: Select all

if [ ! -d "/root/.config/SpiderOakONE" ]; then
    mkdir -p /root/.config/
    ln -s /Apps/opt/SpiderOakONE/conf /root/.config/SpiderOakONE
fi

Setup your account (this will prompt you for details and download prior deduplication data if available, which can take a while)

Code: Select all

spideroak --setup=-

Now help yourself using https://spideroak.com/faq/how-can-i-use ... mmand-line or

Code: Select all

spideroak --help

This is pretty much what you need to get started (-v for "verbose" isn't required, but helps):

Code: Select all

spideroak -v --include-dir=/path/to/directory
spideroak -v --batchmode

Add "spideroak --batchmode" to crontab to create a persistent job: http://wiki.qnap.com/wiki/Add_items_to_crontab

Also, you are likely to run into a problem because Spideroak uses /tmp, which is a small ramdisk. If you have enough ram, you can increase the size of the mount (recommended). For 512mb that would be:

Code: Select all

mount -o remount,size=512m /tmp

If you haven't got enough ram you might be able to do this, but it's risky: http://forum.qnap.com/viewtopic.php?t=44541

Either one needs to run after every reboot: http://wiki.qnap.com/wiki/Running_Your_ ... at_Startup

[friday]

friday, nice instructions. Though I run into a strange issue where when I try to copy the folders to /Apps/* it basically fills the RAMdisk and then everything goes to **. Verified before and after with df -h

How did you avoid this issue?

Thank you,

/Apps/ is ramdisk. Don't put anything here, as it will be wiped on every reboot. Use /Apps/opt/ as in the instructions.
/Apps/opt should be a symlink to /share/CACHEDEV1_DATA/.qpkg/Qnapware/opt (varies depending on your NAS model and possibly firmware version) set up by Qnapware. Using this is easier than setting up your own symlink, as you don't have to create another symlink and set up path (so you can run "spideroak" rather than "/path/to/spideroak" later on).

You can test this with "ls -lh /Apps". The row with "opt" should have an arrow pointing to the the symlinked directory.

Sorry for the late reply. I didn't get any notification.

[chiavelli]

UPDATED AND VERIFIED FOR JUNE 2017

My Setup:
TS-453a
Version 4.3.3.0210

All of this was cobbled together from various other posts in this forum, primarily:
ChillyWilly's initial SpiderOak instructions
- viewtopic.php?f=15&t=31037#p133614
friday's updated 2015 SpiderOak instructions
- viewtopic.php?f=15&t=31037&start=15#p520113
CasaP's instructions for setting up autorun instructions on newer QNAP models
- viewtopic.php?t=112970


1. Add QNAP Club Store to App Center
The QNAP Club store is needed to install Entware-ng, which is no longer available through the official QNAP App Center
Follow the instructions here: https://qnapclub.eu/index.php?act=howto
I recall there being some differences between my interface and the one pictured in that link, but it was close enough that it was easy to figure out.


2. Install Entware-ng
In App Center, look for Entware-ng under the list of QNAP Club apps and install it.
Entware-ng is a package management system which will allow you to install Python in the next step

3. Launch SSH Session
For the remaining steps you will need to be logged into your QNAP on an SSH session.
I used PuTTY.

4. Download and install SpiderOakONE

Code: Select all

# Install prerequisites
opkg install python ar

# Create temporary directory for downloading + extracting and cd to it
mkdir /share/Download/spideroak-tmp && cd $_

# Download deb package (32 bit, not 64)
wget --no-check-certificate --output-document=spideroak.deb "https://spideroak.com/getbuild?platform=ubuntu&arch=i386"

# Extract from deb (ar and tar.gz)
ar x spideroak.deb data.tar.gz && tar -zxvf data.tar.gz

# create a conf dir
mkdir opt/SpiderOakONE/conf

# move to /opt
mv usr/bin/SpiderOakONE /opt/bin/spideroak
mv opt/SpiderOakONE /opt/

# delete temp directory
rm -r /share/Download/spideroak-tmp

5. Edit SpiderOak bash file
A Spideroak symlink needs to be added to the system's /root/.config folder. Your QNAP will wipe this symlink out every time the system boots, so we are going to add the symlink creation to the SpiderOak bash file, ensuring it is always there when we need it.

Open the bash script for editing. If you are uncomfortable using vi, read tutorials on how to use vi.

Code: Select all

vi /opt/bin/spideroak

Add this code after the first line to create the symlink

Code: Select all

if [ ! -d "/root/.config/SpiderOakONE" ]; then
    mkdir -p /root/.config/
    ln -s /opt/SpiderOakONE/conf /root/.config/SpiderOakONE
fi

6. Setup your account
This will prompt you for account credentials. Once it gets going it will take awhile so be patient.

Code: Select all

spideroak --setup=-

Once complete, enter this command to look at how your account has been set up on your QNAP:

Code: Select all

spideroak --selection

7. Increase size of mount for /tmp directory
You are likely to run into a problem because Spideroak uses /tmp, which is a small ramdisk. If you have enough ram, you can increase the size of the mount (recommended). For 512mb that would be:

Code: Select all

mount -o remount,size=512m /tmp

You can verify the size of your mount like so:

Code: Select all

df -T /tmp

This mount will need to be done every time your system boots. I added the mount to my system's autorun.sh via the following steps

NOTE: These steps may not work for your QNAP model, especially if you have an older one. You may be able to find some direction here: https://wiki.qnap.com/wiki/Running_Your ... at_Startup

Open autorun file in vi

Code: Select all

mount $(/sbin/hal_app --get_boot_pd port_id=0)6 /tmp/config
touch /tmp/config/autorun.sh
chmod +x /tmp/config/autorun.sh
vi /tmp/config/autorun.sh

Add this line to the file and save/exit vi:

Code: Select all

mount -o remount,size=512m /tmp

Unmount the file:

Code: Select all

umount /tmp/config

8. Add some directories to backup!
This is pretty much what you need to get started (-v for "verbose" isn't required, but helps):

Code: Select all

spideroak -v --include-dir=/path/to/directory

Check what you have selected for backup:

Code: Select all

spideroak --selection

9. Back Stuff up!
Trigger a backup like so:

Code: Select all

spideroak -v --batchmode

If you do not want to always manually trigger your backups you should add "spideroak --batchmode" to crontab to create a persistent job: http://wiki.qnap.com/wiki/Add_items_to_crontab



And, uhh... that's it! Hopefully these updated steps can help save the rest of you some headaches.

[gilsn]

Hello!

I used the last post from chiavelli to setup spideroak on my TS-251+. Thanks a lot!!!

It works perfectly fine when I run it manually. But I can't make it work with cron.
I get the following error from spideroak :
"Command line arguments not allowed during New User Setup".

I read it may be a problem with the config file in the root directory. But the symlink looks ok, and again, when I run manually it works.

I don't get the difference with cron.

Any ideas?

Thank you in advance!

[friday]

It seems like you don't have the spideroak directory in home http://lukieb.blogspot.se/2013/03/comma ... lowed.html so I'm guessing either that directory doesn't exist at that time, or the command isn't run as the right user. Normally this would happen because you ran "sudo crontab -e" instead of just "crontab -e", but since busybox/qnap isn't set up like that I could only guess.

[gverweijen]

Great tutorial! Thanks for taking the time to set this up.
I've been following step by step, but I get stuck at step 6(setup your account). I suspect something went wrong before, though I've followed every step to the letter.
The error I get is this:

Code: Select all

[/share/Download/spideroak-tmp] # spideroak --setup=-
/opt/bin/spideroak: line 11: /opt/SpiderOakONE/lib/SpiderOakONE: No such file or directory
/opt/bin/spideroak: line 11: /opt/SpiderOakONE/lib/SpiderOakONE: Success

It is erroring on the last line of /opt/bin/spideroak which is:

Code: Select all

exec "/opt/SpiderOakONE/lib/SpiderOakONE" "$@"

It seems that everything is there:

Code: Select all

[~] # cd /opt/SpiderOakONE//lib/
[/opt/SpiderOakONE/lib] # ls
array.so		       _hashlib.so	      OpenSSL.crypto.so
bcrypt._bcrypt.so	       _heapq.so	      OpenSSL.rand.so
binascii.so		       _hotshot.so	      OpenSSL.SSL.so
_bisect.so		       i18n/		      operator.so
BTrees._fsBTree.so	       inotify_dir_watcher*   parser.so
BTrees._IFBTree.so	       _io.so		      persistent.cPersistence.so
BTrees._IIBTree.so	       itertools.so	      persistent.cPickleCache.so
BTrees._IOBTree.so	       _json.so		      persistent._timestamp.so
BTrees._LFBTree.so	       libcrypto.so.1.0.0     Pillow-2.8.1-py2.7-linux-i686.egg
BTrees._LLBTree.so	       libcurl.so.4	      plugins/
BTrees._LOBTree.so	       libdbusmenu-qt.so.2    pycurl.so
BTrees._OIBTree.so	       libexpat.so.0	      pyexpat.so
BTrees._OLBTree.so	       libffi.so.6	      PyQt4.QtCore.so
BTrees._OOBTree.so	       libgcc_s.so.1	      PyQt4.QtDeclarative.so
bz2.so			       libgmp.so.3	      PyQt4.QtGui.so
_cffi_backend.so	       libidn.so.11	      PyQt4.QtNetwork.so
_codecs_cn.so		       libpng12.so.0	      _random.so
_codecs_hk.so		       libQtCore.so.4	      resource.so
_codecs_iso2022.so	       libQtDBus.so.4	      select.so
_codecs_jp.so		       libQtDeclarative.so.4  setuptools-4.0.1-py2.7.egg/
_codecs_kr.so		       libQtGui.so.4	      simplejson._speedups.so
_codecs_tw.so		       libQtNetwork.so.4      sip.so
_collections.so		       libQtScript.so.4       _socket.so
cPickle.so		       libQtSql.so.4	      SpiderOakONE*
Crypto.Cipher.AES.so	       libQtSvg.so.4	      SpideroakVersionMatcher-1.1-py2.7-linux-i686.egg/
Crypto.Cipher.DES3.so	       libQtXmlPatterns.so.4  _sqlite3.so
Crypto.Cipher.XOR.so	       libQtXml.so.4	      _ssl.so
Crypto.Hash.SHA256.so	       library.zip	      strop.so
Crypto.PublicKey._fastmath.so  libsodium.so	      _struct.so
Crypto.Util._counter.so        libsqlite3.so.0	      termios.so
Crypto.Util.strxor.so	       libssh2.so.1	      time.so
cStringIO.so		       libssl.so.1.0.0	      twisted.internet._sigchld.so
_csv.so			       libstdc++.so.6	      twisted.protocols._c_urlarg.so
_ctypes.so		       libz.so.1	      twisted.python._initgroups.so
datetime.so		       _locale.so	      unicodedata.so
dl.so			       _lsprof.so	      zlib.so
fcntl.so		       math.so		      zope.interface._zope_interface_coptimizations.so
_functools.so		       mmap.so
grp.so			       _multibytecodec.so

anyone any ideas how to proceed?
I've got a QNAP TS-469 pro
Current firmware version: 4.3.4.0435

[friday]

Yeah, me too. It seems like a simple problem with paths not being set or being incorrect, but I verified it really don't think I screwed anything up. The spideroak error messages are less than helpful. Without getting more specific errors like "libsomething.so.0: No such file or directory" it's hopeless. After a qnap firmware upgrade that didn't recognize my old raid I had to set this up again, and between all the possible things that could have been changed (Entware-ng, Python version, Qnap firmware, possible issues with paths and spideroak changes) I wasn't able to figure it out.

One possibility would be to use Container Station with a Docker image for this, but I haven't got around to it. It's probably a lot easier to set up, but likely slower and less performant.

[gverweijen]

Indeed.. also spideroak release an update to v7.0 a few weeks ago.
Have you considered asking spideroak for some help?
Would you update us here if you find a solution?

[friday]

Indeed.. also spideroak release an update to v7.0 a few weeks ago.

I tried using several older versions of Spideroak, and this was in November. So I don't think the issue is with any recent Spideroak version.

Have you considered asking spideroak for some help?

No, asking for help goes against my beliefs
Also really doubt cloud storage providers with unlimited plans such as Spideroak want to support NAS. It drives their costs up.

Would you update us here if you find a solution?

Yes, but don't count on it. I'm very busy and haven't touched this since November. It's a shame. I was going to simplify the guide. Spideroak has a slackware release now, so installation is much easier (no entware, ar and temporary paths needed).

For future reference if it helps someone (this will ignore the shellscript which you can copy manually from one of guides and change the path to whereever you want $INSTALLATION_PATH to be):

Code: Select all

cd $INSTALLATION_PATH; wget -qO- "http://spideroak.com/getbuild?platform=slackware&arch=x64" | tar xz opt/SpiderOakONE/ --strip-components=1

[pharahiti]

I experienced the 'no such file' issue also, and managed to fix it by downloading the latest version of SpiderOakOne from this URL instead:

https://spideroak.com/opendownload/

I downloaded it to a linux PC, extracted the .deb file using dpkg -x, and then tarred and transferred the resulting files to my NAS. I then more or less followed the instructions posted by others to edit the launch script and add the lines to create the symlink and corrected the library and binary paths for my installation.

Hope this helps others..

[friday]

I don't think either of those are related to the problem. The different links just redirects to the same file, and ar should work perfectly to extract the deb (even if not, I also tried the slackware release). Getting the paths wrong certainly should lead to an error such as "No such file", but since /opt was mounted at /opt for me there wasn't anything to change.

It's good to know it's working for someone though.

[gverweijen]

Hi Guys,

Any news on this issue?
Today I retried all the steps using latest release SpiderOakONE 7.2.1
I still get stuck on the same exact error.

also tried renaming the /opt/bin/spideroak to /opt/bin/SpiderOakONE
also tried putting it in /usr/bin/SpiderOakONE

Still the same error.

[derekvincent]

I managed to get it going with the latest version of SpiderOakOne.

Code: Select all

## Need additional packages 
opkg install xz xz-utils

## Download the latest deb package from SpiderOak
wget --no-check-certificate --output-document=spideroak.deb "https://spideroak.com/release/spideroak/deb_x64"

## Extract from the deb (ar and tar.xz) NOTE: Compression inside has changed!
ar x spideroak.deb data.tar.xz && tar -zxvf data.tar.xz

Continue on as per chiavelli's instruction above.


Derek

[friday]

I gave up and now run it via Container Station. It's ridiculous how much easier this was:

1. Install Container Station (if you haven't already)
2. Open Container Station and select "Create"
3. Search for "Ubuntu"
4. Install the latest Ubuntu LXC (18.04 is the latest LTS version, but 16.04 is the latest in the list). This will get you a minimal Ubuntu image that runs faster than Virtualization Station or Linux Station and boots up in no time. It's similar to Docker but for running a system. You can start it and stop it, and it will continue where you left off.
5. Set up the container:

• You may want to limit it so it doesn't use too much of the hosts resources, but it needs 2GB RAM.
• Under advanced settings, add the shared folders you need it to backup. Each one you add must have a mount point on the container. One advice is you add them to a single root directly so you only have to sync one directory. So for example add them as "/my-nas/Documents" and "/my-nas/Downloads".
• Add a port forward to port "22" on your container. Use something unused by your host, like "2222".
• You can add shared folders and change the hardware setup later, so you don't have to get it right the first time.

6. It will now download the Ubuntu image and run it. It's just a 85mb file with the bare essentials, so it should only take a minute.
7. Go to overview. Click on the new container name. You are now in a view with some container info and a shell interface.
8. Your username and password are both "ubuntu". You should change your password to something else with "passwd"
9. Update your packages (the ones you get by default are very outdated by now), install ssh, screen and spideroak one:

Code: Select all

sudo sh -c 'apt update && apt upgrade && apt install -y openssh-server screen wget && wget https://spideroak.com/release/spideroak/deb_x64 && sudo dpkg -i deb_x64 && rm deb_x64'

The installer adds SpiderOaks Ubuntu repository, so next time just "sudo apt update" and "sudo apt upgrade"
10. SSH into the container from another computer: "ssh ubuntu@NASIP -p port". Substitute NASIP with the ip address (or domain name) you use for the web interface, and the port with the port you created in the port forward rule. Ex: "ssh ubuntu@192.168.0.6 -p 2222".
11. Run "screen" (and press enter or space).
12. Now run "SpiderOakONE --setup=-". If you already had devices on SpiderOak, you'll get a message "Synchronizing with server (this could take a while)...". Once it's done it will say "Batchmode run complete: shutting down". Since you are running via screen you can detach (Ctrl + A, Ctrl + D) or just close your terminal. To continue later: ssh into the container and run `screen -r`
13. After your initial sync, add the folder(s) to sync ex "SpiderOakONE -v --include-dir=/my-nas" and "SpiderOakONE -v --batchmode" to start syncing ("-v" is an optional flag for verbose).
14. To schedule a recurring sync, run "crontab -e" and add something like this "0 0 * * * /usr/bin/SpiderOakONE --batchmode". Batchmode doesn't start if it's already running, so this is safe if you've got a lot of data to sync initially. Also, if you need your jobs to run at specific times, don't forget to set the time zone first "sudo dpkg-reconfigure tzdata"

For more CLI usage, see the CLI-reference or run "SpiderOakONE --help".

If you're like me and find the name of the executable unpractical, you can create an alias or symlink: "sudo ln -s /usr/bin/SpiderOakONE /usr/bin/spideroak"

If your initial sync exits early you may get stuck with an error "SpiderOakONE has detected that it needs to re-sync with the server. Please run in normal (GUI) mode, headless, or batchmode to re-sync." Following the instructions didn't work for me, so if it's the same for you, at least you can delete your entire Spideroak One user directory `rm -rf ~/.config/SpiderOakONE` and start over.

You can do all this without openssh-server and screen, but the web terminal is not great for this. There's a better one if you press the fullscreen button next to "Console", but neither of them sends "ctrl" to the client.