[HOWTO] SpiderOak Online Backup

Backup, Restore, Netbak Replicator, Cloud Storage Services
datatrack
New here
Posts: 4
Joined: Fri Mar 20, 2015 6:31 pm

Re: [HOWTO] SpiderOak Online Backup

Post by datatrack » Fri Mar 20, 2015 7:10 pm

Hello Chilly Willy, thanks for sharing this nice post. This is very useful because it supports various OS.

friday
New here
Posts: 9
Joined: Wed Sep 16, 2015 11:01 pm

Re: [HOWTO] SpiderOak Online Backup

Post by friday » Wed Dec 02, 2015 5:39 am

Revised and simplified 2015 instructions for SpiderOakONE (yes, that is the official casing) v6
(No Debian or online form required)

Requirements:
X86 CPU (not ARM)
Qnapware: http://apps.qnap.community/11-community/10-qnapware (Update: Qnapware has now been replaced by Entware-ng: https://apps.qnap.community/11-community/10-entware-ng, which I haven't tested myself)

Some parts stolen from the original post by ChillyWilly. Thank you! I've only tested it on my QTS 4.2 TS-251. If you want to adapt it to work with an alternative to Qnapware you may have to change "Apps/opt" to something else. You also need to replace "/share/Download/" if you don't have one.

Follow these commented instructions:

Code: Select all

# Install prerequisites
opkg install python ar

# Create temporary directory for downloading + extracting and cd to it
mkdir /share/Download/spideroak-tmp && cd $_

# Download deb package (32 bit, not 64)
wget --no-check-certificate --output-document=spideroak.deb "https://spideroak.com/getbuild?platform=ubuntu&arch=i386"

# Extract from deb (ar and tar.gz)
ar x spideroak.deb data.tar.gz && tar -zxvf data.tar.gz

# create a conf dir
mkdir opt/SpiderOakONE/conf

# move to /Apps
mv usr/bin/SpiderOakONE /Apps/opt/bin/spideroak
mv opt/SpiderOakONE /Apps/opt/

# delete temp directory
rm -r /share/Download/spideroak-tmp


After that you need to edit the bash script. If you're not comfortable using vi you can "opkg install nano" and use it instead

Code: Select all

vi /Apps/opt/bin/spideroak


Change all references to "/opt" to "/Apps/opt" and add this code after the first line:

Code: Select all

if [ ! -d "/root/.config/SpiderOakONE" ]; then
    mkdir -p /root/.config/
    ln -s /Apps/opt/SpiderOakONE/conf /root/.config/SpiderOakONE
fi


Setup your account (this will prompt you for details and download prior deduplication data if available, which can take a while)

Code: Select all

spideroak --setup=-


Now help yourself using https://spideroak.com/faq/how-can-i-use ... mmand-line or

Code: Select all

spideroak --help


This is pretty much what you need to get started (-v for "verbose" isn't required, but helps):

Code: Select all

spideroak -v --include-dir=/path/to/directory
spideroak -v --batchmode


Add "spideroak --batchmode" to crontab to create a persistent job: http://wiki.qnap.com/wiki/Add_items_to_crontab

Also, you are likely to run into a problem because Spideroak uses /tmp, which is a small ramdisk. If you have enough ram, you can increase the size of the mount (recommended). For 512mb that would be:

Code: Select all

mount -o remount,size=512m /tmp

If you haven't got enough ram you might be able to do this, but it's risky: http://forum.qnap.com/viewtopic.php?t=44541

Either one needs to run after every reboot: http://wiki.qnap.com/wiki/Running_Your_ ... at_Startup
Last edited by friday on Mon Nov 14, 2016 6:36 am, edited 1 time in total.

friday
New here
Posts: 9
Joined: Wed Sep 16, 2015 11:01 pm

Re: [HOWTO] SpiderOak Online Backup

Post by friday » Sun Jun 19, 2016 11:56 pm

unstatusthequo wrote:friday, nice instructions. Though I run into a strange issue where when I try to copy the folders to /Apps/* it basically fills the RAMdisk and then everything goes to **. Verified before and after with df -h

How did you avoid this issue?


Thank you,

/Apps/ is ramdisk. Don't put anything here, as it will be wiped on every reboot. Use /Apps/opt/ as in the instructions.
/Apps/opt should be a symlink to /share/CACHEDEV1_DATA/.qpkg/Qnapware/opt (varies depending on your NAS model and possibly firmware version) set up by Qnapware. Using this is easier than setting up your own symlink, as you don't have to create another symlink and set up path (so you can run "spideroak" rather than "/path/to/spideroak" later on).

You can test this with "ls -lh /Apps". The row with "opt" should have an arrow pointing to the the symlinked directory.

Sorry for the late reply. I didn't get any notification.

chiavelli
New here
Posts: 2
Joined: Mon Jun 26, 2017 3:30 am

Re: [HOWTO] SpiderOak Online Backup

Post by chiavelli » Mon Jun 26, 2017 5:17 am

UPDATED AND VERIFIED FOR JUNE 2017

My Setup:
TS-453a
Version 4.3.3.0210

All of this was cobbled together from various other posts in this forum, primarily:
ChillyWilly's initial SpiderOak instructions
- viewtopic.php?f=15&t=31037#p133614
friday's updated 2015 SpiderOak instructions
- viewtopic.php?f=15&t=31037&start=15#p520113
CasaP's instructions for setting up autorun instructions on newer QNAP models
- viewtopic.php?t=112970


1. Add QNAP Club Store to App Center
The QNAP Club store is needed to install Entware-ng, which is no longer available through the official QNAP App Center
Follow the instructions here: https://qnapclub.eu/index.php?act=howto
I recall there being some differences between my interface and the one pictured in that link, but it was close enough that it was easy to figure out.


2. Install Entware-ng
In App Center, look for Entware-ng under the list of QNAP Club apps and install it.
Entware-ng is a package management system which will allow you to install Python in the next step

3. Launch SSH Session
For the remaining steps you will need to be logged into your QNAP on an SSH session.
I used PuTTY.

4. Download and install SpiderOakONE

Code: Select all

# Install prerequisites
opkg install python ar

# Create temporary directory for downloading + extracting and cd to it
mkdir /share/Download/spideroak-tmp && cd $_

# Download deb package (32 bit, not 64)
wget --no-check-certificate --output-document=spideroak.deb "https://spideroak.com/getbuild?platform=ubuntu&arch=i386"

# Extract from deb (ar and tar.gz)
ar x spideroak.deb data.tar.gz && tar -zxvf data.tar.gz

# create a conf dir
mkdir opt/SpiderOakONE/conf

# move to /opt
mv usr/bin/SpiderOakONE /opt/bin/spideroak
mv opt/SpiderOakONE /opt/

# delete temp directory
rm -r /share/Download/spideroak-tmp



5. Edit SpiderOak bash file
A Spideroak symlink needs to be added to the system's /root/.config folder. Your QNAP will wipe this symlink out every time the system boots, so we are going to add the symlink creation to the SpiderOak bash file, ensuring it is always there when we need it.

Open the bash script for editing. If you are uncomfortable using vi, read tutorials on how to use vi.

Code: Select all

vi /opt/bin/spideroak


Add this code after the first line to create the symlink

Code: Select all

if [ ! -d "/root/.config/SpiderOakONE" ]; then
    mkdir -p /root/.config/
    ln -s /opt/SpiderOakONE/conf /root/.config/SpiderOakONE
fi


6. Setup your account
This will prompt you for account credentials. Once it gets going it will take awhile so be patient.

Code: Select all

spideroak --setup=-

Once complete, enter this command to look at how your account has been set up on your QNAP:

Code: Select all

spideroak --selection


7. Increase size of mount for /tmp directory
You are likely to run into a problem because Spideroak uses /tmp, which is a small ramdisk. If you have enough ram, you can increase the size of the mount (recommended). For 512mb that would be:

Code: Select all

mount -o remount,size=512m /tmp


You can verify the size of your mount like so:

Code: Select all

df -T /tmp


This mount will need to be done every time your system boots. I added the mount to my system's autorun.sh via the following steps

NOTE: These steps may not work for your QNAP model, especially if you have an older one. You may be able to find some direction here: https://wiki.qnap.com/wiki/Running_Your ... at_Startup

Open autorun file in vi

Code: Select all

mount $(/sbin/hal_app --get_boot_pd port_id=0)6 /tmp/config
touch /tmp/config/autorun.sh
chmod +x /tmp/config/autorun.sh
vi /tmp/config/autorun.sh

Add this line to the file and save/exit vi:

Code: Select all

mount -o remount,size=512m /tmp

Unmount the file:

Code: Select all

umount /tmp/config



8. Add some directories to backup!
This is pretty much what you need to get started (-v for "verbose" isn't required, but helps):

Code: Select all

spideroak -v --include-dir=/path/to/directory

Check what you have selected for backup:

Code: Select all

spideroak --selection



9. Back Stuff up!
Trigger a backup like so:

Code: Select all

spideroak -v --batchmode


If you do not want to always manually trigger your backups you should add "spideroak --batchmode" to crontab to create a persistent job: http://wiki.qnap.com/wiki/Add_items_to_crontab



And, uhh... that's it! Hopefully these updated steps can help save the rest of you some headaches.

gilsn
First post
Posts: 1
Joined: Tue Sep 19, 2017 4:54 am

Re: [HOWTO] SpiderOak Online Backup

Post by gilsn » Tue Sep 19, 2017 5:05 am

Hello!

I used the last post from chiavelli to setup spideroak on my TS-251+. Thanks a lot!!!

It works perfectly fine when I run it manually. But I can't make it work with cron.
I get the following error from spideroak :
"Command line arguments not allowed during New User Setup".

I read it may be a problem with the config file in the root directory. But the symlink looks ok, and again, when I run manually it works.

I don't get the difference with cron.

Any ideas?

Thank you in advance!

friday
New here
Posts: 9
Joined: Wed Sep 16, 2015 11:01 pm

Re: [HOWTO] SpiderOak Online Backup

Post by friday » Tue Sep 19, 2017 10:37 am

It seems like you don't have the spideroak directory in home http://lukieb.blogspot.se/2013/03/comma ... lowed.html so I'm guessing either that directory doesn't exist at that time, or the command isn't run as the right user. Normally this would happen because you ran "sudo crontab -e" instead of just "crontab -e", but since busybox/qnap isn't set up like that I could only guess.

gverweijen
New here
Posts: 4
Joined: Mon May 25, 2015 4:39 am

Re: [HOWTO] SpiderOak Online Backup

Post by gverweijen » Sun Jan 21, 2018 2:23 am

Great tutorial! Thanks for taking the time to set this up.
I've been following step by step, but I get stuck at step 6(setup your account). I suspect something went wrong before, though I've followed every step to the letter.
The error I get is this:

Code: Select all

[/share/Download/spideroak-tmp] # spideroak --setup=-
/opt/bin/spideroak: line 11: /opt/SpiderOakONE/lib/SpiderOakONE: No such file or directory
/opt/bin/spideroak: line 11: /opt/SpiderOakONE/lib/SpiderOakONE: Success

It is erroring on the last line of /opt/bin/spideroak which is:

Code: Select all

exec "/opt/SpiderOakONE/lib/SpiderOakONE" "$@"


It seems that everything is there:

Code: Select all

[~] # cd /opt/SpiderOakONE//lib/
[/opt/SpiderOakONE/lib] # ls
array.so             _hashlib.so         OpenSSL.crypto.so
bcrypt._bcrypt.so          _heapq.so         OpenSSL.rand.so
binascii.so             _hotshot.so         OpenSSL.SSL.so
_bisect.so             i18n/            operator.so
BTrees._fsBTree.so          inotify_dir_watcher*   parser.so
BTrees._IFBTree.so          _io.so            persistent.cPersistence.so
BTrees._IIBTree.so          itertools.so         persistent.cPickleCache.so
BTrees._IOBTree.so          _json.so            persistent._timestamp.so
BTrees._LFBTree.so          libcrypto.so.1.0.0     Pillow-2.8.1-py2.7-linux-i686.egg
BTrees._LLBTree.so          libcurl.so.4         plugins/
BTrees._LOBTree.so          libdbusmenu-qt.so.2    pycurl.so
BTrees._OIBTree.so          libexpat.so.0         pyexpat.so
BTrees._OLBTree.so          libffi.so.6         PyQt4.QtCore.so
BTrees._OOBTree.so          libgcc_s.so.1         PyQt4.QtDeclarative.so
bz2.so                libgmp.so.3         PyQt4.QtGui.so
_cffi_backend.so          libidn.so.11         PyQt4.QtNetwork.so
_codecs_cn.so             libpng12.so.0         _random.so
_codecs_hk.so             libQtCore.so.4         resource.so
_codecs_iso2022.so          libQtDBus.so.4         select.so
_codecs_jp.so             libQtDeclarative.so.4  setuptools-4.0.1-py2.7.egg/
_codecs_kr.so             libQtGui.so.4         simplejson._speedups.so
_codecs_tw.so             libQtNetwork.so.4      sip.so
_collections.so             libQtScript.so.4       _socket.so
cPickle.so             libQtSql.so.4         SpiderOakONE*
Crypto.Cipher.AES.so          libQtSvg.so.4         SpideroakVersionMatcher-1.1-py2.7-linux-i686.egg/
Crypto.Cipher.DES3.so          libQtXmlPatterns.so.4  _sqlite3.so
Crypto.Cipher.XOR.so          libQtXml.so.4         _ssl.so
Crypto.Hash.SHA256.so          library.zip         strop.so
Crypto.PublicKey._fastmath.so  libsodium.so         _struct.so
Crypto.Util._counter.so        libsqlite3.so.0         termios.so
Crypto.Util.strxor.so          libssh2.so.1         time.so
cStringIO.so             libssl.so.1.0.0         twisted.internet._sigchld.so
_csv.so                libstdc++.so.6         twisted.protocols._c_urlarg.so
_ctypes.so             libz.so.1         twisted.python._initgroups.so
datetime.so             _locale.so         unicodedata.so
dl.so                _lsprof.so         zlib.so
fcntl.so             math.so            zope.interface._zope_interface_coptimizations.so
_functools.so             mmap.so
grp.so                _multibytecodec.so


anyone any ideas how to proceed?
I've got a QNAP TS-469 pro
Current firmware version: 4.3.4.0435

friday
New here
Posts: 9
Joined: Wed Sep 16, 2015 11:01 pm

Re: [HOWTO] SpiderOak Online Backup

Post by friday » Sun Jan 21, 2018 3:14 am

Yeah, me too. It seems like a simple problem with paths not being set or being incorrect, but I verified it really don't think I screwed anything up. The spideroak error messages are less than helpful. Without getting more specific errors like "libsomething.so.0: No such file or directory" it's hopeless. After a qnap firmware upgrade that didn't recognize my old raid I had to set this up again, and between all the possible things that could have been changed (Entware-ng, Python version, Qnap firmware, possible issues with paths and spideroak changes) I wasn't able to figure it out.

One possibility would be to use Container Station with a Docker image for this, but I haven't got around to it. It's probably a lot easier to set up, but likely slower and less performant.

gverweijen
New here
Posts: 4
Joined: Mon May 25, 2015 4:39 am

Re: [HOWTO] SpiderOak Online Backup

Post by gverweijen » Sun Jan 21, 2018 3:38 am

Indeed.. also spideroak release an update to v7.0 a few weeks ago.
Have you considered asking spideroak for some help?
Would you update us here if you find a solution?

friday
New here
Posts: 9
Joined: Wed Sep 16, 2015 11:01 pm

Re: [HOWTO] SpiderOak Online Backup

Post by friday » Sun Jan 21, 2018 4:05 am

Indeed.. also spideroak release an update to v7.0 a few weeks ago.

I tried using several older versions of Spideroak, and this was in November. So I don't think the issue is with any recent Spideroak version.

Have you considered asking spideroak for some help?

No, asking for help goes against my beliefs ;)
Also really doubt cloud storage providers with unlimited plans such as Spideroak want to support NAS. It drives their costs up.

Would you update us here if you find a solution?

Yes, but don't count on it. I'm very busy and haven't touched this since November. It's a shame. I was going to simplify the guide. Spideroak has a slackware release now, so installation is much easier (no entware, ar and temporary paths needed).

For future reference if it helps someone (this will ignore the shellscript which you can copy manually from one of guides and change the path to whereever you want $INSTALLATION_PATH to be):

Code: Select all

cd $INSTALLATION_PATH; wget -qO- "http://spideroak.com/getbuild?platform=slackware&arch=x64" | tar xz opt/SpiderOakONE/ --strip-components=1

pharahiti
New here
Posts: 2
Joined: Thu Sep 20, 2012 11:51 pm

Re: [HOWTO] SpiderOak Online Backup

Post by pharahiti » Sun Mar 25, 2018 2:54 am

I experienced the 'no such file' issue also, and managed to fix it by downloading the latest version of SpiderOakOne from this URL instead:

https://spideroak.com/opendownload/

I downloaded it to a linux PC, extracted the .deb file using dpkg -x, and then tarred and transferred the resulting files to my NAS. I then more or less followed the instructions posted by others to edit the launch script and add the lines to create the symlink and corrected the library and binary paths for my installation.

Hope this helps others..

friday
New here
Posts: 9
Joined: Wed Sep 16, 2015 11:01 pm

Re: [HOWTO] SpiderOak Online Backup

Post by friday » Sun Mar 25, 2018 9:52 pm

I don't think either of those are related to the problem. The different links just redirects to the same file, and ar should work perfectly to extract the deb (even if not, I also tried the slackware release). Getting the paths wrong certainly should lead to an error such as "No such file", but since /opt was mounted at /opt for me there wasn't anything to change.

It's good to know it's working for someone though.

gverweijen
New here
Posts: 4
Joined: Mon May 25, 2015 4:39 am

Re: [HOWTO] SpiderOak Online Backup

Post by gverweijen » Thu Aug 16, 2018 1:07 am

Hi Guys,

Any news on this issue?
Today I retried all the steps using latest release SpiderOakONE 7.2.1
I still get stuck on the same exact error.

also tried renaming the /opt/bin/spideroak to /opt/bin/SpiderOakONE
also tried putting it in /usr/bin/SpiderOakONE

Still the same error.

derekvincent
New here
Posts: 8
Joined: Mon Mar 21, 2016 2:02 am

Re: [HOWTO] SpiderOak Online Backup

Post by derekvincent » Fri Sep 28, 2018 8:26 pm

I managed to get it going with the latest version of SpiderOakOne.

Code: Select all

## Need additional packages 
opkg install xz xz-utils

## Download the latest deb package from SpiderOak
wget --no-check-certificate --output-document=spideroak.deb "https://spideroak.com/release/spideroak/deb_x64"

## Extract from the deb (ar and tar.xz) NOTE: Compression inside has changed!
ar x spideroak.deb data.tar.xz && tar -zxvf data.tar.xz


Continue on as per chiavelli's instruction above.


Derek

friday
New here
Posts: 9
Joined: Wed Sep 16, 2015 11:01 pm

Re: [HOWTO] SpiderOak Online Backup

Post by friday » Wed Nov 28, 2018 11:06 am

I gave up and now run it via Container Station. It's ridiculous how much easier this was:

1. Install Container Station (if you haven't already)
2. Open Container Station and select "Create"
3. Search for "Ubuntu"
4. Install the latest Ubuntu LXC (18.04 is the latest LTS version, but 16.04 is the latest in the list). This will get you a minimal Ubuntu image that runs faster than Virtualization Station or Linux Station and boots up in no time. It's similar to Docker but for running a system. You can start it and stop it, and it will continue where you left off.
5. Set up the container:
  • You may want to limit it so it doesn't use too much of the hosts resources, but it needs 2GB RAM.
  • Under advanced settings, add the shared folders you need it to backup. Each one you add must have a mount point on the container. One advice is you add them to a single root directly so you only have to sync one directory. So for example add them as "/my-nas/Documents" and "/my-nas/Downloads".
  • Add a port forward to port "22" on your container. Use something unused by your host, like "2222".
  • You can add shared folders and change the hardware setup later, so you don't have to get it right the first time.
6. It will now download the Ubuntu image and run it. It's just a 85mb file with the bare essentials, so it should only take a minute.
7. Go to overview. Click on the new container name. You are now in a view with some container info and a shell interface.
8. Your username and password are both "ubuntu". You should change your password to something else with "passwd"
9. Update your packages (the ones you get by default are very outdated by now), install ssh, screen and spideroak one:

Code: Select all


sudo sh -c 'apt update && apt upgrade && apt install -y openssh-server screen wget && wget https://spideroak.com/release/spideroak/deb_x64 && sudo dpkg -i deb_x64 && rm deb_x64'
The installer adds SpiderOaks Ubuntu repository, so next time just "sudo apt update" and "sudo apt upgrade"
10. SSH into the container from another computer: "ssh ubuntu@NASIP -p port". Substitute NASIP with the ip address (or domain name) you use for the web interface, and the port with the port you created in the port forward rule. Ex: "ssh ubuntu@192.168.0.6 -p 2222".
11. Run "screen" (and press enter or space).
12. Now run "SpiderOakONE --setup=-". If you already had devices on SpiderOak, you'll get a message "Synchronizing with server (this could take a while)...". Once it's done it will say "Batchmode run complete: shutting down". Since you are running via screen you can detach (Ctrl + A, Ctrl + D) or just close your terminal. To continue later: ssh into the container and run `screen -r`
13. After your initial sync, add the folder(s) to sync ex "SpiderOakONE -v --include-dir=/my-nas" and "SpiderOakONE -v --batchmode" to start syncing ("-v" is an optional flag for verbose).
14. To schedule a recurring sync, run "crontab -e" and add something like this "0 0 * * * /usr/bin/SpiderOakONE --batchmode". Batchmode doesn't start if it's already running, so this is safe if you've got a lot of data to sync initially. Also, if you need your jobs to run at specific times, don't forget to set the time zone first "sudo dpkg-reconfigure tzdata"

For more CLI usage, see the CLI-reference or run "SpiderOakONE --help".

If you're like me and find the name of the executable unpractical, you can create an alias or symlink: "sudo ln -s /usr/bin/SpiderOakONE /usr/bin/spideroak"

If your initial sync exits early you may get stuck with an error "SpiderOakONE has detected that it needs to re-sync with the server. Please run in normal (GUI) mode, headless, or batchmode to re-sync." Following the instructions didn't work for me, so if it's the same for you, at least you can delete your entire Spideroak One user directory `rm -rf ~/.config/SpiderOakONE` and start over.

You can do all this without openssh-server and screen, but the web terminal is not great for this. There's a better one if you press the fullscreen button next to "Console", but neither of them sends "ctrl" to the client.

Post Reply

Return to “Backup & Restore”