[SCRIPT] sherpa.sh

[OneCD]

Description

A package manager to install various Usenet media-management apps into QNAP NAS.

sherpa is able to install several Usenet-related search and download apps but will only install ONE app at a time. If you elect to install an app that is already installed, the app will be fully reinstalled. This means the app configuration and settings are saved, the old app is uninstalled, a new version is installed, then the original settings are restored.

To install additional apps, run it again.

If the installer script is successful, your requested package and any required packages will be installed. Any existing installation of Entware will be used automatically. If Entware is not installed, a version appropriate to your NAS will be installed.

• Entware (standard)
• Git v2.22.0
• Python v3.8.3 (or v3.7.4 if required)
• SABYenc v4.0.3
• Par2 or Par2cmdline-MT, except on ARMv5. This will use the single-thread par2 instead.
• Cryptography
• OpenSSL
• and one or more of the following apps installed from source (GitHub or GitLab):
  • SABnzbd (develop)
  • Watcher3
  • LazyLibrarian
  • Medusa

• This one is installed via Entware and is a compiled binary:
  • NZBGet

My only request is that community response on this thread should only be about this installer. If your apps were not installed using this method, please create a new thread in the appropriate sub-forum.


Current status

• STABLE

Usage

1. SSH / PuTTY into your NAS as the 'admin' user,
2. Change to the Public share directory:

   Code: Select all

   cd /share/Public

3. Download the installer script and make it executable:

   Code: Select all

   curl -skLO https://git.io/sherpa.sh && chmod +x sherpa.sh

4. Then, to install (or reinstall) an app, run sherpa.sh with the name of your required app as an argument.
   
   So, to install SABnzbd, use:

   Code: Select all

   ./sherpa.sh SABnzbd

   Or:

   Code: Select all

   ./sherpa.sh OWatcher3
   ./sherpa.sh LazyLibrarian
   ./sherpa.sh Medusa
   ./sherpa.sh NZBGet

5. When you're done, delete the installer and debug log:

   Code: Select all

   rm sherpa.*

Known issues

• Python 2.7.16 is no-longer available via Entware/OpenWRT so the SABnzbdplus, SickChill and Headphones QPKGs can no-longer be installed. I'll need to find another Python2 source with installable modules, but I'm not hopeful.
  
  If you need SickChill, the devs apparently consider QSickChill to be their "official" version.

• Sometimes, it seems existing installations of Entware can become "difficult" to work with. So, Entware can also be reinstalled, but this should only be used as a last resort. Using:

  Code: Select all

  ./sherpa.sh Entware

  ... will force sherpa to uninstall your existing Entware QPKG, then install a new one. Note: Entware will be reverted back to default, and only the IPKGs required to support your installed sherpa apps will be installed.

• sherpa is incompatible with Optware-NG as it's missing a few required packages.

Problems?

• This will happen from time-to-time as the environment changes. If it's not shown above in 'Known issues' then you may have found something new, so please add to this thread with the details of the problem you encountered. Diagnose where you can and provide a solution if you're able. The functions in this script are a community effort.
  
  Sometimes the debug log will be required. This is always created. You can view this with:

  Code: Select all

  cat sherpa.debug.log

  Or run the installer in debug-mode to see it realtime. e.g.:

  Code: Select all

  ./sherpa.sh SABnzbd --debug

  There's now an option to check that all application dependencies have been satisfied. This will install any additional QPKGs or IPKGs to support any sherpa-installed applications present on your NAS:

  Code: Select all

  ./sherpa.sh --check

Firmware compatibility

• QTS 4.4.x - OK
• QTS 4.3.x - OK
• QTS 4.2.x - OK
• QTS 4.1.x - Unknown

Notes

• Supports application configuration-only backup and restore via the 'backup' and 'restore' arguments. This can be scripted via cron to create a regular backup of each app.
  
  QPKG (configuration-only) backups will be stored in a new hidden directory located under your default userdata volume called [.qpkg_config_backup/]. Hopefully, QNAP won't mess with this location (I'm looking at you Malware Remover).
  
  To jump to this path:

  Code: Select all

  cd $(getcfg SHARE_DEF defVolMP -f /etc/config/def_share.info)/.qpkg_config_backup

  Each QPKG has a single [config.tar.gz] file to backup into. Each new backup replaces the old one (so, no versioning available).
  
  Example: to backup SABnzbd:

  Code: Select all

  /etc/init.d/sabnzbd3.sh backup

  Example: to restore SABnzbd:

  Code: Select all

  /etc/init.d/sabnzbd3.sh restore

  This will 'stop' the QPKG, restore from the backup file (if it exists), then 'start' the QPKG again.

• The information contained therein was constructed from the efforts of many community members, both here and on the SABnzbd forum. Thank you to everyone who has contributed.

• Each of these packages continues the idea of 'self-update-on-launch' that was used in Clinton Hall's wrapper scripts. These scripts are my own version and require a few packages to be installed via Entware (this is what sherpa does). Updating an app is easy - just restart the app via its init script, or in the QTS App Center - "Stop" then "Start" will force an update.

• Medusa will appear as 'OMedusa' in your App Center to avoid conflict with the existing Medusa package available in the Qnapclub Store.

• Watcher3 will appear as 'OWatcher3' in your App Center to avoid conflict with the existing Watcher3 package available in the Qnapclub Store.

• Watcher3 is currently incompatible with Python 3.8.x so, if you install Watcher3, your Python 3.8.? will be downgraded to Python 3.7.4. This will affect all apps that use Python 3. When the devs fix this problem, the downgrade will no longer be necessary.

• The source for this project can be found on GitHub.

[dirkonline]

I will keep my eyes open and read along for now. This is so exciting. Thanks for the great work OneCD!

[Xmantium]

Are you able to put this on GitHub so others can input into it?

Any chance Sonarr can be added? Its way better than Sickrage, try and see for yourself

nzb-multi installer is a confusing name, think its best to put "Sabnzbd+ and Addons Installer" would be a better fit

[OneCD]

Are you able to put this on GitHub so others can input into it?

Already there: https://github.com/OneCDOnly/sherpa

Any chance Sonarr can be added? Its way better than Sickrage, try and see for yourself

Ah, personally not a fan of Mono - or anything that runs .NET code. So, there's not much chance of that happening.

nzb-multi installer is a confusing name, think its best to put "Sabnzbd+ and Addons Installer" would be a better fit

I agree it's confusing - but I'm hoping to add NZBGet in there at some point - so can't call it 'SABnzbd' anything.

edit: Decided to call it 'sherpa'. Because it ventures first and guides other apps through the dangerous wilderness. Y'know, something terribly meaningful like that.

[The Mask]

Hi OneCD,

I ran your sherpa script. It updated Sabnzbd and installed SR and CP. I encountered one issue.

Issue: I ran the command wget https://goo.gl/Qot9QK -qO sherpa.tar.gz && tar -zxvf sherpa.tar.gz but got an error

wget: not an http or ftp url: https://goo.gl/Qot9QK

I solved this by downloading it manually.

For the rest it ran smoothly, no problems at all.

Thanks for this script and the Qpkg packages.

[OneCD]

Hi and thanks for the feedback.

Issue: I ran the command wget https://goo.gl/Qot9QK -qO sherpa.tar.gz && tar -zxvf sherpa.tar.gz but got an error

wget: not an http or ftp url: https://goo.gl/Qot9QK

If you're able, can you try these two and let me know if they work for you?

Code: Select all

/usr/bin/wget https://goo.gl/Qot9QK -O sherpa.tar.gz

Code: Select all

wget http://goo.gl/Qot9QK -O sherpa.tar.gz

[scorche]

Thanks for your work on greatly simplifying the headache with keeping this stuff maintained!

I recently upgraded my QNAP and thought I'd try your script to get my SABNZBd updated to 2.x. However, the script fails at the pip modules. Below is the debug log:

Code: Select all

# ./SABnzbdplus --debug
[ dbug ] (>>) <Init>
[ dbug ] (II) ====================================================================
[ dbug ] (**) SCRIPT:          started: Sun May 7 12:04:22 MST 2017
[ dbug ] (**) SCRIPT:             file: sherpa.sh
[ dbug ] (**) SCRIPT:          version: 2017.05.07b
[ dbug ] (**) SCRIPT:      launched as: ./SABnzbdplus
[ dbug ] (**) SCRIPT:       target app: SABnzbdplus
[ dbug ] (II) --------------------------------------------------------------------
[ dbug ] (II) Markers: (**) detected, (II) information, (WW) warning, (EE) error,
[ dbug ] (II)          (--) done, (>>) function entry, (<<) function exit,
[ dbug ] (II)          (vv) variable name & value, ($1) positional argument value.
[ dbug ] (II) --------------------------------------------------------------------
[ dbug ] (**) NAS:               model: TS-869 Pro
[ dbug ] (**) NAS:    firmware version: 4.3.3
[ dbug ] (**) NAS:      firmware build: 20170413
[ dbug ] (**) NAS:              kernel: 3.4.6 x86_64
[ dbug ] (**) NAS:           OS uptime: 30 min
[ dbug ] (**) NAS:         system load: 1 min=0.12, 5 min=0.28, 15 min=0.39
[ dbug ] (**) NAS:      default volume: /share/CACHEDEV1_DATA
[ dbug ] (**) NAS:               $PATH: /bin:/sbin:/usr/bin:/usr/sbin:/usr/bin/X11
[ dbug ] (**) NAS:                /opt: /share/CACHEDEV1_DATA/.qpkg/Entware-ng
[ dbug ] (**) NAS:     /share/Download: CACHEDEV1_DATA/Download
[ dbug ] (II) --------------------------------------------------------------------
[ dbug ] (**) QPKG:     'QSabNZBdPlus': not installed
[ dbug ] (**) QPKG:       'Entware-ng': installed
[ dbug ] (**) QPKG:       'Entware-3x': not installed
[ dbug ] (II) found a suitable ARCH for Clinton's QPKG (x64)
[ dbug ] (II) found a suitable ARCH for Stephane's QPKG (x64)
[ dbug ] (**) QPKG:       'Entware-ng': installed
[ dbug ] (II) found a suitable Entware package (Entware-ng)
[ dbug ] (<<) <Init> [0]
[ dbug ] (>>) <PauseSabs>
[ dbug ] (**) QPKG:      'SABnzbdplus': installed
[ dbug ] (--) paused existing SABnzbd queue
[ dbug ] (<<) <PauseSabs> [0]
[ dbug ] (>>) <DownloadQPKGs>
[ dbug ] (**) QPKG:       'Entware-ng': installed
[ dbug ] (**) QPKG:   'Par2cmdline-MT': installed
[ dbug ] (>>) <DownloadQPKG>
[ dbug ] (II) existing QPKG checksum correct (SABnzbdplus_170131.qpkg)
[ dbug ] (<<) <DownloadQPKG> [0]
[ dbug ] (<<) <DownloadQPKGs> [0]
[ dbug ] (>>) <RemovePackageInstallers>
[ dbug ] (**) QPKG:          'Optware': not installed [250]
[ dbug ] (<<) <RemovePackageInstallers> [0]
[ dbug ] (>>) <InstallEntware>
[ dbug ] (**) QPKG:       'Entware-ng': installed
[ dbug ] (II) patch: do the "opt shuffle" - already done
[ dbug ] (<<) <InstallEntware> [0]
[ dbug ] (>>) <InstallOther>
[ dbug ] (**) QPKG:   'Par2cmdline-MT': installed
[ dbug ] (>>) <InstallFakeQPKGs>
[ dbug ] (**) QPKG:           'Python': installed
[ dbug ] (**) QPKG:              'git': installed
[ dbug ] (vv) fakeQPKG_python [false]
[ dbug ] (vv) fakebin_python [false]
[ dbug ] (vv) fakeQPKG_git [false]
[ dbug ] (vv) fakebin_git [false]
[ dbug ] (<<) <InstallFakeQPKGs> [0]
[ dbug ] (>>) <InstallIPKs>
[ dbug ] (>>) <UpdateEntware>
[ proc ] updating 'Entware' ...
[ done ] updated 'Entware'
[ dbug ] (<<) <UpdateEntware> [0]
[ dbug ] (--) reloaded environment
[ proc ] downloading & installing IPKs (various) ...
[ done ] downloaded & installed IPKs (various)
[ proc ] downloading & installing IPK (python-dev) ...
[ done ] downloaded & installed IPK (python-dev)
[ dbug ] (<<) <InstallIPKs> [0]
[ dbug ] (>>) <InstallPIPs>
[ proc ] downloading & installing (pip modules) ...
[ fail ] Download & install failed (pip modules) [1]
[ dbug ] (<<) <InstallPIPs> [17]
[ dbug ] (<<) <InstallOther> [17]
[ dbug ] (>>) <Cleanup>
[ dbug ] (**) QPKG:      'SABnzbdplus': installed
[ dbug ] (--) resumed existing SABnzbd queue
[ dbug ] (<<) <Cleanup> [17]
[ dbug ] (>>) <DisplayResult>
[ fail ] SABnzbdplus install failed! :S [17]
[ dbug ] (**) SCRIPT:         finished: Sun May  7 12:04:49 MST 2017
[ dbug ] (**) SCRIPT:     elapsed time: 00h:00m:27s
[ dbug ] (II) ====================================================================
[ dbug ] (<<) <DisplayResult> [17]

[OneCD]

However, the script fails at the pip modules.

Okiedoke, there's a separate log for pip modules. You can view this with:

Code: Select all

cat /share/Public/sherpa.tmp/pip_modules.install.log

Can you please post it back here?

[scorche]

Looks like I need to get CXXABI_1.3.9 on my QNAP...

Code: Select all

Requirement already up-to-date: pip in /share/CACHEDEV1_DATA/.qpkg/Entware-ng/lib/python2.7/site-packages
Requirement already up-to-date: setuptools in /share/CACHEDEV1_DATA/.qpkg/Entware-ng/lib/python2.7/site-packages
Requirement already up-to-date: packaging>=16.8 in /share/CACHEDEV1_DATA/.qpkg/Entware-ng/lib/python2.7/site-packages (from setuptools)
Requirement already up-to-date: appdirs>=1.4.0 in /share/CACHEDEV1_DATA/.qpkg/Entware-ng/lib/python2.7/site-packages (from setuptools)
Requirement already up-to-date: six>=1.6.0 in /share/CACHEDEV1_DATA/.qpkg/Entware-ng/lib/python2.7/site-packages (from setuptools)
Requirement already up-to-date: pyparsing in /share/CACHEDEV1_DATA/.qpkg/Entware-ng/lib/python2.7/site-packages (from packaging>=16.8->setuptools)
Collecting sabyenc
  Using cached sabyenc-3.0.2.tar.gz
Collecting cheetah
  Using cached Cheetah-2.4.4.tar.gz
Collecting Markdown>=2.0.1 (from cheetah)
  Using cached Markdown-2.6.8.tar.gz
Installing collected packages: sabyenc, Markdown, cheetah
  Running setup.py install for sabyenc: started
    Running setup.py install for sabyenc: finished with status 'error'
    Complete output from command /opt/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/share/CACHEDEV1_DATA/.qpkg/Entware-ng/tmp/pip-build-qbGM6r/sabyenc/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('
', '
');f.close();exec(compile(code, __file__, 'exec'))" install --record /opt/tmp/pip-6qg2mo-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    building 'sabyenc' extension
    creating build
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/src
    x86_64-openwrt-linux-gnu-gcc -fno-strict-aliasing -O2 -pipe -fomit-frame-pointer -fno-caller-saves -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -DNDEBUG -fno-inline -DNDEBUG -O2 -pipe -fomit-frame-pointer -fno-caller-saves -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fPIC -I/opt/include/python2.7 -c src/sabyenc.c -o build/temp.linux-x86_64-2.7/src/sabyenc.o -O2
    x86_64-openwrt-linux-gnu-gcc: /opt/lib/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by x86_64-openwrt-linux-gnu-gcc)
    error: command 'x86_64-openwrt-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/opt/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/share/CACHEDEV1_DATA/.qpkg/Entware-ng/tmp/pip-build-qbGM6r/sabyenc/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('
', '
');f.close();exec(compile(code, __file__, 'exec'))" install --record /opt/tmp/pip-6qg2mo-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /share/CACHEDEV1_DATA/.qpkg/Entware-ng/tmp/pip-build-qbGM6r/sabyenc/
result=[1]

[OneCD]

Hmmm... are you able to use Entware-3x instead of Entware-ng?

If so, please uninstall Entware-ng via the QTS App Center and re-run sherpa. It will install Entware-3x itself.

If not, please uninstall Entware-ng and reinstall it manually. Then re-run sherpa.

[scorche]

Looks good! Thanks!

[The Mask]

If you're able, can you try these two and let me know if they work for you?

Code: Select all

/usr/bin/wget https://goo.gl/Qot9QK -O sherpa.tar.gz

Code: Select all

wget http://goo.gl/Qot9QK -O sherpa.tar.gz

Only the first one works for me, the other one is giving the following error:

Connecting to goo.gl (216.58.212.206:80)
wget: not an http or ftp url: https://github.com/OneCDOnly/sherpa/blo ... z?raw=true

[OneCD]

Looks good! Thanks!

So, which one did you end up using? 3x or ng?

[OneCD]

Only the first one works for me, the other one is giving the following error:

Connecting to goo.gl (216.58.212.206:80)
wget: not an http or ftp url: https://github.com/OneCDOnly/sherpa/blo ... z?raw=true

Thanks for checking that. I'll change the download link code to specify the full path to the wget binary.

I think the second one failed on your NAS as it was trying to use BusyBox wget which doesn't support HTTPS.

Code: Select all

which wget; echo; wget -V

[scorche]

So, which one did you end up using? 3x or ng?

I just removed ng and let the script install 3x. I figured I would address any issues that popped up separately if there was a reason why I needed ng for something else.