[QPKG] sherpa mini-package-manager

This is the best place for community developers to publish their genius work. Your Apps enrich the QNAP Turbo NAS.
Post Reply
rs1001
Starting out
Posts: 31
Joined: Thu Jun 18, 2015 11:44 am

Re: [SCRIPT] sherpa.sh

Post by rs1001 » Sat May 30, 2020 2:54 am

Enhancement Request: ie; This comes from me doing something stupid rather than any functionality issue.

So I was trying to get my hand on sab3 and it was failing.
Came here saw that I may need to reinstall Entware.
Went back, reinstalled. -- Haven't tested Sab3 yet.. I digress.

In uninstalling Entware, I lost Transmission which I had installed via Entware and with it my seeding history.

Would it be possible to when uninstalling Entware:

- Gentle: 1) Put in a warning that anything installed via Entware manually will be lost. [Agree/Cancel]
2) Export a list of what was previously installed. opkg list_installed > some.log (or something tidier)

- Hardcore: Provide an option to backup all config of installed items. (Probably tough consider all the things people may have installed).
QNAP TS-653 Pro -- with 16Gb RAM
6 * Western Digital WD60EFRX WD Reds (RAID5) as a Single 27.1TB Volume
Firmware: QTS 4.4.3.1354

rs1001
Starting out
Posts: 31
Joined: Thu Jun 18, 2015 11:44 am

Re: [SCRIPT] sherpa.sh

Post by rs1001 » Sat May 30, 2020 3:46 am

On the testing Sab3 front. Install looked to go fine, but I can't get to the Web Interface.
Get the pesky "Connection Refused" jobbie. Anyone else got any further with it?
QNAP TS-653 Pro -- with 16Gb RAM
6 * Western Digital WD60EFRX WD Reds (RAID5) as a Single 27.1TB Volume
Firmware: QTS 4.4.3.1354

User avatar
OneCD
Ask me anything
Posts: 7804
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [SCRIPT] sherpa.sh

Post by OneCD » Sat May 30, 2020 2:34 pm

rs1001 wrote:
Sat May 30, 2020 2:54 am
- Gentle: 1) Put in a warning that anything installed via Entware manually will be lost. [Agree/Cancel]
This is described in the OP for this topic, but I don't know how many people notice it. ;)

The plan for this installer has always been for it to be non-interactive. But this isn't set-in-stone. Maybe interactive would be better?

I think your idea (agree/cancel) is a good one, and I'll work on adding support for user-interactions.
rs1001 wrote:
Sat May 30, 2020 2:54 am
2) Export a list of what was previously installed. opkg list_installed > some.log (or something tidier)
No problem. Will do.
rs1001 wrote:
Sat May 30, 2020 2:54 am
- Hardcore: Provide an option to backup all config of installed items. (Probably tough consider all the things people may have installed).
Now, this is a difficult one. I suspect IPKG configs are a bit like QPKG configs - they can be put almost anywhere. :(

If you're able to figure this out, I'd be happy to include an "external" Entware config backup process. Each of the sherpa QPKGs are being modified to permit configuration backup and restore via the CLI. It would be great to include Entware too.

The problem is that additional packages can affect those required for running the sherpa apps in bad ways. It’s why I originally never added support for Entware backup and restore as it would be too easy to restore a conflicting package (required by another IPKG installed by the user) that then prevents a sherpa app from running.

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage

User avatar
OneCD
Ask me anything
Posts: 7804
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [SCRIPT] sherpa.sh

Post by OneCD » Sat May 30, 2020 2:37 pm

rs1001 wrote:
Sat May 30, 2020 3:46 am
On the testing Sab3 front. Install looked to go fine, but I can't get to the Web Interface.
Get the pesky "Connection Refused" jobbie. Anyone else got any further with it?
I noticed SAB3 can take a while to appear. And sometimes, the first application run doesn't work, and the package must be stopped then started again. I'll be working on this issue tomorrow morning (after watching the Demo-2 SpaceX launch). :geek:

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage

User avatar
OneCD
Ask me anything
Posts: 7804
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [SCRIPT] sherpa.sh

Post by OneCD » Sat Jun 06, 2020 6:54 am

Major update this morning! (ah, it's great to get back to shell-scripting) :D

All sherpa QPKGs (except Entware and Par2) have been upgraded with my new init script that has a few extra features:
  • supports application configuration-only backup and restore via the 'backup' and 'restore' arguments. This can be scripted via cron to create a regular backup of each app.

    QPKG (configuration-only) backups will be stored in a new hidden directory located under your default userdata volume called [.qpkg_config_backup/]. Hopefully, QNAP won't mess with this location (I'm looking at you Malware Remover).

    To jump to this path:

    Code: Select all

    cd $(getcfg SHARE_DEF defVolMP -f /etc/config/def_share.info)/.qpkg_config_backup
    Each QPKG has a single [config.tar.gz] file to backup into. Each new backup replaces the old one (so, no versioning available).

    Example: to backup SABnzbdplus:

    Code: Select all

    /etc/init.d/sabnzbd2.sh backup
    Example: to restore SABnzbdplus:

    Code: Select all

    /etc/init.d/sabnzbd2.sh restore
    This will 'stop' the QPKG, restore from the backup file (if it exists), then 'start' the QPKG again.
  • check that the application presents a responsive UI port after launch,
  • include a few helpful info/warning/error messages in the QTS system log which should reduce user-time at the CLI,
  • improved on-screen display accuracy when running the init script manually,
  • ... and all QPKGs have had minor tweaks for stability and compatibility with each other.
Bugs? I've not seen any yet, but please report them if you find them.

The new versions are all 200606 or later.

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage

User avatar
OneCD
Ask me anything
Posts: 7804
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [SCRIPT] sherpa.sh

Post by OneCD » Sat Jun 06, 2020 2:59 pm

OneCD wrote:
Sat May 30, 2020 2:34 pm
rs1001 wrote:
Sat May 30, 2020 2:54 am
- Gentle: 1) Put in a warning that anything installed via Entware manually will be lost. [Agree/Cancel]
...
I think your idea (agree/cancel) is a good one, and I'll work on adding support for user-interactions.
This is now done, although I had to make a change to the prompt and acceptable characters. I eventually realised (a)gree vs (c)ancel was too easily confused with the opposite: (a)bort vs (c)ontinue. :DD

So, I switched to (y)es vs anything-else to abort.

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage

rs1001
Starting out
Posts: 31
Joined: Thu Jun 18, 2015 11:44 am

Re: [SCRIPT] sherpa.sh

Post by rs1001 » Sat Jun 06, 2020 8:57 pm

Very cool. Thank you. Testing now.

A worthy add might be to do a "opkg list_installed > /Public/had_installed.txt" in the log directory to make the re-install easier.
In my particular instance, it's only Transmission that it messes with and I will have to re-add and re-seed them all one by one.

Would be nice if it could omit Transmission, but that's probably too particular an ask.
QNAP TS-653 Pro -- with 16Gb RAM
6 * Western Digital WD60EFRX WD Reds (RAID5) as a Single 27.1TB Volume
Firmware: QTS 4.4.3.1354

User avatar
OneCD
Ask me anything
Posts: 7804
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [SCRIPT] sherpa.sh

Post by OneCD » Sun Jun 07, 2020 2:57 am

rs1001 wrote:
Sat Jun 06, 2020 8:57 pm
A worthy add might be to do a "opkg list_installed > /Public/had_installed.txt" in the log directory to make the re-install easier.
It's on today's 'todo' list. ;)
rs1001 wrote:
Sat Jun 06, 2020 8:57 pm
Would be nice if it could omit Transmission, but that's probably too particular an ask.
"omit"? :'

I'll see if I can make Transmission into a new sherpa QPKG. It will just be a link to the Entware-installed Transmission installation. ;)

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage

User avatar
OneCD
Ask me anything
Posts: 7804
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [SCRIPT] sherpa.sh

Post by OneCD » Sun Jun 07, 2020 5:27 am

OneCD wrote:
Sun Jun 07, 2020 2:57 am
rs1001 wrote:
Sat Jun 06, 2020 8:57 pm
A worthy add might be to do a "opkg list_installed > /Public/had_installed.txt" in the log directory to make the re-install easier.
It's on today's 'todo' list. ;)
... and done! :geek:

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage

User avatar
OneCD
Ask me anything
Posts: 7804
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [SCRIPT] sherpa.sh

Post by OneCD » Fri Jun 19, 2020 3:13 pm

I've just found out that Python 2.7.x is no-longer available via Entware as it's been removed from OpenWRT. :(

So, this means re-installations or new installations of SABnzbd 2.3.9, SickChill and Headphones will be temporarily unavailable. I'll see if I can find a usable Python 2.7.x source to get those packages working again, but it may not be possible to install all required Python 2 modules. This could take some time to fix - if it can be fixed at all.

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage

zyxmon
Been there, done that
Posts: 861
Joined: Mon Oct 27, 2014 2:27 pm
Location: Moscow, Russia
Contact:

Re: [SCRIPT] sherpa.sh

Post by zyxmon » Fri Jun 19, 2020 10:13 pm

FYI Entware was updated yesterday. It does not have Python 2.x any more. This is openwrt decision to remove Python2 from the repo. There is a working copy of the previous Entware repo - but it will live a month or two.

User avatar
OneCD
Ask me anything
Posts: 7804
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [SCRIPT] sherpa.sh

Post by OneCD » Sat Jun 20, 2020 12:38 am

Thanks @zyxmon. :geek:

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage

Zuzler
New here
Posts: 8
Joined: Sat Aug 03, 2013 7:32 am

Re: [SCRIPT] sherpa.sh

Post by Zuzler » Sun Jun 21, 2020 2:41 am

I'm currently trying to install LazyLibrarian. After following all steps of the initial post, I'm getting this error message:

Code: Select all

[ fail ] A required NAS system file is missing (/opt/bin/pip3)

* Ugh! The usual fix is to let sherpa reinstall Entware at least once.
        ./sherpa.sh ew
If it happens again after reinstalling Entware, please create a new issue for this on GitHub.
I ran ./sherpa.sh ew - no luck there. Did I miss something? Thanks :)

The logfile:

Code: Select all

[ dbug ] (vv) $satisfy_dependencies_only [false]
[ dbug ] (vv) $help_only [false]
[ dbug ] (vv) $suggest_issue [false]
[ dbug ] (>>) LogRuntimeParameters()
[ dbug ] (II) ========================================================================
[ dbug ] (**) SCRIPT:             started: Sat Jun 20 20:36:08 CEST 2020
[ dbug ] (**) SCRIPT:             version: 200620
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (II) Markers: (**) detected, (II) information, (WW) warning, (LL) log file,
[ dbug ] (II)  (EE) error, (==) processing, (--) done, (>>) f entry, (<<) f exit,
[ dbug ] (II)  (vv) variable name & value, ($1) positional argument value.
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (**) NAS   :               model: TS-421
[ dbug ] (**) NAS   :                 RAM: 1035288 kB
[ dbug ] (**) NAS   :                 RAM: less-than or equal-to 1048576 kB
[ note ] QTS with 1GB RAM or less can lead to unstable SABnzbd uptimes :(
[ dbug ] (**) NAS   :    firmware version: 4.3.3
[ dbug ] (**) NAS   :      firmware build: 20200409
[ dbug ] (**) NAS   :              kernel: 3.4.6 armv5tel
[ dbug ] (**) NAS   :           OS uptime: 30 days
[ dbug ] (**) NAS   :         system load: 1 min=11.24, 5 min=11.86, 15 min=10.79
[ dbug ] (**) NAS   :                USER: admin
[ dbug ] (**) NAS   :                EUID: 0
[ dbug ] (**) NAS   :      default volume: /share/MD0_DATA
[ dbug ] (**) NAS   :               $PATH: /opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr
[ dbug ] (**) NAS   :                /opt: /share/MD0_DATA/.qpkg/Entware
[ dbug ] (**) NAS   :     /share/Download: MD0_DATA/Download
[ dbug ] (**) SCRIPT:  unparsed arguments: ew
[ dbug ] (**) SCRIPT:   app(s) to install: Entware
[ dbug ] (**) SCRIPT: app(s) to uninstall
[ dbug ] (**) SCRIPT: app(s) to reinstall
[ dbug ] (**) SCRIPT:    app(s) to update
[ dbug ] (**) SCRIPT:    app(s) to backup
[ dbug ] (**) SCRIPT:   app(s) to restore
[ dbug ] (**) SCRIPT:        working path: /share/Public/sherpa.tmp
[ dbug ] (**) QPKG  :       download path: /share/Public/sherpa.tmp/qpkg-downloads
[ dbug ] (**) IPKG  :       download path: /share/Public/sherpa.tmp/ipkg-downloads
[ dbug ] (**) QPKG  :                arch: none
[ dbug ] (**) QPKG  :   Entware installer: std
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (<<) LogRuntimeParameters() [0]
[ dbug ] (>>) DownloadQPKGs()
[ dbug ] (>>) DownloadQPKG()
[ proc ] downloading file (Entware_1.02std.qpkg) ...
[ done ] downloaded file (Entware_1.02std.qpkg)
[ dbug ] (<<) DownloadQPKG() [0]
[ dbug ] (<<) DownloadQPKGs() [0]
[ dbug ] (>>) RemoveUnwantedQPKGs()
[ dbug ] (**) QPKG  :           'Optware': not installed [250]
[ dbug ] (**) QPKG  :        'Entware-3x': not installed [250]
[ dbug ] (**) QPKG  :        'Entware-ng': not installed [250]
[ note ] Reinstalling Entware will revert all IPKGs to defaults and only those required to support your sherpa apps will be reinstalled
[ note ] The currently installed IPKG list will be saved to (/share/Public/Entware.previously.installed.list)
[ warn ] Also, the SABnzbdplus, SickChill and Headphones packages CANNOT BE REINSTALLED as Python 2.7.16 is no-longer available
[ quiz ] Press (y) if you agree to remove all current Entware IPKGs and their configs, or any other key to abort
[ dbug ] (vv) $response [y]
[ dbug ] (--) saved current Entware IPKG list to (/share/Public/Entware.previously.installed.list)
[ proc ] uninstalling 'Entware' ...
[ done ] uninstalled 'Entware'
[ dbug ] (<<) RemoveUnwantedQPKGs() [0]
[ dbug ] (>>) InstallBase()
[ proc ] installing file (Entware_1.02std.qpkg) - this can take a while ...
[ done ] installed file (Entware_1.02std.qpkg)
[ dbug ] (--) adjusted $PATH
[ dbug ] (vv) $PATH [/opt/bin:/opt/sbin:/opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin/X11:/usr/local/sbin:/usr/local/bin:/Apps/bin:/Apps/sbin]
[ dbug ] (>>) PatchBaseInit()
[ dbug ] (--) patch: do the "opt shuffle"
[ dbug ] (<<) PatchBaseInit() [0]
[ dbug ] (<<) InstallBase() [0]
[ dbug ] (>>) InstallBaseAddons()
[ dbug ] (>>) InstallIPKGs()
[ dbug ] (>>) UpdateEntware()
[ proc ] updating 'Entware' package list ...
[ done ] updated 'Entware' package list
[ dbug ] (<<) UpdateEntware() [0]
[ dbug ] (>>) InstallIPKGBatch()
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (**) STAGE :   start stage timer
[ proc ] calculating number and total size of IPKGs required ...
[ dbug ] (II) requested IPKGs: ca-certificates git git-http less nano nzbget par2cmdline
[ dbug ] (==) finding all IPKG dependencies ...
[ dbug ] (--) complete
[ dbug ] (II) found all IPKG dependencies in 4 iterations
[ dbug ] (==) excluding packages already installed ...
[ dbug ] (--) complete
[ dbug ] (II) IPKGs to download: ca-certificates
[ dbug ] (==) calculating size of IPKG to download ...
[ dbug ] (--) complete
[ dbug ] (vv) $IPKG_download_size [125484]
[ dbug ] (**) STAGE :        elapsed time: 00h:00m:03s
[ dbug ] (II) ------------------------------------------------------------------------
[ done ] 1 IPKG (125kB) to be downloaded
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (**) STAGE :   start stage timer
[ proc ] downloading & installing 1 IPKG ...
[ done ] downloaded & installed 1 IPKG
[ dbug ] (**) STAGE :        elapsed time: 00h:00m:04s
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (<<) InstallIPKGBatch() [0]
[ dbug ] (<<) InstallIPKGs() [0]
[ dbug ] (>>) InstallPy3Modules()
[ fail ] A required NAS system file is missing (/opt/bin/pip3)
[ dbug ] (<<) InstallBaseAddons() [20]
[ dbug ] (>>) Cleanup()
[ dbug ] (<<) Cleanup() [20]
[ dbug ] (>>) ShowResult()
[ fail ] 'Entware' reinstall failed! [20]
[ dbug ] (vv) $suggest_issue [true]
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (**) SCRIPT:            finished: Sat Jun 20 20:37:30 CEST 2020
[ dbug ] (**) SCRIPT:        elapsed time: 00h:01m:22s
[ dbug ] (II) ========================================================================
[ dbug ] (<<) ShowResult() [20]

User avatar
OneCD
Ask me anything
Posts: 7804
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [SCRIPT] sherpa.sh

Post by OneCD » Sun Jun 21, 2020 4:16 am

Zuzler wrote:
Sun Jun 21, 2020 2:41 am
I'm currently trying to install LazyLibrarian. After following all steps of the initial post, I'm getting this error message:

Code: Select all

[ fail ] A required NAS system file is missing (/opt/bin/pip3)
Oops! My bad. :oops:

This is now fixed. Please use sherpa v200621 or later and try your install again.

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage

Zuzler
New here
Posts: 8
Joined: Sat Aug 03, 2013 7:32 am

Re: [SCRIPT] sherpa.sh

Post by Zuzler » Sun Jun 21, 2020 5:46 am

That was quick :shock:
Thank you so much, v200621 works flawlessly :lol:

One more question, do I still need to install / keep the regular Python 2.x and 3.x from the official AppCenter, or are they made superflous by the installation done by sherpa? In my case it's 2.7.3 and 3.5.0.2

Post Reply

Return to “Community Apps”