[QPKG] sherpa: a mini-package-manager (CLI) beta

[rs1001]

Enhancement Request: ie; This comes from me doing something stupid rather than any functionality issue.

So I was trying to get my hand on sab3 and it was failing.
Came here saw that I may need to reinstall Entware.
Went back, reinstalled. -- Haven't tested Sab3 yet.. I digress.

In uninstalling Entware, I lost Transmission which I had installed via Entware and with it my seeding history.

Would it be possible to when uninstalling Entware:

- Gentle: 1) Put in a warning that anything installed via Entware manually will be lost. [Agree/Cancel]
2) Export a list of what was previously installed. opkg list_installed > some.log (or something tidier)

- Hardcore: Provide an option to backup all config of installed items. (Probably tough consider all the things people may have installed).

[rs1001]

On the testing Sab3 front. Install looked to go fine, but I can't get to the Web Interface.
Get the pesky "Connection Refused" jobbie. Anyone else got any further with it?

[OneCD]

- Gentle: 1) Put in a warning that anything installed via Entware manually will be lost. [Agree/Cancel]

This is described in the OP for this topic, but I don't know how many people notice it.

The plan for this installer has always been for it to be non-interactive. But this isn't set-in-stone. Maybe interactive would be better?

I think your idea (agree/cancel) is a good one, and I'll work on adding support for user-interactions.

2) Export a list of what was previously installed. opkg list_installed > some.log (or something tidier)

No problem. Will do.

- Hardcore: Provide an option to backup all config of installed items. (Probably tough consider all the things people may have installed).

Now, this is a difficult one. I suspect IPKG configs are a bit like QPKG configs - they can be put almost anywhere.

If you're able to figure this out, I'd be happy to include an "external" Entware config backup process. Each of the sherpa QPKGs are being modified to permit configuration backup and restore via the CLI. It would be great to include Entware too.

The problem is that additional packages can affect those required for running the sherpa apps in bad ways. It’s why I originally never added support for Entware backup and restore as it would be too easy to restore a conflicting package (required by another IPKG installed by the user) that then prevents a sherpa app from running.

[OneCD]

On the testing Sab3 front. Install looked to go fine, but I can't get to the Web Interface.
Get the pesky "Connection Refused" jobbie. Anyone else got any further with it?

I noticed SAB3 can take a while to appear. And sometimes, the first application run doesn't work, and the package must be stopped then started again. I'll be working on this issue tomorrow morning (after watching the Demo-2 SpaceX launch).

[OneCD]

Major update this morning! (ah, it's great to get back to shell-scripting)

All sherpa QPKGs (except Entware and Par2) have been upgraded with my new init script that has a few extra features:

• supports application configuration-only backup and restore via the 'backup' and 'restore' arguments. This can be scripted via cron to create a regular backup of each app.
  
  QPKG (configuration-only) backups will be stored in a new hidden directory located under your default userdata volume called [.qpkg_config_backup/]. Hopefully, QNAP won't mess with this location (I'm looking at you Malware Remover).
  
  To jump to this path:

  Code: Select all

  cd $(getcfg SHARE_DEF defVolMP -f /etc/config/def_share.info)/.qpkg_config_backup

  Each QPKG has a single [config.tar.gz] file to backup into. Each new backup replaces the old one (so, no versioning available).
  
  Example: to backup SABnzbdplus:

  Code: Select all

  /etc/init.d/sabnzbd2.sh backup

  Example: to restore SABnzbdplus:

  Code: Select all

  /etc/init.d/sabnzbd2.sh restore

  This will 'stop' the QPKG, restore from the backup file (if it exists), then 'start' the QPKG again.
• check that the application presents a responsive UI port after launch,
• include a few helpful info/warning/error messages in the QTS system log which should reduce user-time at the CLI,
• improved on-screen display accuracy when running the init script manually,
• ... and all QPKGs have had minor tweaks for stability and compatibility with each other.

Bugs? I've not seen any yet, but please report them if you find them.

The new versions are all 200606 or later.

[OneCD]

- Gentle: 1) Put in a warning that anything installed via Entware manually will be lost. [Agree/Cancel]

...
I think your idea (agree/cancel) is a good one, and I'll work on adding support for user-interactions.

This is now done, although I had to make a change to the prompt and acceptable characters. I eventually realised (a)gree vs (c)ancel was too easily confused with the opposite: (a)bort vs (c)ontinue.

So, I switched to (y)es vs anything-else to abort.

[rs1001]

Very cool. Thank you. Testing now.

A worthy add might be to do a "opkg list_installed > /Public/had_installed.txt" in the log directory to make the re-install easier.
In my particular instance, it's only Transmission that it messes with and I will have to re-add and re-seed them all one by one.

Would be nice if it could omit Transmission, but that's probably too particular an ask.

[OneCD]

A worthy add might be to do a "opkg list_installed > /Public/had_installed.txt" in the log directory to make the re-install easier.

It's on today's 'todo' list.

Would be nice if it could omit Transmission, but that's probably too particular an ask.

"omit"?

I'll see if I can make Transmission into a new sherpa QPKG. It will just be a link to the Entware-installed Transmission installation.

[OneCD]

A worthy add might be to do a "opkg list_installed > /Public/had_installed.txt" in the log directory to make the re-install easier.

It's on today's 'todo' list.

... and done!

[OneCD]

I've just found out that Python 2.7.x is no-longer available via Entware as it's been removed from OpenWRT.

So, this means re-installations or new installations of SABnzbd 2.3.9, SickChill and Headphones will be temporarily unavailable. I'll see if I can find a usable Python 2.7.x source to get those packages working again, but it may not be possible to install all required Python 2 modules. This could take some time to fix - if it can be fixed at all.

[zyxmon]

FYI Entware was updated yesterday. It does not have Python 2.x any more. This is openwrt decision to remove Python2 from the repo. There is a working copy of the previous Entware repo - but it will live a month or two.

[OneCD]

Thanks @zyxmon.

[Zuzler]

I'm currently trying to install LazyLibrarian. After following all steps of the initial post, I'm getting this error message:

Code: Select all

[ fail ] A required NAS system file is missing (/opt/bin/pip3)

* Ugh! The usual fix is to let sherpa reinstall Entware at least once.
        ./sherpa.sh ew
If it happens again after reinstalling Entware, please create a new issue for this on GitHub.

I ran ./sherpa.sh ew - no luck there. Did I miss something? Thanks

The logfile:

Code: Select all

[ dbug ] (vv) $satisfy_dependencies_only [false]
[ dbug ] (vv) $help_only [false]
[ dbug ] (vv) $suggest_issue [false]
[ dbug ] (>>) LogRuntimeParameters()
[ dbug ] (II) ========================================================================
[ dbug ] (**) SCRIPT:             started: Sat Jun 20 20:36:08 CEST 2020
[ dbug ] (**) SCRIPT:             version: 200620
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (II) Markers: (**) detected, (II) information, (WW) warning, (LL) log file,
[ dbug ] (II)  (EE) error, (==) processing, (--) done, (>>) f entry, (<<) f exit,
[ dbug ] (II)  (vv) variable name & value, ($1) positional argument value.
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (**) NAS   :               model: TS-421
[ dbug ] (**) NAS   :                 RAM: 1035288 kB
[ dbug ] (**) NAS   :                 RAM: less-than or equal-to 1048576 kB
[ note ] QTS with 1GB RAM or less can lead to unstable SABnzbd uptimes :(
[ dbug ] (**) NAS   :    firmware version: 4.3.3
[ dbug ] (**) NAS   :      firmware build: 20200409
[ dbug ] (**) NAS   :              kernel: 3.4.6 armv5tel
[ dbug ] (**) NAS   :           OS uptime: 30 days
[ dbug ] (**) NAS   :         system load: 1 min=11.24, 5 min=11.86, 15 min=10.79
[ dbug ] (**) NAS   :                USER: admin
[ dbug ] (**) NAS   :                EUID: 0
[ dbug ] (**) NAS   :      default volume: /share/MD0_DATA
[ dbug ] (**) NAS   :               $PATH: /opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr
[ dbug ] (**) NAS   :                /opt: /share/MD0_DATA/.qpkg/Entware
[ dbug ] (**) NAS   :     /share/Download: MD0_DATA/Download
[ dbug ] (**) SCRIPT:  unparsed arguments: ew
[ dbug ] (**) SCRIPT:   app(s) to install: Entware
[ dbug ] (**) SCRIPT: app(s) to uninstall
[ dbug ] (**) SCRIPT: app(s) to reinstall
[ dbug ] (**) SCRIPT:    app(s) to update
[ dbug ] (**) SCRIPT:    app(s) to backup
[ dbug ] (**) SCRIPT:   app(s) to restore
[ dbug ] (**) SCRIPT:        working path: /share/Public/sherpa.tmp
[ dbug ] (**) QPKG  :       download path: /share/Public/sherpa.tmp/qpkg-downloads
[ dbug ] (**) IPKG  :       download path: /share/Public/sherpa.tmp/ipkg-downloads
[ dbug ] (**) QPKG  :                arch: none
[ dbug ] (**) QPKG  :   Entware installer: std
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (<<) LogRuntimeParameters() [0]
[ dbug ] (>>) DownloadQPKGs()
[ dbug ] (>>) DownloadQPKG()
[ proc ] downloading file (Entware_1.02std.qpkg) ...
[ done ] downloaded file (Entware_1.02std.qpkg)
[ dbug ] (<<) DownloadQPKG() [0]
[ dbug ] (<<) DownloadQPKGs() [0]
[ dbug ] (>>) RemoveUnwantedQPKGs()
[ dbug ] (**) QPKG  :           'Optware': not installed [250]
[ dbug ] (**) QPKG  :        'Entware-3x': not installed [250]
[ dbug ] (**) QPKG  :        'Entware-ng': not installed [250]
[ note ] Reinstalling Entware will revert all IPKGs to defaults and only those required to support your sherpa apps will be reinstalled
[ note ] The currently installed IPKG list will be saved to (/share/Public/Entware.previously.installed.list)
[ warn ] Also, the SABnzbdplus, SickChill and Headphones packages CANNOT BE REINSTALLED as Python 2.7.16 is no-longer available
[ quiz ] Press (y) if you agree to remove all current Entware IPKGs and their configs, or any other key to abort
[ dbug ] (vv) $response [y]
[ dbug ] (--) saved current Entware IPKG list to (/share/Public/Entware.previously.installed.list)
[ proc ] uninstalling 'Entware' ...
[ done ] uninstalled 'Entware'
[ dbug ] (<<) RemoveUnwantedQPKGs() [0]
[ dbug ] (>>) InstallBase()
[ proc ] installing file (Entware_1.02std.qpkg) - this can take a while ...
[ done ] installed file (Entware_1.02std.qpkg)
[ dbug ] (--) adjusted $PATH
[ dbug ] (vv) $PATH [/opt/bin:/opt/sbin:/opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin/X11:/usr/local/sbin:/usr/local/bin:/Apps/bin:/Apps/sbin]
[ dbug ] (>>) PatchBaseInit()
[ dbug ] (--) patch: do the "opt shuffle"
[ dbug ] (<<) PatchBaseInit() [0]
[ dbug ] (<<) InstallBase() [0]
[ dbug ] (>>) InstallBaseAddons()
[ dbug ] (>>) InstallIPKGs()
[ dbug ] (>>) UpdateEntware()
[ proc ] updating 'Entware' package list ...
[ done ] updated 'Entware' package list
[ dbug ] (<<) UpdateEntware() [0]
[ dbug ] (>>) InstallIPKGBatch()
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (**) STAGE :   start stage timer
[ proc ] calculating number and total size of IPKGs required ...
[ dbug ] (II) requested IPKGs: ca-certificates git git-http less nano nzbget par2cmdline
[ dbug ] (==) finding all IPKG dependencies ...
[ dbug ] (--) complete
[ dbug ] (II) found all IPKG dependencies in 4 iterations
[ dbug ] (==) excluding packages already installed ...
[ dbug ] (--) complete
[ dbug ] (II) IPKGs to download: ca-certificates
[ dbug ] (==) calculating size of IPKG to download ...
[ dbug ] (--) complete
[ dbug ] (vv) $IPKG_download_size [125484]
[ dbug ] (**) STAGE :        elapsed time: 00h:00m:03s
[ dbug ] (II) ------------------------------------------------------------------------
[ done ] 1 IPKG (125kB) to be downloaded
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (**) STAGE :   start stage timer
[ proc ] downloading & installing 1 IPKG ...
[ done ] downloaded & installed 1 IPKG
[ dbug ] (**) STAGE :        elapsed time: 00h:00m:04s
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (<<) InstallIPKGBatch() [0]
[ dbug ] (<<) InstallIPKGs() [0]
[ dbug ] (>>) InstallPy3Modules()
[ fail ] A required NAS system file is missing (/opt/bin/pip3)
[ dbug ] (<<) InstallBaseAddons() [20]
[ dbug ] (>>) Cleanup()
[ dbug ] (<<) Cleanup() [20]
[ dbug ] (>>) ShowResult()
[ fail ] 'Entware' reinstall failed! [20]
[ dbug ] (vv) $suggest_issue [true]
[ dbug ] (II) ------------------------------------------------------------------------
[ dbug ] (**) SCRIPT:            finished: Sat Jun 20 20:37:30 CEST 2020
[ dbug ] (**) SCRIPT:        elapsed time: 00h:01m:22s
[ dbug ] (II) ========================================================================
[ dbug ] (<<) ShowResult() [20]

[OneCD]

I'm currently trying to install LazyLibrarian. After following all steps of the initial post, I'm getting this error message:

Code: Select all

[ fail ] A required NAS system file is missing (/opt/bin/pip3)

Oops! My bad.

This is now fixed. Please use sherpa v200621 or later and try your install again.

[Zuzler]

That was quick
Thank you so much, v200621 works flawlessly

One more question, do I still need to install / keep the regular Python 2.x and 3.x from the official AppCenter, or are they made superflous by the installation done by sherpa? In my case it's 2.7.3 and 3.5.0.2