[QPKG] sherpa: a mini-package-manager (CLI) beta

[winstona]

Cheers @OneCD I hope life is treating you well and that you and your loved ones are safe and healthy.

Hey Dirk, you too.

So, because I didn’t see it when I had a look, what is the intended way to migrate the existing SAB 2.3.9 to 3x using Sherpa?

OK, the new SAB3 has been created as a new QPKG called 'SABnzbd'. The previous one was called 'SABnzbdplus'.

If you were using an older version of the 'SABnzbdplus' QPKG, then suggest backing-up your settings, then restoring them into the new 'SABnzbd' QPKG.

Please download the latest sherpa.sh (as per the notes in the first post in this topic) and install 'SABnzbd' (SAB3):

Code: Select all

./sherpa.sh sab

... then create a manual backup from your old 'SABnzbdplus':

Code: Select all

/bin/tar --create --gzip --file=$(getcfg SHARE_DEF defVolMP -f /etc/config/def_share.info)/.qpkg_config_backup/SABnzbd.config.tar.gz --directory=$(getcfg SABnzbdplus Install_Path -f /etc/config/qpkg.conf)/config .

... then stop the old package:

Code: Select all

[[ -e /etc/init.d/sabnzbd2.sh ]] && /etc/init.d/sabnzbd2.sh stop || /etc/init.d/sabnzbd.sh stop

... then restore your backup into 'SABnzbd':

Code: Select all

/etc/init.d/sabnzbd3.sh restore

Once you've confirmed 'SABnzbd' works fine, you can uninstall the older 'SABnzbdplus' QPKG from your App Center.

This did the trick for me. Thank you very much!

[OneCD]

This did the trick for me. Thank you very much!

Great!

Now that you're using the newest SABnzbd QPKG, you can perform configuration backups with:

Code: Select all

/etc/init.d/sabnzbd3.sh backup

[OneCD]

I did see, that OpenSSL seems to be an older version "OpenSSL 1.1.1d 10 Sep 2019" is that correct? I remember we saw that at some point and you considered integrating this into the update check when performing a restart of the packages.

I'll check on this when I next power-up my old NAS and get back to you.

Yes, the current version available via 'opkg' is 1.1.1.g-1

I still hope to include an automated IPKG updater at some point. The "spanner-in-the-works" is the current need to downgrade Python 3 to suit Watcher3 if it's installed. The Python 3 packages must therefore be "held" at their existing versions while all other IPKGs are upgraded. I'll have a crack at this on the weekend.

In the meantime, you can manually upgrade your OpenSSL:

Code: Select all

opkg update
opkg upgrade libopenssl

... then restart any QPKGs using it.

[OneCD]

Some new packages are now available for those who wanted them: SickGear and Transmision.

[dirkonline]

I did see, that OpenSSL seems to be an older version "OpenSSL 1.1.1d 10 Sep 2019" is that correct? I remember we saw that at some point and you considered integrating this into the update check when performing a restart of the packages.

I'll check on this when I next power-up my old NAS and get back to you.

Yes, the current version available via 'opkg' is 1.1.1.g-1

I still hope to include an automated IPKG updater at some point. The "spanner-in-the-works" is the current need to downgrade Python 3 to suit Watcher3 if it's installed. The Python 3 packages must therefore be "held" at their existing versions while all other IPKGs are upgraded. I'll have a crack at this on the weekend.

In the meantime, you can manually upgrade your OpenSSL:

Code: Select all

opkg update
opkg upgrade libopenssl

... then restart any QPKGs using it.

Thanks for the help again I’ll upgrade later today. Maintaining specific versions sounds like a bummer because they all want something different. Hopefully the rest of the apps will follow through on using the latest version of Python soon.

[PaulM505]

Just upgraded from SickChill and SABNZBplus to Medusa and SABNZB and all seemed to be great but nzbtosickbeard.py fails now.(ImportError: No module named imghdr) with the output below. It's not a massive issue as I can get Medusa to post process (it works manually and there is a function to check automatically but I've not tested that yet - I'm a new Medusa user). But I wondered what I can do to get nzbtosickbeard working again, if anything?

Thanks in advance


/share/CACHEDEV1_DATA/Download/nzbToMedia/eol.py:147: LifetimeWarning: Python 2.7 is no longer supported.
warnings.warn(msg, LifetimeWarning)
fatal: not a git repository (or any parent up to mount point /share)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Changing to directory: /share/CACHEDEV1_DATA/Download/nzbToMedia

-- Cleaning bytecode --
WARNING: Automatic cleanup could not be executed.
If errors occur, manual cleanup may be required.
REASON : Error Code: 128

Directory is not a git repository

Force cleaning folder: libs
Skipping required subfolders [u'common', u'custom', u'py2', u'win']

Force cleaning folder: core
Skipping required subfolders [u'auto_process', u'extractor', u'plugins', u'utils']
Returning to directory: /share/CACHEDEV1_DATA/.qpkg/SABnzbd/SABnzbd

-- Cleanup finished --

Traceback (most recent call last):
File "/share/Download/nzbToMedia/nzbToSickBeard.py", line 270, in <module>
import nzbToMedia
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/nzbToMedia.py", line 730, in <module>
import core
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/core/__init__.py", line 52, in <module>
from core import logger, main_db, version_check, databases, transcoder
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/core/databases.py", line 11, in <module>
from core.utils import backup_versioned_file
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/core/utils/__init__.py", line 13, in <module>
from core.utils.common import clean_dir, flatten, get_dirs, process_dir
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/core/utils/common.py", line 14, in <module>
from core.utils.files import list_media_files, move_file
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/core/utils/files.py", line 14, in <module>
import beets.mediafile
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/libs/common/beets/mediafile.py", line 51, in <module>
import imghdr
ImportError: No module named imghdr

[OneCD]

Hi Paul and welcome to the forum.

But I wondered what I can do to get nzbtosickbeard working again, if anything?
...
ImportError: No module named imghdr

This was recently reported by @redmercury2006, but we haven't worked out a fix yet.

This happens if nzbToMedia tries to run in a Python environment that doesn't have the required modules. So, best-guess is that it's found the built-in QTS Python 2.7.5, which is quite basic.

We can either change the hashbang line for nzbToMedia.py to explicitly request Python 3 (which will cause problems when updating nzbToMedia from GitHub), or else we can ensure when nzbToMedia looks for 'Python', it finds the Entware Python 3 binary first.

Let's check your Entware Python binaries:

Code: Select all

ll /opt/bin/python*

Can you please post the result back here?

I'm hoping yours will look something like this:

Code: Select all

[~] # ll /opt/bin/python*
lrwxrwxrwx    1 admin    administ         9 Aug  6 08:45 /opt/bin/python -> python3.8*
lrwxrwxrwx    1 admin    administ        16 Aug  6 08:50 /opt/bin/python-config -> python3.8-config*
lrwxrwxrwx    1 admin    administ         9 Aug  6 08:45 /opt/bin/python3 -> python3.8*
lrwxrwxrwx    1 admin    administ        16 Aug  6 08:50 /opt/bin/python3-config -> python3.8-config*
-rwxr-xr-x    1 admin    administ     14.0k Jun 13 06:24 /opt/bin/python3.8*
-rwxr-xr-x    1 admin    administ      3.2k Jun 13 06:24 /opt/bin/python3.8-config*

[PaulM505]

Hi OneCD. Thanks for the reply. Yep, looks like the same issue, at least I know it's not just a problem at my end! I ran the command an pasted the output below. Looks similar to me (not an expert though ) although it is python 3.7 not 3.8 - is that an issue do you think?

Thanks again for the help, much appreciated.



[~] # ll /opt/bin/python*
lrwxrwxrwx 1 admin administrators 9 2020-08-05 12:29 /opt/bin/python3 -> python3.7*
-rwxr-xr-x 1 admin administrators 14K 2019-10-02 12:13 /opt/bin/python3.7*
-rwxr-xr-x 1 admin administrators 3.1K 2019-10-02 12:13 /opt/bin/python3.7-config*
lrwxrwxrwx 1 admin administrators 16 2020-08-05 12:29 /opt/bin/python3-config -> python3.7-config*

[OneCD]

Yup, Python 3.7 is fine. You'll typically have Python 3.7.4 if you've also installed the Watcher3 package via sherpa.

Let's try creating a symlink to Python 3, but we'll call it 'python':

Code: Select all

ln -s /opt/bin/python3 /opt/bin/python

... then try post-processing something with nzbToMedia. Please advise if this works or not.

[OneCD]

NEW FEATURE! Your sherpa debug log can now be posted online courtesy of https://termbin.com:

Code: Select all

./sherpa.sh --paste

A link will be generated to view this log online. Share it here if you need assistance.

Note: your log will be publicly accessible to anyone who knows the link details. It will be automatically deleted after 1 month. The debug log typically won't contain any personally-identifiable information. Your public and private IP addresses, email address and so-on are not recorded. However, there is a chance some info may leak if an app generates a backtrace and this is pasted into the sherpa debug log. So, check it first before pasting online.

[PaulM505]

Hmmm, different error this time - pasted below.

Did as you suggested and after I sorted out Medusa (weirdly for the second time it lost all my shows and I had to use the backup I wisely created after it did it the first time!). Then had to to add the relevant bits in autoprocessmedia.cfg that I'd forgotten in the first run through




fatal: not a git repository (or any parent up to mount point /share)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[07:51:42] [INFO]::MAIN: Loading config from [/share/CACHEDEV1_DATA/Download/nzbToMedia/autoProcessMedia.cfg]
[07:51:42] [INFO]::MAIN: Python v3.7 will reach end of life in 1054 days.
[07:51:42] [INFO]::MAIN: Checking database structure...
[07:51:42] [INFO]::MAIN: Checking if source needs an update
[07:51:43] [ERROR]::MAIN: Unknown current version number, don't know if we should update or not
[07:51:43] [INFO]::MAIN: nzbToMedia Version:12.1.05 Branch:master (Linux 3.4.6)
[07:51:43] [INFO]::MAIN: #########################################################
[07:51:43] [INFO]::MAIN: ## ..::[nzbToMedia.py]::.. ##
[07:51:43] [INFO]::MAIN: #########################################################
[07:51:43] [INFO]::MAIN: Script triggered from SABnzbd Version 3.0.0.
[07:51:43] [INFO]::MAIN: Auto-detected SECTION:SickBeard
[07:51:43] [INFO]::MAIN: Calling SickBeard:tv to post-process:Billions.S04E12.Extreme.Sandbox.1080p.AMZN.WEB-DL.DDP5.1.H.264-NTb
[07:51:44] [INFO]::MAIN: Attempting to auto-detect tv fork
[07:51:44] [ERROR]::MAIN: Failed to get JSON data from response
Changing to directory: /share/CACHEDEV1_DATA/Download/nzbToMedia

-- Cleaning bytecode --
WARNING: Automatic cleanup could not be executed.
If errors occur, manual cleanup may be required.
REASON : Error Code: 128

Directory is not a git repository

Force cleaning folder: libs
Skipping required subfolders ['common', 'custom', 'py2', 'win']

Force cleaning folder: core
Skipping required subfolders ['auto_process', 'extractor', 'plugins', 'utils']
Returning to directory: /share/CACHEDEV1_DATA/.qpkg/SABnzbd/SABnzbd

-- Cleanup finished --

Traceback (most recent call last):
File "/share/Download/nzbToMedia/nzbToSickBeard.py", line 273, in <module>
result = nzbToMedia.main(sys.argv, section)
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/nzbToMedia.py", line 968, in main
failure_link=os.environ['SAB_FAILURE_URL'])
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/nzbToMedia.py", line 853, in process
result = tv.process(section_name, input_directory, input_name, status, client_agent, download_id, input_category, failure_link)
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/core/auto_process/tv.py", line 56, in process
fork, fork_params = auto_fork(section, input_category)
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/core/forks.py", line 140, in auto_fork
rem_params, found = api_check(r, params, rem_params)
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/core/forks.py", line 19, in api_check
json_data = r.json()
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/libs/common/requests/models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "/__init__.py", line 348, in loads
File "/decoder.py", line 337, in decode
File "/decoder.py", line 355, in raw_decode
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)
Exception ignored in: <function PosixProcess.__del__ at 0x2177b60>
Traceback (most recent call last):
File "/share/CACHEDEV1_DATA/Download/nzbToMedia/core/utils/processes.py", line 95, in __del__
AttributeError: 'NoneType' object has no attribute 'isfile'

[niwa2]

SickChill seems to be broken as well now

when i try to start a fresh install of SickChill i get the following:

Code: Select all

= daemon is not active
= daemon is not active
* cloning 'SickChill' from remote repository: failed!
= ExecuteAndLog()
= output: "Cloning into '/share/CACHEDEV1_DATA/.qpkg/SickChill/SickChill'...
warning: Could not find remote branch py3-again to clone.
fatal: Remote branch py3-again not found in upstream origin
Cloning into '/share/CACHEDEV1_DATA/.qpkg/SickChill/SickChill'...
warning: redirecting to https://github.com/sickchill/sickchill.git/
warning: Could not find remote branch py3-again to clone.
fatal: Remote branch py3-again not found in upstream origin"
= result: [128]
/share/CACHEDEV1_DATA/.qpkg/SickChill/sickchill.sh: line 134: cd: /share/CACHEDEV1_DATA/.qpkg/SickChill/SickChill: No such file or directory

It seems we need to use a different branch now.
Apparently the branch py3-again has been merged into develop

I changed sickchill.sh to use the develop branch and it is up and running again.

[niwa2]

regarding the postprocessing and python3 issue.
I ran into the exact same issue.
Once i uninstalled OWatcher3 and reinstalled Sab and Sickchill postprocessing was working again.
So it seems that the downgrade of python3.8 to python3.7 for OWatcher has something to do with the postprocessing issue.

[niwa2]

By the way. Something is not right regarding the configuration of nzbToMedia.
i had to manually enable the TV category under [SickBeard] and change the port in autoProcessMedia.cfg to get the postprocessing of SabNzb --> SickChill working

[OneCD]

Hmmm, different error this time - pasted below.
...
AttributeError: 'NoneType' object has no attribute 'isfile'

I was seeing this exact same thing yesterday with SickChill and nzbToMedia.

I'll post back once I've tested a few things.