[QPKG] sherpa mini-package-manager (CLI) beta

This is the best place for community developers to publish their genius work. Your Apps enrich the QNAP Turbo NAS.
Post Reply
User avatar
OneCD
Guru
Posts: 12039
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by OneCD »

Excellent, good work guys! Image

Now, I just need to understand what's required so I can fix sherpa. :DD

@zyxmon, I switched to installing the 'python3-cryptography' and 'python3-cffi' IPKGs, so I don't understand why PIP was still trying to compile CFFI on x19? It doesn't do that on my a64?

How should I fix this? What does the 'include' archive downloaded earlier do?

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage
zyxmon
Experience counts
Posts: 1001
Joined: Mon Oct 27, 2014 2:27 pm
Location: Moscow, Russia
Contact:

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by zyxmon »

OneCD wrote: Thu Apr 22, 2021 3:12 am ... so I don't understand why PIP was still trying to compile CFFI on x19? It doesn't do that on my a64?

How should I fix this? What does the 'include' archive downloaded earlier do?
I do not know how pip works and handles dependencies. One thing to note - sometimes there are missing symlinks to so - openwrt was designed to run binaries and not for native compilation.
The second one - headers were not installed for x19. Are they installed for other architectures? Headers are usually needed for compilation. Some headers are installed with gcc package.
The problems solved today are missing ffi.h header and missing libffi.so (symlink, not the library). Similar problems may occur for other architectures and pip install commands.
zyxmon
Experience counts
Posts: 1001
Joined: Mon Oct 27, 2014 2:27 pm
Location: Moscow, Russia
Contact:

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by zyxmon »

Code: Select all

pip install [some package] --no-deps
can be used. (may be)
zyxmon
Experience counts
Posts: 1001
Joined: Mon Oct 27, 2014 2:27 pm
Location: Moscow, Russia
Contact:

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by zyxmon »

One more thing the new cryptography package needs rust that is missing.
User avatar
OneCD
Guru
Posts: 12039
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by OneCD »

zyxmon wrote: Thu Apr 22, 2021 3:27 am One thing to note - sometimes there are missing symlinks to so - openwrt was designed to run binaries and not for native compilation.
OK.
zyxmon wrote: Thu Apr 22, 2021 3:27 am The second one - headers were not installed for x19. Are they installed for other architectures? Headers are usually needed for compilation. Some headers are installed with gcc package.
I'm not sure, I think I previously resolved the header issue by installing 'python3-dev' (which still occurs today). sherpa no-longer downloads anything other than IPKGs for Entware - so no direct files anymore. It used to do this when I was downgrading the Python version some time ago, but not any more.
zyxmon wrote: Thu Apr 22, 2021 3:27 am The problems solved today are missing ffi.h header and missing libffi.so (symlink, not the library). Similar problems may occur for other architectures and pip install commands.
I think this is going to require some experimentation to get right. :DD
zyxmon wrote: Thu Apr 22, 2021 3:42 am One more thing the new cryptography package needs rust that is missing.
Ah, thank you. Looks like it will cause fewer problems if we stick with the cryptography IPKG module instead of installing it via 'pip'.

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage
zyxmon
Experience counts
Posts: 1001
Joined: Mon Oct 27, 2014 2:27 pm
Location: Moscow, Russia
Contact:

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by zyxmon »

IMHO the problem is - pip ignores already installed packages for x19 reinstalls them and failes because of missing *.h or *.so files.
This may help -

Code: Select all

PIP_IGNORE_INSTALLED=0 pip install [some package]
It can be used for all architectures.
User avatar
OneCD
Guru
Posts: 12039
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by OneCD »

I think you’re right. Thanks @zyxmon. :D

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage
User avatar
OneCD
Guru
Posts: 12039
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by OneCD »

Setup my "new" old ARMv5 NAS this morning and first thing I installed was SABnzbd via sherpa: no problems, no compilation errors - just my luck! :lol:

Anyway, based on @zyxmon's great suggestions, I'll start modifying sherpa so it will (hopefully) rely a lot less on PIP module compilation. At the very least, it should make for faster execution. ;)

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage
Brutalis
Starting out
Posts: 27
Joined: Thu Mar 15, 2012 6:33 pm

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by Brutalis »

OneCD wrote: Thu Apr 08, 2021 9:09 am
Brutalis wrote: Wed Apr 07, 2021 5:13 pm Thanks for all your help, but... again an error:
Apologies: I've still a long way to go with regard-to debugging these Python module installs. :oops:

I can't make this error occur on my test NAS (TS-230 with ARMv8), no-matter what I do. I suspect it's a compiling error only occurring on ARMv5.

I wonder if installing the 'wheel' pip module is needed on your arch to get 'cffi' installed?

After a lot of installs/reinstalls and testing on my own NAS this morning, here's the process I'm hoping you'll be willing to run through to confirm this. There are 2 methods: if the first fails, please run the second.

Method 1
  • The quick-and-dirty way, run this on your existing config:

    Code: Select all

    /opt/bin/pip3 install wheel cffi
Method 2
  • Remove the existing Entware installation. We're going to replace it with a vanilla Entware instead:

    Code: Select all

    sherpa remove ew
    cd /share/Public
    wget http://bin.entware.net/other/Entware_1.03std.qpkg
    sh Entware_1.03std.qpkg
    When Entware has finished installing, please logout your terminal session then back in again (to refresh $PATH).
  • Then run the following commands. These will install a basic Python module set, upgrade 'pip', then install 'wheel' (which I'm hoping will be the only required fix here) and 'apprise' (so 'cffi' gets pulled-in as a dependency):

    Code: Select all

    opkg install python3-pip python3-setuptools python3-dev
    /opt/bin/python3 -m pip install --upgrade pip
    /opt/bin/pip3 install wheel apprise
Now check the 'pip' module list:

Code: Select all

/opt/bin/pip3 list
I finally had time to try it out again. I just reinstalled ew, sabnzbd... and that worked :shock: . I don't know why it works now, but the only thing I did, I installed a update for my Qnap yesterday.
So I didn't try out the above commands... but thanks for all your help!

Code: Select all

sherpa reinstall ew
sherpa reinstall sabnzbd
browellm
Know my way around
Posts: 120
Joined: Wed Apr 08, 2015 11:42 pm

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by browellm »

Getting an error starting OMedusa using the latest firmware 4.5.3.1652

Code: Select all

>>>>>>>>>> datetime:'Thu Apr 29 23:54:22 BST 2021',request:'starting',QPKG:'210325',app:'0.5.11' <<<<<<<<<<
daemon: NOT active
file (/opt/bin/git): exists
update 'OMedusa' from remote repository: OK
current git branch: master
file (/opt/bin/python3): exists
load UI ports from application: OK
start daemon: failed!
= ExecuteAndLog()
output: "ffprobe: error while loading shared libraries: libx264.so.161: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/start.py", line 6, in <module>
    from medusa.__main__ import main
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/medusa/__init__.py", line 24, in <module>
    initialize()
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/medusa/init/__init__.py", line 39, in initialize
    _configure_knowit()
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/medusa/init/__init__.py", line 240, in _configure_knowit
    api.initialize({'mediainfo': suggested_path})
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/ext/knowit/api.py", line 35, in initialize
    available_providers[name] = provider_cls(config, context.get(name) or config.general.get(name))
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/ext/knowit/providers/ffmpeg.py", line 218, in __init__
    self.executor = FFmpegExecutor.get_executor_instance(suggested_path)
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/ext/knowit/providers/ffmpeg.py", line 108, in get_executor_instance
    executor = exec_cls.create(os_family, suggested_path)
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/ext/knowit/providers/ffmpeg.py", line 131, in create
    output = ensure_text(check_output([candidate, '-version']))
  File "/opt/lib/python3.9/subprocess.py", line 424, in check_output
  File "/opt/lib/python3.9/subprocess.py", line 528, in run
subprocess.CalledProcessError: Command '['ffprobe', '-version']' returned non-zero exit status 127."
result: [1]
TS-451
zyxmon
Experience counts
Posts: 1001
Joined: Mon Oct 27, 2014 2:27 pm
Location: Moscow, Russia
Contact:

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by zyxmon »

The problem is probably with libx264 package. Here are possible solutions - https://github.com/Entware/Entware/issues/644
User avatar
OneCD
Guru
Posts: 12039
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by OneCD »

zyxmon wrote: Fri Apr 30, 2021 2:37 pm The problem is probably with libx264 package. Here are possible solutions - https://github.com/Entware/Entware/issues/644
Thanks mate! :D

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage
User avatar
dirkonline
Easy as a breeze
Posts: 305
Joined: Mon Oct 18, 2010 4:08 pm

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by dirkonline »

With the latest QTS Update 4.5.3.165 , OMedusa no longer starts
Is that an issue for anybody else? Ideas? I wish they would stop breaking things all the time, sigh :(
Reiinstall did not fix the issue

Code: Select all

>>>>>>>>>> datetime:'Fri Apr 30 15:51:05 CEST 2021',request:'starting',QPKG:'210325',app:'0.5.11' <<<<<<<<<<
daemon: NOT active
file (/opt/bin/git): exists
update 'OMedusa' from remote repository: OK
current git branch: master
file (/opt/bin/python3): exists
load UI ports from application: OK
start daemon: failed!
= ExecuteAndLog()
output: "ffprobe: error while loading shared libraries: libx264.so.161: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/start.py", line 6, in <module>
    from medusa.__main__ import main
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/medusa/__init__.py", line 24, in <module>
    initialize()
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/medusa/init/__init__.py", line 39, in initialize
    _configure_knowit()
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/medusa/init/__init__.py", line 240, in _configure_knowit
    api.initialize({'mediainfo': suggested_path})
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/ext/knowit/api.py", line 35, in initialize
    available_providers[name] = provider_cls(config, context.get(name) or config.general.get(name))
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/ext/knowit/providers/ffmpeg.py", line 218, in __init__
    self.executor = FFmpegExecutor.get_executor_instance(suggested_path)
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/ext/knowit/providers/ffmpeg.py", line 108, in get_executor_instance
    executor = exec_cls.create(os_family, suggested_path)
  File "/share/CACHEDEV1_DATA/.qpkg/OMedusa/OMedusa/ext/knowit/providers/ffmpeg.py", line 131, in create
    output = ensure_text(check_output([candidate, '-version']))
  File "/opt/lib/python3.9/subprocess.py", line 424, in check_output
  File "/opt/lib/python3.9/subprocess.py", line 528, in run
subprocess.CalledProcessError: Command '['ffprobe', '-version']' returned non-zero exit status 127."
result: [1]
zyxmon
Experience counts
Posts: 1001
Joined: Mon Oct 27, 2014 2:27 pm
Location: Moscow, Russia
Contact:

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by zyxmon »

dirkonline wrote: Fri Apr 30, 2021 9:53 pm With the latest QTS Update 4.5.3.165 , OMedusa no longer starts
Is that an issue for anybody else? Ideas? I wish they would stop breaking things all the time, sigh :(
Reiinstall did not fix the issue
It is probably the problem with libx256 entware package upgrade (from the opkg point of view - it is a downgrade). opkg cannot handle it. See my previous post with the probable solutions.
I do not know sherpa, if there is an option to reinstall entware - it will also fix the problem.
User avatar
OneCD
Guru
Posts: 12039
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: [QPKG] sherpa mini-package-manager (CLI)

Post by OneCD »

zyxmon wrote: Sat May 01, 2021 3:54 am
dirkonline wrote: Fri Apr 30, 2021 9:53 pm With the latest QTS Update 4.5.3.165 , OMedusa no longer starts
Is that an issue for anybody else? Ideas? I wish they would stop breaking things all the time, sigh :(
Reiinstall did not fix the issue
It is probably the problem with libx256 entware package upgrade (from the opkg point of view - it is a downgrade). opkg cannot handle it. See my previous post with the probable solutions.
I've been testing a fresh install with only the Entware and OMedusa QPKGs installed via sherpa.

'ffmpeg' isn't installed as it's not required by these packages. Which means 'libx264' also isn't installed. Medusa uses the 'mediainfo' package to extract video dimensions instead of 'ffmpeg'. edit: .. but it will use 'ffmpeg' if 'mediainfo' isn't installed.

I think 'ffmpeg' is leftover from a previous sherpa version.

Might be better to remove 'libx264' and dependent packages like 'ffmpeg' as they're not needed anyway:

Code: Select all

opkg remove libx264 --force-removal-of-dependent-packages
@Dirk, apologies for the trouble. Most of the time, these breakages are because I've been refactoring the sherpa code to make it "better". ;)

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage
Post Reply

Return to “Community Apps”