[QPKG] sherpa: a mini-package-manager (CLI) beta

[OneCD]

Code: Select all

[/share/CACHEDEV1_DATA/.qpkg] # /sbin/qpkg_service stop Entware; echo $?
Disable Entware/opkg
0
[/share/CACHEDEV1_DATA/.qpkg] # /sbin/qpkg_service disable Entware; echo $?
0
[/share/CACHEDEV1_DATA/.qpkg] # /sbin/qpkg_service enable Entware; echo $?
0
[/share/CACHEDEV1_DATA/.qpkg] # /sbin/qpkg_service start Entware; echo $?
Enable Entware/opkg
0
[/share/CACHEDEV1_DATA/.qpkg] # /sbin/qpkg_service start Entware; echo $?
Enable Entware/opkg
0
[/share/CACHEDEV1_DATA/.qpkg] # 

Hmm, that's not what I thought we'd see. The sherpa debug log indicated an exitcode of 1 when starting Entware. It appears running these manually results in 0.

Can you please check the Qboost app as per this post and enable the sherpa packages like 'sherpa', 'Entware', 'SickChill', etc... ?

[OneCD]

BTW: for anyone interested in reading-up on Qboost, here's the current promo-page:

[aslakege]

Hmm, that's not what I thought we'd see. The sherpa debug log indicated an exitcode of 1 when starting Entware. It appears running these manually results in 0.

Can you please check the Qboost app as per this post and enable the sherpa packages like 'sherpa', 'Entware', 'SickChill', etc... ?

All of the packages were set at "Always enabled", but I tried disabling them and then re-enabling them. I then ran

sherpa reinstall SickChill debug

Code: Select all

[/share/CACHEDEV1_DATA/.qpkg] # sherpa reinstall SickChill debug

dbug: (vv) $_object_session_debug_to_screen_flag_ [true] 
dbug: (vv) $_object_session_summary_flag_ [true] 
dbug: (>>) Session.ParseArguments() 
dbug: (<<) Session.ParseArguments()|0|25ms 
dbug: (vv) $_object_session_debug_to_file_flag_ [true] 
dbug: (II) ============================================================================================ 
dbug: (**)    SCRIPT:                 started: Tue Oct 20 08:00:23 CEST 2020 
dbug: (**)    SCRIPT:                 version: package: 200923, manager: 201019, loader 200923 
dbug: (**)    SCRIPT:                     PID: 15884 
dbug: (II) -------------------------------------------------------------------------------------------- 
dbug: (II) Markers: (**) detected, (II) information, (WW) warning, (EE) error, (LL) log file, 
dbug: (II) (==) processing, (--) done, (>>) f entry, (<<) f exit, (vv) variable name & value, 
dbug: (II) ($1) positional argument value 
dbug: (II) -------------------------------------------------------------------------------------------- 
dbug: (>>) QPKGs.DepAndIndep.Build() 
dbug: (<<) QPKGs.DepAndIndep.Build()|0|28ms 
dbug: (>>) QPKGs.InstallationState.Build() 
dbug: (<<) QPKGs.InstallationState.Build()|0|180ms 
dbug: (>>) QPKGs.Upgradable.Build() 
dbug: (<<) QPKGs.Upgradable.Build()|0|50ms 
dbug: (<<) Session.Init()|0|1,597ms 
dbug: (>>) Session.Validate() 
dbug: (II) -------------------------------------------------------------------------------------------- 
dbug: (**)  HARDWARE:                   model: TS-451 
dbug: (**)  HARDWARE:                     RAM: 8,049,412 kB 
dbug: (**)  FIRMWARE:        firmware version: 4.5.1 
dbug: (**)  FIRMWARE:          firmware build: 20201015 
dbug: (**)  FIRMWARE:                  kernel: 4.14.24-qnap x86_64 
dbug: (**) USERSPACE:               OS uptime: 1 day 
dbug: (**) USERSPACE:             system load: 1 min=3.31, 5 min=3.97, 15 min=3.32 
dbug: (**) USERSPACE:                   $USER: admin 
dbug: (**) USERSPACE:                   $EUID: 0 
dbug: (**) USERSPACE:                    BASH: GNU bash, version 3.2.57(1)-release (x86_64-QNAP-linux-gnu) 
dbug: (**) USERSPACE:          default volume: /share/CACHEDEV1_DATA 
dbug: (**) USERSPACE:                    /opt: /share/CACHEDEV1_DATA/.qpkg/Entware 
dbug: (**) USERSPACE:                   $PATH: /opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/u... 
dbug: (**) USERSPACE:          'python2' path: /usr/local/bin/python2 
dbug: (**) USERSPACE:       'python2' version: Python 2.7.5 
dbug: (**) USERSPACE:          'python3' path: /opt/bin/python3 
dbug: (**) USERSPACE:       'python3' version: Python 3.8.5 
dbug: (**) USERSPACE:           'python' path: /opt/bin/python 
dbug: (**) USERSPACE:        'python' version: Python 3.8.5 
dbug: (**) USERSPACE: unparsed user arguments: 'reinstall SickChill debug' 
dbug: (**)    SCRIPT:               logs path: /share/CACHEDEV1_DATA/.qpkg/sherpa/logs 
dbug: (**)    SCRIPT:               work path: /share/CACHEDEV1_DATA/.qpkg/sherpa/cache 
dbug: (**)    SCRIPT:   object reference hash: a7804e08f41d962a08fab384c557f865 
dbug: (**)      QPKG:       Entware installer: std 
dbug: (**)      QPKG:                    arch: x64 
dbug: (**)      QPKG:   upgradable package(s): 
dbug: (II) -------------------------------------------------------------------------------------------- 
dbug: (>>) QPKGs.Assignment.Check() 
dbug: (>>) QPKGs.DepAndIndep.Build() 
dbug: (<<) QPKGs.DepAndIndep.Build()|0|39ms 
dbug: (>>) QPKGs.InstallationState.Build() 
dbug: (<<) QPKGs.InstallationState.Build()|0|180ms 
dbug: (>>) QPKGs.Upgradable.Build() 
dbug: (<<) QPKGs.Upgradable.Build()|0|50ms 
dbug: (>>) QPKGs.Assignment.List() 
dbug: (**)      QPKG:             to download: SickChill 
dbug: (**)      QPKG:               to backup: 
dbug: (**)      QPKG:            to uninstall: 
dbug: (**)      QPKG:                 to stop: 
dbug: (**)      QPKG:        to force-upgrade: 
dbug: (**)      QPKG:              to upgrade: 
dbug: (**)      QPKG:            to reinstall: SickChill 
dbug: (**)      QPKG:              to install: 
dbug: (**)      QPKG:              to restore: 
dbug: (**)      QPKG:                to start: Entware 
dbug: (**)      QPKG:              to restart: 
dbug: (<<) QPKGs.Assignment.List()|0|339ms 
dbug: (<<) QPKGs.Assignment.Check()|0|774ms 
dbug: (II) -------------------------------------------------------------------------------------------- 
dbug: (<<) Session.Validate()|0|1,654ms 
dbug: (>>) Packages.Download() 
dbug: (>>) QPKG.Download() 
dbug: (II) local package (SickChill_200916.qpkg) checksum correct, so skipping download 
dbug: (<<) QPKG.Download()|0|129ms 
dbug: (<<) Packages.Download()|0|174ms 
dbug: (>>) Packages.Backup() 
dbug: (<<) Packages.Backup()|0|40ms 
dbug: (>>) Packages.Stop() 
dbug: (<<) Packages.Stop()|0|21ms 
dbug: (>>) Packages.Uninstall() 
dbug: (>>) IPKGs.Uninstall() 
dbug: (>>) IPKGs.Uninstall.Batch() 
dbug: (>>) CalcAllIPKGDepsToUninstall() 
dbug: (II) IPKGs requested:  
dbug: (==) excluding IPKGs not installed ... 
dbug: (--) complete 
dbug: (II) IPKGs to uninstall:  
dbug: (<<) CalcAllIPKGDepsToUninstall()|0|434ms 
dbug: (<<) IPKGs.Uninstall.Batch()|0|595ms 
dbug: (<<) IPKGs.Uninstall()|0|1,013ms 
dbug: (<<) Packages.Uninstall()|0|1,245ms 
dbug: (>>) Packages.Reinstall.Independents() 
dbug: (<<) Packages.Reinstall.Independents()|0|94ms 
dbug: (>>) Packages.Install.Independents() 
dbug: (<<) Packages.Install.Independents()|0|23ms 
dbug: (>>) Packages.Start.Independents() 
dbug: (>>) QPKG.Enable() 
dbug: (<<) QPKG.Enable()|0|103ms 
dbug: (>>) QPKG.Start() 
proc: starting 'Entware' ... 
dbug: (>>) RunThisAndLogResultsRealtime() 
/share/CACHEDEV1_DATA/.qpkg/sherpa/cache/sherpa.manager.sh: line 4009: /dev/fd/62: No such file or directory
dbug: (LL) -------------------------------------------------------------------------------------------- 
dbug: (LL) adding external log to main log ... 
dbug: (vv) $_object_session_debug_to_screen_flag_ [true] 
dbug: (LL) -------------------------------------------------------------------------------------------- 
dbug: (<<) RunThisAndLogResultsRealtime()|0|684ms 
warn: Could not start 'Entware' [1] 
dbug: (<<) QPKG.Start()|0|918ms   
dbug: (<<) Packages.Start.Independents()|0|1,299ms 
dbug: (>>) Packages.Start.Addons() 
dbug: (vv) $_object_session_ipkgs_install_flag_ [true] 
dbug: (--) adjusted $PATH for Entware 
dbug: (vv) $PATH [/opt/bin:/opt/sbin::/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin/X11:/usr/local/sbin:/usr/local/bin] 
dbug: (II) patch: do the "opt shuffle" - already done 
proc: updating 'Entware' package list ... 
dbug: (>>) RunThisAndLogResultsRealtime()      
/share/CACHEDEV1_DATA/.qpkg/sherpa/cache/sherpa.manager.sh: line 4009: /dev/fd/62: No such file or directory
dbug: (LL) -------------------------------------------------------------------------------------------- 
dbug: (LL) adding external log to main log ... 
dbug: (vv) $_object_session_debug_to_screen_flag_ [true] 
dbug: (LL) -------------------------------------------------------------------------------------------- 
dbug: (<<) RunThisAndLogResultsRealtime()|0|187ms 
warn: Unable to update 'Entware' package list [1] 
dbug: (>>) IPKGs.Install()                     
dbug: (>>) IPKGs.Upgrade.Batch()               
dbug: (<<) IPKGs.Upgrade.Batch()|0|460ms       
dbug: (>>) IPKGs.Install.Batch()               
dbug: (>>) CalcAllIPKGDepsToInstall()          
proc: calculating IPKGs required ... 
dbug: (II) IPKGs requested: ca-certificates findutils gcc git git-http less nano python3-dev python3-pip python3-setuptools sed 
dbug: (>>) IPKGs.Archive.Open()           
dbug: (>>) RunThisAndLogResultsRealtime() 
/share/CACHEDEV1_DATA/.qpkg/sherpa/cache/sherpa.manager.sh: line 4009: /dev/fd/62: No such file or directory
dbug: (LL) -------------------------------------------------------------------------------------------- 
dbug: (LL) adding external log to main log ... 
dbug: (vv) $_object_session_debug_to_screen_flag_ [true] 
dbug: (LL) -------------------------------------------------------------------------------------------- 
dbug: (<<) RunThisAndLogResultsRealtime()|0|249ms 
fail: Could not open the IPKG list file    
dbug: (vv) $_object_session_skippackageprocessing_flag_ [true] 
dbug: (vv) $_script_error_flag [true]     
dbug: (<<) IPKGs.Archive.Open()|0|361ms   
dbug: (<<) CalcAllIPKGDepsToInstall()|0|462ms 
dbug: (<<) IPKGs.Install()|0|1,273ms      
dbug: (<<) Packages.Start.Addons()|0|1,717ms 
dbug: (II) -------------------------------------------------------------------------------------------- 
dbug: (**)    SCRIPT:                finished: Tue Oct 20 08:00:31 CEST 2020 
dbug: (**)    SCRIPT:            elapsed time: 00h:00m:08s 
dbug: (II) ============================================================================================ 

As you can see it fails to start Enware [1]. I then ran

opkg

and then got the usual prompts:

Code: Select all

[/share/CACHEDEV1_DATA/.qpkg] # opkg
opkg must have one sub-command argument
usage: opkg [options...] sub-command [arguments...]
where sub-command is one of:

Package Manipulation:
	update			Update list of available packages
	upgrade <pkgs>		Upgrade packages
	install <pkgs>		Install package(s)
	configure <pkgs>	Configure unpacked package(s)
	remove <pkgs|regexp>	Remove package(s)
	flag <flag> <pkgs>	Flag package(s)
	 <flag>=hold|noprune|user|ok|installed|unpacked (one per invocation)

Informational Commands:
	list			List available packages
	list-installed		List installed packages
	list-upgradable		List installed and upgradable packages
	list-changed-conffiles	List user modified configuration files
	files <pkg>		List files belonging to <pkg>
	search <file|regexp>	List package providing <file>
	find <regexp>		List packages whose name or description matches <regexp>
	info [pkg|regexp]	Display all info for <pkg>
	status [pkg|regexp]	Display all status for <pkg>
	download <pkg>		Download <pkg> to current directory
	compare-versions <v1> <op> <v2>
	                    compare versions using <= < > >= = << >>
	print-architecture	List installable package architectures
	depends [-A] [pkgname|pat]+
	whatdepends [-A] [pkgname|pat]+
	whatdependsrec [-A] [pkgname|pat]+
	whatrecommends[-A] [pkgname|pat]+
	whatsuggests[-A] [pkgname|pat]+
	whatprovides [-A] [pkgname|pat]+
	whatconflicts [-A] [pkgname|pat]+
	whatreplaces [-A] [pkgname|pat]+

Options:
	-A			Query all packages not just those installed
	-V[<level>]		Set verbosity level to <level>.
	--verbosity[=<level>]	Verbosity levels:
					0 errors only
					1 normal messages (default)
					2 informative messages
					3 debug
					4 debug level 2
	-f <conf_file>		Use <conf_file> as the opkg configuration file
	--conf <conf_file>
	--cache <directory>	Use a package cache
	-d <dest_name>		Use <dest_name> as the the root directory for
	--dest <dest_name>	package installation, removal, upgrading.
				<dest_name> should be a defined dest name from
				the configuration file, (but can also be a
				directory name in a pinch).
	-o <dir>		Use <dir> as the root directory for
	--offline-root <dir>	offline installation of packages.
	--add-arch <arch>:<prio>	Register architecture with given priority
	--add-dest <name>:<path>	Register destination with given path

Force Options:
	--force-depends		Install/remove despite failed dependencies
	--force-maintainer	Overwrite preexisting config files
	--force-reinstall	Reinstall package(s)
	--force-overwrite	Overwrite files from other package(s)
	--force-downgrade	Allow opkg to downgrade packages
	--force-space		Disable free space checks
	--force-postinstall	Run postinstall scripts even in offline mode
	--force-remove	Remove package even if prerm script fails
	--force-checksum	Don't fail on checksum mismatches
	--no-check-certificate Don't validate SSL certificates
	--noaction		No action -- test only
	--download-only	No action -- download only
	--nodeps		Do not follow dependencies
	--nocase		Perform case insensitive pattern matching
	--size			Print package size when listing available packages
	--force-removal-of-dependent-packages
				Remove package and all dependencies
	--autoremove		Remove packages that were installed
				automatically to satisfy dependencies
	-t			Specify tmp-dir.
	--tmp-dir		Specify tmp-dir.
	-l			Specify lists-dir.
	--lists-dir		Specify lists-dir.

 regexp could be something like 'pkgname*' '*file*' or similar
 e.g. opkg info 'libstd*' or opkg search '*libop*' or opkg remove 'libncur*'
[/share/CACHEDEV1_DATA/.qpkg] # 

I think I will try to downgrade the firmware (if possible), and report back.

Sincerely,
Aslak

[craZyC]

Hi OneCD..it has been a while and I see a great deal of modifications. Top Job..
Question ( I searched but couldn't find an answer.)
I am pretty sure I am still running your sherpa.sh script as I don't recall running the sherpa.qpkg, last time I changed anything was 20/08/2020 when you moved from SabNzbPlus to Sabnzbd.
Do you have anywhere a process/guide to upgrade to your new qpkg version. Do we just install the .qpkg and delete the sherpa.sh script. Anything to delete or move etc ???
More than happy to be flamed if it is somewhere obvious

[splatmannl]

Hi OneCD,

I now think the github authentication issue was not related to the Sherpa/Sickchill installation issue.

After trying a lot of different ways to install Sickchill, I ultimately succeeded by doing the following:
- I tried to install Sikchill via Sherpa; the installation hangs as before. I quit my SSH terminal and the restart it.
- I notice Sickchill is not installed correctly (there is a Sickchill folder, but its empty except for sickchill.sh and the "config" folder; a "sickchill" folder which shoudl be there is missing)
- I then used the "git clone https://github.com/SickChill/SickChill" command. This creates a sickchill folder

In the end sickchill now runs, so I am happy. But I was not able to install it using (only) Sherpa, so I think you do need some kind of fix.
I'm not very technical myself, so I'm affraid I can't help you out further, but I hope you will find a solution!

Best regards,
Splatman

Hi @Splatman.

On October 18th, 2020 at 21:14 (UTC) you used a password to access an endpoint through the GitHub API using curl/7.43.0:

https://api.github.com/user

Basic authentication using a password to the API is deprecated and will soon no longer work. Visit https://developer.github.com/changes/20 ... word-auth/ for more information around suggested workarounds and removal dates.

That's interesting. None of the sherpa apps use authentication when cloning or pulling from GitHub. Here are the relevant lines in each of the service scripts:

Code: Select all

if [[ ! -d $QPKG_GIT_PATH/.git ]]; then
    ExecuteAndLog "clone $(FormatAsPackageName "$1") from remote repository" "$GIT_CMD clone --branch $3 $DEPTH -c advice.detachedHead=false $GIT_HTTPS_URL $QPKG_GIT_PATH || $GIT_CMD clone --branch $3 $DEPTH -c advice.detachedHead=false $GIT_HTTP_URL $QPKG_GIT_PATH"
else
    ExecuteAndLog "update $(FormatAsPackageName "$1") from remote repository" "$GIT_CMD -C $QPKG_GIT_PATH reset --hard; $GIT_CMD -C $QPKG_GIT_PATH pull"
fi

Hmm... do you push any commits to GitHub from your NAS?

Could this have something to do with the Sickchill installation issue?

Could be. I need someone with access to QTS 4.5.1 to do some peeking and work out what's going on there.

Latest advice I've seen from @alfredborges is that the QNAP Qboost app needs to be reconfigured to allow these 3rd party apps.

[aslakege]

I think I will try to downgrade the firmware (if possible), and report back.

Hi OneCD,

Downgrading worked.

I am now running 4.4.3.1439, and sherpa installed everything with no glitches whatsoever. Working like a charm.

I guess I'll wait on upgrading a while. Unless you need someone to test potential solutions on newer firmware - I can totally reapply the latest at any time.

Sincerely,
Aslak

[OneCD]

All of the packages were set at "Always enabled", but I tried disabling them and then re-enabling them. I then ran

sherpa reinstall SickChill debug

Code: Select all

/share/CACHEDEV1_DATA/.qpkg/sherpa/cache/sherpa.manager.sh: line 4009: /dev/fd/62: No such file or directory

This error is something I can't do much about. It means the current QTS BASH isn't handling file-descriptors properly.

It occurred several times in your last sherpa session. I suspect this is another breakage in QTS 4.5.1

I am now running 4.4.3.1439, and sherpa installed everything with no glitches whatsoever. Working like a charm.

No worries Aslak, downgrading looks like a good idea. Hopefully we can get sherpa working on the new QTS versions. I don't have access to it myself so it's a bit difficult to do.

[OneCD]

Do you have anywhere a process/guide to upgrade to your new qpkg version. Do we just install the .qpkg and delete the sherpa.sh script. Anything to delete or move etc ???

I've just the thing: https://github.com/OneCDOnly/sherpa/wik ... ious-users

However, be aware QTS 4.5.1 is preventing unsigned 3rd party apps from running at the moment, so if you're on a QTS version earlier than 4.5.1, it might be a good idea to stick with it.

[OneCD]

In the end sickchill now runs, so I am happy. But I was not able to install it using (only) Sherpa, so I think you do need some kind of fix.
I'm not very technical myself, so I'm affraid I can't help you out further, but I hope you will find a solution!

No worries Splatman, I'm hopeful we'll get sherpa working again once the issues in QTS 4.5.1 issues are sorted.

In the meantime, temporarily downgrading QTS is another solution.

[craZyC]

I've just the thing: https://github.com/OneCDOnly/sherpa/wik ... ious-users

However, be aware QTS 4.5.1 is preventing unsigned 3rd party apps from running at the moment, so if you're on a QTS version earlier than 4.5.1, it might be a good idea to stick with it.

See Blind as a Bat...
So just delete the sherp.sh, got it I was just a little concerned as I thought I read the installation directories where different between the packages.
My current firmware version is 4.4.3.1439 which is latest official for my model so I will give it a go , thanks again..Edit that latest firmware down so will wait for your update before trying

[dirkonline]

@OneCD the moment I step out for a few weeks due to the new job you go ahead and make it all brand new and shiny. Thank you for all the work!

Cheers Dirk. Good to see you're still with us.

BTW: this project got so involved, I ended-up creating a wiki. It's quite new and needs a bit more work - are you still interested in managing the wiki? It's OK to say "no".

Sure! Just swamped with my new job currently., but I would happily work on that with you.

[dirkonline]

Oh dear, I was just about to upgrade QTS to 4.5.1 when I read the changelog and...

Code: Select all

Known Issues
[b]After a system update to QTS 4.5.1, users cannot run applications that lack digital signatures[/b].

Thanks QNAP. Now hoping they did not break anything significant in the version I am currently running.

[craZyC]

After a system update to QTS 4.5.1, users cannot run applications that lack digital signatures.
I actually do not think that is 100% the case, I may be different as i have the original sherpa script..
noticed that Sab, Par 2, Sort my QPKG and Entware all showed an exclamation as no digital signature with the only option to remove. I checked and all apps were actually running and working fine.
The work around or way to check that I found after a google search was to was to first go into the app centre and settings and ensure Allow applications without a digital signature to be installed. Step 2 is to go into Qboost and select the elliptical at the bottom right which is app management and ensure your app is enabled. For me everything was running fine.
I hope this helps until they fix it

[OneCD]

The work around or way to check that I found after a google search was to was to first go into the app centre and settings and ensure Allow applications without a digital signature to be installed. Step 2 is to go into Qboost and select the elliptical at the bottom right which is app management and ensure your app is enabled. For me everything was running fine.

Awesome! Good work!

[mrx9qn1]

Hi - thanks for this. I have version 3 in the app centre, but version 2 is no longer showing but it is still running so not sure how to uninstall.