ipgk problem

Introduce yourself to us and other members here, or share your own product reviews, suggestions, and tips and tricks of using QNAP products.
Post Reply
Morkeleb
Starting out
Posts: 17
Joined: Sun Dec 26, 2010 4:16 am

ipgk problem

Post by Morkeleb » Tue Apr 12, 2011 5:55 am

Hi.

I'm having problem using various applications installed with ipgk properly.

Just running ipkg causes:
ipkg: command not found

I have to run it like this: /share/MD0_DATA/.qpkg/Optware/bin/ipkg

Any other program like gcc also has to be executed with the full path inside /share/MD0_DATA/.qpkg/Optware
For simple programs that may be ok, but when I want to execute complex scripts like configuring and making applications from source code that does not suffice since ./configure does not even find gcc properly.

What can I do that all programs i installed using ipkg that then are stored in c are found like they reside for instance in /bin (but this includes all other installed software and libraryies in /share/MD0_DATA/.qpkg/Optware/bin, /share/MD0_DATA/.qpkg/Optware/lib, etc).

Many people explaining how to install or compile something just mention ipkg install commands and then immediately use them directly, but they never mention how to configure the system so that this works.

Btw, i'm using a qnap 459 pro.

Please help!

User avatar
Don
Guru
Posts: 12000
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: ipgk problem

Post by Don » Tue Apr 12, 2011 10:30 am

It sounds like the ipkg QPKG is not enabled. Enabling it changes to path statement to include the optware bin path.
Read the Online Manuals and use the forum search feature before posting.

It is a recommended to use RAID and have external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

Submit bugs and feature requests to QNAP via their Helpdesk app.

NAS: TVS-882BR | F/W: 4.3.6.0895 | 40GB | 2 x M.2 SATA RAID 1 (System/VMs) | 4 x M.2 NMVe QM2-4P-384A RAID 5 (Cache) | 5 x 4TB HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-663 | F/W: 4.4.2.1320 | 16GB | 2 x M.2 NMVe QM2-2P RAID 1 (Cache) | 4 x 4TB RAID 5
Apps: Boinc, Squid, DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS, Entware, DLstation, +others

Morkeleb
Starting out
Posts: 17
Joined: Sun Dec 26, 2010 4:16 am

Re: ipgk problem

Post by Morkeleb » Tue Apr 12, 2011 5:24 pm

If you mean enabling it in the administration website under
Application Servers->QPKG Plugins->QPKG Installed->Optware->Enable button, this was enabled all the time, i have the latest version

I researched further and found this:
http://wiki.qnap.com/wiki/Install_Optwa ... on_startup
I was not sure if my firmare has this problem, but I applied the patches and restarted.

However I found that echo $PATH still provides:
/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin/X11:/usr/local/sbin

So no /opt/bin at all.
However /etc/profile does contain the appropriate modifications.
Even when i changed the the access so i myself could execute it (chmod 775 /etc/profile)
and executed it, the last lines apparently where not executed.

This is the content of my /etc/profile:

Code: Select all

# ~/.bashrc: executed by bash(1) for non-login interactive shells.

export PATH=\
/bin:\
/sbin:\
/usr/bin:\
/usr/sbin:\
/usr/bin/X11:\
/usr/local/sbin

# If running interactively, then:
if [ "$PS1" ]; then

    if [ "$BASH" ]; then
   export PS1="[\u@\h \W]\\$ "
   alias ll='/bin/ls -laFh'
   alias ls='/bin/ls -F'
   export LS_COLORS='no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.jpg=01;35:*.jpeg=01;35:*.png=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.mpg=01;35:*.mpeg=01;35:*.avi=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:';
    else
      if [ "`id -u`" -eq 0 ]; then
   export PS1='# '
      else
   export PS1='$ '
      fi
    fi

    export USER=`id -un`
    export LOGNAME=$USER
    export HOSTNAME=`/bin/hostname`
    export HISTSIZE=1000
    export HISTFILESIZE=1000
    export PAGER='/bin/more '
    export EDITOR='/bin/vi'
    export INPUTRC=/etc/inputrc
    export DMALLOC_OPTIONS=debug=0x34f47d83,inter=100,log=logfile
    export LANG=en_US.UTF-8
    export LC_ALL=en_US.UTF-8
    export LC_CTYPE=en_US.UTF-8

    ### Some aliases
    alias ps2='ps facux '
    alias ps1='ps faxo "%U %t %p %a" '
    alias af='ps af'
    alias cls='clear'
    alias df='df -h'
    alias indent='indent -bad -bap -bbo -nbc -br -brs -c33 -cd33 -ncdb -ce -ci4 -cli0 -cp33 -cs -d0 -di1 -nfc1 -nfca -hnl -i4 -ip0 -l75 -lp -npcs -npsl -nsc -nsob -nss -ts4 '
    #alias bc='bc -l'
    alias minicom='minicom -c on'
    alias calc='calc -Cd '
    alias bc='calc -Cd '
fi;
export JAVA_HOME=/usr/local/jre
export PATH=$PATH:$JAVA_HOME/bin
export PATH=/opt/bin:/opt/sbin:$PATH



What's wrong?
How can I fix this?

btw: if i manually execute the last few exports, I can configure make and make install fine
Last edited by Morkeleb on Tue Apr 12, 2011 6:43 pm, edited 1 time in total.

User avatar
schumaku
Guru
Posts: 43663
Joined: Mon Jan 21, 2008 4:41 pm
Location: Kloten (Zurich), Switzerland -- Skype: schumaku
Contact:

Re: ipgk problem

Post by schumaku » Tue Apr 12, 2011 6:42 pm

Changed to an alternate shell in /etc/passwd ? Default is /bin/sh

# /bin/sh --version
GNU bash, version 3.2.0(4)-release (i686-pc-linux-gnu)
Copyright (C) 2005 Free Software Foundation, Inc.

Morkeleb
Starting out
Posts: 17
Joined: Sun Dec 26, 2010 4:16 am

Re: ipgk problem

Post by Morkeleb » Tue Apr 12, 2011 11:36 pm

No. I merely installed bash (ipkg install bash, since i need it for some compilation), but i did not set it to the default.

"/bin/sh --version" gets me:
GNU bash, version 3.2.0(135)-release (i686-pc-linux-gnu)
Copyright (C) 2005 Free Software Foundation, Inc.

Anything else I can check or do so the profile is properly executed?

The purpose of this all is to improve video and audio encoding used in subsonic (which can be installed on a qnap system after installing java and tompcat).
It uses ffmpeg for the encoding that shall also support x264, and I wish compile the latest version of ffmpeg with optimized options.
I already got a basic compile working, now I want to optimzize it.
- I mostly followed those:
viewtopic.php?f=177&t=32731
viewtopic.php?p=189008

Now I will also try to add the option --enable-pthreads since i have a dual-core nas.
Furthermore I am just compiling gcc 4.5 which has optimization options for atom processors (when i got it working, i can add -march=atom and -mtune=atom).

Compiling gcc is not easy..
For starters, i had to install more current versions of gawk and sed (using ipkg install).

Then gcc needs gmp, mpfr, mpc and at best also ppl and gloog-ppl.
I downloaded gmp-4.3.2, mpfr-3.0.1, mpc-0.9, ppl-0.11 and cloog-ppl-0.15.

Since ppl needs gmp with c++ extenstions, i had to configure it like this: ./configure --prefix=/opt --enable-cxx
But the enable-cxx poption causes problems when linking since it searches libstdc++.la in a strange location.
I fixed this by executing those:
mkdir -p /home/slug/optware/ts509/toolchain/i686-unknown-linux-gnu/bin/
mkdir -p /home/slug/optware/ts509/toolchain/i686-unknown-linux-gnu/i686-unknown-linux-gnu/lib/
ln -s /share/MD0_DATA/.qpkg/Optware/bin/ld /home/slug/optware/ts509/toolchain/i686-unknown-linux-gnu/bin/i686-unknown-linux-gnu-ld
ln -s /share/MD0_DATA/.qpkg/Optware/bin/as /home/slug/optware/ts509/toolchain/i686-unknown-linux-gnu/bin/i686-unknown-linux-gnu-as
ln -s /share/MD0_DATA/.qpkg/Optware/lib/libstdc++.la /home/slug/optware/ts509/toolchain/i686-unknown-linux-gnu/i686-unknown-linux-gnu/lib

Then ppl also has some linking problems with libstdc++, and I am still working on those.
I executed 'export LD_LIBRARY_PATH=/share/MD0_DATA/.qpkg/Optware/lib'
and then tried './configure --prefix=/opt --with-gmp-includes=/share/MD0_DATA/.qpkg/Optware/include --with-gmp-dir=/share/MD0_DATA/.qpkg/Optware/lib'
or './configure --prefix=/opt CFLAGS="-O3 -march=i686"'

but this always gives me:
configure:10644: checking how to link with libgmpxx
configure:11117: result: /opt/lib/libgmpxx.so /opt/lib/libgmp.so -Wl,-rpath -Wl,/opt/lib -Wl,-rpath -Wl,/opt/lib
configure:11163: checking for the GMP library version 4.1.3 or above
configure:11252: g++ -o conftest -g -O2 -frounding-math -I/opt/include conftest.cpp /opt/lib/libgmpxx.so /opt/lib/libgmp.so -Wl,-rpath -Wl,/opt/lib -Wl,-rpath -Wl,/opt/lib >&5
/tmp/ccCNCID7.o: In function `__exchange_and_add':
/share/MD0_DATA/.qpkg/Optware/bin/../lib/gcc/i686-unknown-linux-gnu/4.2.1/../../../../include/c++/4.2.1/ext/atomicity.h:51: undefined reference to `__sync_fetch_and_add_4'

And idea how to fix this?

Furthermore when compiling gcc without the optional extenstions, i get this error:
The directory that should contain system headers does not exist: /usr/include

I guess i have to get the right headers somewhere.
Where can I get the appopriate (kernel?) headers?

Once i get this working, hopefully this will help coax the most out of my NAS so i can at least stream any SD content (it stutters in some videos). I guess the processor is just to slow for any 720p content.

User avatar
Don
Guru
Posts: 12000
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: ipgk problem

Post by Don » Wed Apr 13, 2011 10:37 am

Not sure why it is not working but just enter 'export PATH=/opt/bin:/opt/sbin:$PATH' after you logon.
Read the Online Manuals and use the forum search feature before posting.

It is a recommended to use RAID and have external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

Submit bugs and feature requests to QNAP via their Helpdesk app.

NAS: TVS-882BR | F/W: 4.3.6.0895 | 40GB | 2 x M.2 SATA RAID 1 (System/VMs) | 4 x M.2 NMVe QM2-4P-384A RAID 5 (Cache) | 5 x 4TB HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-663 | F/W: 4.4.2.1320 | 16GB | 2 x M.2 NMVe QM2-2P RAID 1 (Cache) | 4 x 4TB RAID 5
Apps: Boinc, Squid, DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS, Entware, DLstation, +others

Morkeleb
Starting out
Posts: 17
Joined: Sun Dec 26, 2010 4:16 am

Re: ipgk problem

Post by Morkeleb » Wed Apr 13, 2011 10:58 pm

I have some updates:

When logging in with ssh from a mac the PATH is set correctly. Have to track this down further but its no real issue anymore.

I manged to configure ppl now by executing those before (not sure if the second line is necessary):

export CFLAGS="-O3 -march=i686 -mtune=i686"
export CXXFLAGS="${CFLAGS}"

I think it's best to do all other compiles with that too (please note though that you should remove those again when compiling mplayer, as mplayer detects the best fitting CFLAG options itself that work - the ones provided here caused an error while compiling).
Plus, for gmp, it's recommended to configure it like this: CPPFLAGS=-fexceptions ./configure --enable-cxx --prefix=/opt
Otherwise ppl will warn you about this.

I've found some guide how to create kerrnel headers, and i'm on it: http://cross-lfs.org/view/svn/x86_64-64 ... aders.html
However i have a problem with executing make mrproper (some problem with find command, but i'm not sure).

I downloaded, compiled and installed mplayer/mencoder and compared performance.
I tried to have the settings as close together as possible, and with the commands below i get about 16-17 fps with ffmpeg and 28-29 fps (!!) with mencoder.

ffmpeg -i "/share/MD0_DATA/Multimedia/video1.mkv" -b 500k -s 800x448 -acodec libmp3lame -ab 96k -ar 44100 -ac 2 -f avi -vcodec libx264 -threads 4 -fpre /var/subsonic/transcode/presets/libx264-ultrafast.ffpreset "/share/MD0_DATA/Multimedia/video1-ffmpeg.avi"

mencoder "/share/MD0_DATA/Multimedia/video1.mkv" -lavdopts fast:threads=4 -oac mp3lame -lameopts cbr:br=96:mode=1 -srate 44100 -ovc x264 -x264encopts preset=ultrafast:threads=4:bitrate=500 -vf scale=800:448 -of avi -o "/share/MD0_DATA/Multimedia/video1-mencoder.avi"

Maybe i'm missing out on some important video decoding option to also use multiple threads or ffmpeg just does not work as well, not sure. Fact is mencoder saturates the cpu much better in all 4 cores, whereas ffmpeg does not fully use them at all and usage varies very much.

To my horror i found that current versions (i checked out the latest one) of x264 has a problem and after some time of encoding throws such an error:
mencoder: encoder/slicetype.c:1632: x264_rc_analyse_slice: Assertion `cost >= 0' failed.

I read that older versions (November 2010) work ok.
So I got the last snapshot from november from here: ftp://ftp.videolan.org/pub/videolan/x264/snapshots/
and compiled and installed it, aferwards I had to compile and install mencoder again.
I then ran an encode again (one that cause this error quite quickly) and it worked without errors!

I'm writing this all up so others can benefit of the knowledge I gathered here....

Morkeleb
Starting out
Posts: 17
Joined: Sun Dec 26, 2010 4:16 am

Re: ipgk problem

Post by Morkeleb » Fri Apr 22, 2011 10:37 pm

I've done much since my last post, so much in fact that I cannot in detail explain how I got some things working, but I'll try to cover the important parts.

Firstly, I found that configuring /etc/ld.so.conf correctly with all libraries needed is important.
I changed the /etc/init.d/Optware.sh script to add some paths. I include all changes here, some are also documented elsewhere (as mentioned in an above post):

Code: Select all

        # adding Ipkg apps into system path ...
        /bin/cat /etc/profile | /bin/grep "PATH" | /bin/grep "/opt/bin" 1>>/dev/null 2>>/dev/null
        # Bug fix for following: put IPKG first, per http://forum.qnap.com/viewtopic.php?f=124&t=15663
        # was [ $? -ne 0 ] && /bin/echo "export PATH=$PATH":/opt/bin:/opt/sbin >> /etc/profile
        [ $? -ne 0 ] && /bin/echo "export PATH=/opt/bin:/opt/sbin:\$PATH" >> /etc/profile

        # Added by Lukas Oesterreicher to enable finding any /opt libraries
        /bin/echo "/opt/lib" >> /etc/ld.so.conf
        /bin/echo "/opt/local/lib" >> /etc/ld.so.conf
        /bin/echo "/opt/i686-unknown-linux-gnu/lib" >> /etc/ld.so.conf
        /sbin/ldconfig

        # Patch per http://wiki.qnap.com/wiki/Install_Optware_IPKG
        /bin/echo "Run Optware/ipkg /opt/etc/init.d/*"
        source /etc/profile
        # Start all init scripts in /opt/etc/init.d
        # executing them in numerical order.
        #
        for i in /opt/etc/init.d/S??* ;do
            # Ignore dangling symlinks (if any).
            #[ ! -f "$i" ] && continue

            case "$i" in
                *.sh)
                    # Source shell script for speed.
                    (
                        trap - INT QUIT TSTP
                        set start
                        . $i
                    )
                ;;
                *)
                    # No sh extension, so fork subprocess.
                    $i start
                ;;
            esac
        done
        # End patch


Just search for # adding Ipkg apps into system path ... in the existing Optware.sh to know where to put this in.

Also, i found it can help to set an environment variable with the library paths, like this:

export LD_CONFIG_PATH=/opt/lib:/opt/local/lib:/opt/i686-unknown-linux-gnu/lib:/lib:/usr/lib:/usr/local/lib

Finally some header files may not be correctly found, this helps here:

export C_INCLUDE_PATH=/usr/include:/opt/include
export CPLUS_INCLUDE_PATH="${C_INCLUDE_PATH}"

Also, I found if some libraries are needed in tomcat (calling an external program like mplayer that uses them) can cause problem because i think by default tomcat starts up before Optware.sh, and then the additonal libary paths are not visible inside tomcat.

Restarting tomcat helps.

I also had to compile a new version of perl that supports threading. This is needed in some of my compiles.

As to the real problem of streaming 720p videos to flash:
I got this working finally, using a modified version of mplayer2.
See here for details on getting and building it:

https://github.com/divVerent/mplayer/wi ... au-vo-lavc

For info on mplayer2, see here: http://www.mplayer2.org/

If done right mplayer2 building should not be too difficult,
but please when changing anything clean up after yourself,
like on a new x264 version, remove the old x264 before.
If you had a standard ffmpeg built before, remove that.

I fiddled much on a script that can be used in subsonic for video streaming.
This is my result (to this point):

Code: Select all

#!/bin/sh

# $1=offset in seconds
# $2=source file
# $3 bitrate in kbit/s
# $4 width in pixels (optional, currently ignored and derived from height and aspect ratio)
# $5 hight in pixels (optional, currently ignored and derived from bitrate)
# $6 output: if provided output file name, if not provided output is sdtout
offset=$1; source=$2; bitrate=$3; width=$4; height=$5; output=$6;

VIDEO_FPS=$(/opt/etc/mplayer/TOOLS/midentify.sh "$source" | sed -n 's/^ID_VIDEO_FPS=//p' | sed 's/\.//g')

if [ $VIDEO_FPS -ge 100000 ]; then
  FPS_OPTION="-ofps 24000/1001";
else
  FPS_OPTION="-oautofps";
fi

if [ $bitrate -le 200 ]; then
  MAXRATE=200k; CRF=23; AUDIORATE=64k; RESOLUTION=240;SPEEED=@faster;
elif [ $bitrate -le 300 ]; then
  MAXRATE=300k; CRF=22; AUDIORATE=64k; RESOLUTION=240;SPEEED=@faster;
elif [ $bitrate -le 400 ]; then
  MAXRATE=400k; CRF=23; AUDIORATE=64k; RESOLUTION=360;SPEEED=@superfast;
elif [ $bitrate -le 500 ]; then
  MAXRATE=500k; CRF=22; AUDIORATE=64k; RESOLUTION=360;SPEEED=@superfast;
elif [ $bitrate -le 700 ]; then
  MAXRATE=700k; CRF=23; AUDIORATE=96k; RESOLUTION=384;SPEEED=@superfast;
elif [ $bitrate -le 1000 ]; then
  MAXRATE=1000k; CRF=22; AUDIORATE=96k; RESOLUTION=384;SPEEED=@superfast;
elif [ $bitrate -le 1200 ]; then
  MAXRATE=1200k; CRF=22; AUDIORATE=96k; RESOLUTION=416;SPEEED=@ultrafast;
elif [ $bitrate -le 1500 ]; then
  MAXRATE=1500k; CRF=21; AUDIORATE=96k; RESOLUTION=416;SPEEED=@ultrafast;
elif [ $bitrate -le 2000 ]; then
  MAXRATE=2000k; CRF=21; AUDIORATE=112k; RESOLUTION=448;SPEEED=@ultrafast;
else
  MAXRATE=3000k; CRF=20; AUDIORATE=128k; RESOLUTION=448;SPEEED=@ultrafast;
fi


INPUT_OPTIONS="-lavdopts fast:threads=4:skiploopfilter=all -af volnorm -channels 2 -srate 44100"
AUDIO_ENCODING="-oac aac -oacopts b=$AUDIORATE"
VIDEO_ENCODING="-ovc libx264 -ovcopts $SPEEED,@film,@baseline,@fastdecode,level=21,threads=4,maxrate=$MAXRATE,bufsize=3000k,rc_init_occupancy=2800k,crf=$CRF"
OUTPUT_OPTIONS="-vf scale=-2:$RESOLUTION -sws 0 $FPS_OPTION -of flv"

# if we did not want different loggin options depending on streaming or not, we may be able to use use -o "${6-/dev/fd/3}"
if [ -z "$output" ]; then
/opt/bin/mplayer -ss $offset "$source" $INPUT_OPTIONS $AUDIO_ENCODING $VIDEO_ENCODING $OUTPUT_OPTIONS -o /dev/fd/3 3>&1 1>>/var/subsonic/transcode/logs.log 2>>/var/subsonic/transcode/errors.log
else
echo "/opt/bin/mplayer -ss $offset \"$source\" $INPUT_OPTIONS $AUDIO_ENCODING $VIDEO_ENCODING $OUTPUT_OPTIONS -o \"$output\""
/opt/bin/mplayer -ss $offset "$source" $INPUT_OPTIONS $AUDIO_ENCODING $VIDEO_ENCODING $OUTPUT_OPTIONS -o "$output"
fi


As you can see i currently do not use bitrate, height and with.
Setting the resulution statically like this is needed since otherwise for high-resolution videos it would try to output to the same resolution, which can't be done in real time. Please note that this maintains the aspect ratio of the input. This setting is right for 16:9 videos. For 4:3 videos you will probably get 480x360, and for film-widescreen shou will get somewere around 852x360.

The quality of a crf option is better than a bitrate (but you can use b=$3k instead of crf=264).

To compile gcc, i followed this to create header files: http://gd.tuwien.ac.at/opsys/linux/moll ... aders.html ( you might need to translate it, but the commands should be clear).
Please make sure to get the kernel sources in the version of the kernel that you are using.

To compile gcc, i extracted gcc (4.5.2) and also created a directory gcc-4.5.2-build and from this second directory called /opt/etc/gcc-4.5.2/configure --prefix=/opt --disable-multilib and then make

I still have problems compiling though, even after adding --disable-multilib to disable compiling for both 32 and 64 bit.
This is what I get:
/home/slug/optware/ts509/toolchain/i686-unknown-linux-gnu/bin/i686-unknown-linux-gnu-ld: crti.o: No such file: No such file or directory
collect2: ld returned 1 exit status

Maybe you have an idea how to fix this?

Btw, the speed increase of an updated gcc will be very small, but i heared the standard provided gcc is a rather buggy one. Others could not reproduce the x264 problem in the latest version, so I will try to get gcc updated and recompile x264 with it and see if I still have the problem then..

Morkeleb
Starting out
Posts: 17
Joined: Sun Dec 26, 2010 4:16 am

Re: ipgk problem

Post by Morkeleb » Wed Apr 27, 2011 4:47 am

I did quite some documentation work on how to

* install and configure subsonic on a qnap system
* compile, install and configure mplayer2 on a qnap system for use with subsonic

Please see here:
http://www.activeobjects.no/subsonic/fo ... hp?p=25289

Post Reply

Return to “Users' Corner”