HD Station Stability & Automatic Restarts?

[canadiantrekkie]

Hi,

Before I start, my system & load line up is:

NAS: TS-251
Firmware version: 4.1.3 (20150408)
HD Station: 2.0.21

Problem: Frequently when using applications (Chrome / QTS / Kodi ) connected to a TV, the application will cause HD_Station to crash and necessitate a manual restart of the HD Station, then all is well for a little while.

Workaround: I'm trying create a daemon health monitoring script to restart HD station when it dies / crashes, so I don't always have to manually restart it. Any tips? I've pulled apart /etc/init.d/HD_Station.sh to see what it's doing, but pointers would be helpful from anyone that has already dug down into it's core processes and logs (if there are any) which would save me some time from having to slowly reverse engineer it?

thanks

[canadiantrekkie]

*UPDATED: 22-May-2015 *

What I viewed as a single problem is actually a series of instabilities. By monitoring the kmsg logs at this path [ /mnt/HDA_ROOT/.logs ], I can see several processes that drove off a cliff with a segfault:

- kodi.bin (many occurences of this)
- getcfg
- qtvagent
- LoginScreen
- etc

I have a rudimentary script that traps these particular segfaults from the logs and then calls directly into the /etc/init.d/HD_Station.sh restart, within 10-15 seconds it's' back to normal, but it doesn't solve the original segfault. I need to get some debug tools in this environment so I can debug pstacks and backtraces for these binaries. I can't find any instances of core files being generated, it could be that QNAP linux as turned these options off.

Checking kodi bugtracker reveals several segfault variations, so it could simply be growing pains with this new platform and integration a bunch of off-the-shelf and open source projects on to it. Once I get my script working cleanly I'll post it here for anyone interested. It's only about 10 lines of script now. it's pretty dumb but does the trick.

cheers

[canadiantrekkie]

Here is a perl script I have running on my NAS which is actively checking for segfaults from kmsg and reacting by restarting the HD_Station. So far it's resolved the last 3 issues I've experienced while trying to control the NAS with QRemote + Kore / Sybu. Once the issue occurs, within 30 seconds things are back to normal. It's not an awesome solution, but it works.

I had to install nohup, because I didn't have it, so I started with:
ipkg install coreutils

Then to get it running, I did the following:

nohup /root/monitor.pl > monitor.log 2>&1 &

monitor.pl

Code: Select all

#!/opt/bin/perl

#flush stdout buffers immediately
$| = 1;

sub getUptimeInSeconds() {
   $uptimeStr = `cat /proc/uptime`;

   $secondsSinceBoot = 0;

   if ( $uptimeStr =~ /(.*) / ) {
      $secondsSinceBoot = $1;
   }

   return $secondsSinceBoot;
}

while(1==1) {

   $segfaultMonitorWindowStart = getUptimeInSeconds() - 10;
   #print "\n\nChecking for segfault occurrences within the last 10 seconds starting at time index = $segfaultMonitorWindowStart";

   @segfaults = `cat /mnt/HDA_ROOT/.logs/kmsg | grep segfault`;

   for $segfault ( @segfaults ) {

      if ( $segfault =~ /(\[.*\]) / ) {
        $segfaultTimeIndex = $1;
        #print "\nsegaultTimeIndex = $segfaultTimeIndex";
        if ( $segfaultTimeIndex > $segfaultMonitorWindowStart ) {
          print "\nFound a segault occurence that happened recently = '$segfault', restarting HD Station";
          `/etc/init.d/HD_Station.sh restart`;
          # Don't want to double trigger restarts, so wait a minute between restarts
          print "\n Sleeping for 1 minute after HD Station restart";
          sleep 60;
        }
      }
   }

   sleep 10;
}

[canadiantrekkie]

The script I posted was a bit messed up, it some how worked when I was testing it, but it wasn't working properly. I've tested this one more thoroughly and this is working properly now. I could further refine it to filter out the segfaults by process type, but most of the time (for me) the segfaults are in relation to something crapping out in an associated HD_Station process.

Code: Select all

# cat monitor.pl
#!/opt/bin/perl

#flush stdout buffers immediately
$| = 1;

sub getTimeIndexOfLastSegfault() {

  $lastSegfaultIndexStr = `cat /mnt/HDA_ROOT/.logs/kmsg | grep segfault | tail -1`;
  if ( $lastSegfaultIndexStr =~ /\[(.*)\] / ) {
     return $1;
  }

  return -1;

}

$timeIndexOfLastSegfault = getTimeIndexOfLastSegfault();

while(1==1) {

   #print "\nMonitoring\n";

   sleep 5;

   $latestTimeIndexOfSegfault = getTimeIndexOfLastSegfault();

   if ( $timeIndexOfLastSegfault < $latestTimeIndexOfSegfault ) {
       $dateStr = `date`;
       $dateStr =~ s/\n//;
       print "\nFound a new segfault on [$dateStr], restarting HD station";
       $timeIndexOfLastSegfault = $latestTimeIndexOfSegfault;
       `cd /share/CACHEDEV1_DATA/.qpkg/HD_Station; ./HD_Station.sh restart`;
       sleep 10;
   }

}