Page 1 of 2

QNAP Hangs on Shutdown or Reboot

Posted: Fri Jul 07, 2017 12:26 am
by MRVigliotti
I'm having an issue where my TS-853A hangs during a shutdown or reboot. I can tell it unmounts all the drives and starts the shutdown process but then just hangs at "rebooting system". I have to do a hard power off (with the power button). It then boots cleanly when I turn it back on with no errors or filesystem checks.

I was able to solve this problem about a year ago with the help of a forum post and it turned out to be related to Plex. Basically, Plex was not shutting down in time and hanging the QNAP shutdown process. I rooted into the console and did some sort of prioritization of what was shut down. When I placed Plex higher in the order, it had more time to shutdown and everything worked.

Low and behold, a year passes, and the issue is back (probably a firmware update) and I can't find the link.

Can anyone help?

Re: QNAP Hangs on Shutdown or Reboot

Posted: Fri Jul 07, 2017 2:19 am
by Toxic17
have your run the QNAP Diagnostics tool? if not i would strongly suggest running this (available in app center) once you have run it (Dump log) check the errors and see if it points to anything in particular.

You have not said what firmware version and build number you are on, i will guess 4.3.3.0238 ????

I suggest downloading this app to make the correct app priority.

[QPKG] Sort my QPKGs

Re: QNAP Hangs on Shutdown or Reboot

Posted: Fri Jul 07, 2017 8:52 pm
by MRVigliotti
Thanks very much for the info! Yes, I'm on 4.3.3.0238. I was able to install and run the diagnostic tool. It's a very powerful tool and good to have but I'm getting log overload here and I'm not exactly sure what I should be looking for. The event log output for the last hang is below, not much to infer from it.

Code: Select all

1115,  0,2017-07-05,21:45:01,admin,192.168.10.1,---,[Network Recycle Bin] Network recycle bin emptied.,  0,1499305501,
1116,  0,2017-07-05,21:45:26,admin,192.168.10.1,---,[Firmware Update] Start downloading firmware 4.3.3.0238 Build 20170703.,  0,1499305526,
1117,  0,2017-07-05,21:46:50,admin,192.168.10.1,---,[Firmware Update] Start unzipping TS-X53A_20170703-4.3.3.0238.zip,  0,1499305610,
1118,  0,2017-07-05,21:47:02,admin,192.168.10.1,---,[Firmware Update] Start updating firmware 4.3.3.0238 Build 20170703.,  0,1499305622,
1119,  0,2017-07-05,21:47:18,System,127.0.0.1,localhost,[Firmware Update] Started updating the firmware.,  0,1499305638,
1120,  0,2017-07-05,21:49:01,System,127.0.0.1,localhost,[Firmware Update] System updated successfully from 4.3.3.0229(20170624) to 4.3.3.0238(20170703).,  0,1499305741,
1121,  0,2017-07-05,21:50:05,admin,192.168.10.1,---,[Power Management] System restarting.,  0,1499305805,
1122,  0,2017-07-05,21:50:14,System,127.0.0.1,localhost,Stopping Plex Media Server...,  0,1499305814,
1123,  0,2017-07-05,21:50:16,System,127.0.0.1,localhost,[Network & Virtual Switch]Stop Network & Virtual Switch.,  0,1499305816,
1124,  0,2017-07-05,21:51:12,System,127.0.0.1,localhost,System was shut down on Wed Jul  5 21:51:12 EDT 2017.,  0,1499305872,
1125,  0,2017-07-05,22:04:08,System,127.0.0.1,localhost,System started.,  0,1499306648,
1126,  0,2017-07-05,22:05:31,System,127.0.0.1,localhost,[App Center] Resource Monitor enabled.,  0,1499306731,
1127,  0,2017-07-05,22:06:02,System,127.0.0.1,localhost,[App Center] Helpdesk 1.1.04 has been installed in /mnt/HDA_ROOT/update_pkg/helpdesk successfully.,  0,1499306762,
1128,  0,2017-07-05,22:06:02,System,127.0.0.1,localhost,[App Center] Helpdesk enabled.,  0,1499306762,
1129,  0,2017-07-05,22:06:10,System,127.0.0.1,localhost,Starting Plex Media Server...,  0,1499306770,
1130,  0,2017-07-05,22:06:44,System,127.0.0.1,localhost,[App Center] Helpdesk 1.1.04 has been installed in /mnt/HDA_ROOT/update_pkg/helpdesk successfully.,  0,1499306804,
1131,  0,2017-07-05,22:06:45,System,127.0.0.1,localhost,[App Center] Helpdesk enabled.,  0,1499306805,
1132,  0,2017-07-05,22:07:39,System,127.0.0.1,localhost,[App Center] Qsync Central enabled.,  0,1499306859,
1133,  0,2017-07-05,22:07:40,System,127.0.0.1,localhost,[App Center] Qsync Central 3.0.1 installation succeeded.,  0,1499306860,
Also, the sort my QPKG that was recommended, does this tool also have the capability of dictating the shutdown order? If I'm having the same problem as last time, there is a hang on the shutdown, not on boot up.

Re: QNAP Hangs on Shutdown or Reboot

Posted: Fri Jul 07, 2017 8:56 pm
by Toxic17
one the diagtool has been run download the file it creates (think it is a zip file) and click on the HTML file inside. this will list a few headings, and links. look for "errors"

You would need to ask the author of that program. though I doubt it controls the shutdown procedure process. it is only listing start up of QPKGs and not all services or processes.

have you checked all your hard disks via the SMART testing, and also setup a schedule to check your hard disk array (Data Scrubbing) one of which maybe causing your issue.

Re: QNAP Hangs on Shutdown or Reboot

Posted: Fri Jul 07, 2017 9:35 pm
by MRVigliotti
I have been diving deeper into this issue and I am positive that the Plex QPKG is causing this issue. I have no errors, the SMART testing in clear, and I data scrub every week as is now default. In fact, when the shutdown hangs, the disk array is shutdown before. Everything else is working fine.

I disabled the Plex QPKG before shutdown and the NAS rebooted cleanly. Enabled it back and restarted again it hangs. This is the exact problem that I fixed in the past. Plex just needs a few extra seconds to stop it's service. I fixed it by moving it up higher in the shutdown priority in the console. I have been searching to no avail to find the walkthrough that helped me.

Does anyone know which configuration file controls the QPKG shutdown process? There has to be one as I edited before, I just can't find it.

Re: QNAP Hangs on Shutdown or Reboot

Posted: Fri Jul 07, 2017 11:03 pm
by Toxic17
are you using the latest package from Plex themselves?

1.7.5.4035-313f93718 June 28, 2017

might be worth asking on their forums if you can replicate this when Plex is enabled.

https://forums.plex.tv/categories/qnap

Re: QNAP Hangs on Shutdown or Reboot

Posted: Sat Jul 08, 2017 12:19 am
by OneCD
MRVigliotti wrote:Plex just needs a few extra seconds to stop it's service. I fixed it by moving it up higher in the shutdown priority in the console. I have been searching to no avail to find the walkthrough that helped me.

Does anyone know which configuration file controls the QPKG shutdown process? There has to be one as I edited before, I just can't find it.
The order QPKGs are shutdown is the reverse of the order they are loaded. e.g. last QPKG launched on bootup is the first one killed on shutdown.

There are a couple of things that might help you here:
  • change the kill order name for Plex in (/etc/rcK.d) so it has a lower number. So, if the automatically generated symlink is currently something like 'QK120Plex', rename this to 'QK100Plex'.

    Note: this will not be a persistent change and will need to be re-applied after each NAS startup, so maybe create an autorun script to do this.
  • or you could modify the Plex init script 'stop' function and force it to sleep for a few seconds. This would be a persistent change and would not need to be re-applied after NAS startup.

Re: QNAP Hangs on Shutdown or Reboot

Posted: Sat Jul 08, 2017 3:37 am
by MRVigliotti
Thanks for the revelant info OneCD, I'm almost there.

I did a test of your first suggestion and it worked great, the machine rebooted cleanly! But as you said, it was not done persistently and is back to normal after reboot. Was wondering if you could guide me a little further on setting up an autorun script?

I didn't really explore option 2 since I'm not that great with scripting and I would think that we want the shutdown of the process to start earlier vs. have it take longer? It's taking too long in the first place.

Re: QNAP Hangs on Shutdown or Reboot

Posted: Sat Jul 08, 2017 4:03 am
by OneCD
MRVigliotti wrote:I didn't really explore option 2 since I'm not that great with scripting and I would think that we want the shutdown of the process to start earlier vs. have it take longer? It's taking too long in the first place.
I suggested that as you said:
MRVigliotti wrote:Plex just needs a few extra seconds to stop it's service.
Sleeping the 'stop' function would give it a few extra seconds.
MRVigliotti wrote:I did a test of your first suggestion and it worked great, the machine rebooted cleanly! But as you said, it was not done persistently and is back to normal after reboot. Was wondering if you could guide me a little further on setting up an autorun script?
No worries. Checkout this post - it will create the autorun system for you. Then all you need is a script to run.

Yours will be quite simple:

Code: Select all

#!/usr/bin/env bash
mv /etc/rcK.d/QK???Plex /etc/rcK.d/QK100Plex
I don't use Plex so I don't know the name of the symlink that appears on a NAS with Plex installed, so you may have to tweak that command string. ;)

Save the contents above into a new text file and place it in the autorun scripts directory. Mark it executable. Run it manually to check. Done.

Re: QNAP Hangs on Shutdown or Reboot

Posted: Sat Jul 08, 2017 4:14 am
by OneCD
I've just had a thought - if your autorun script runs before Plex is re-integrated during NAS boot, then nothing will happen.

It may be better to rename the kill symlink as part of the Plex init 'start' function. ;)

Re: QNAP Hangs on Shutdown or Reboot

Posted: Sat Jul 08, 2017 4:43 am
by MRVigliotti
Can confirm, just rebooted, no change. Will test out integrating the kill symlink into the init start tonight when I get home. Many thanks for pointing me in the right direction.

Re: QNAP Hangs on Shutdown or Reboot

Posted: Sat Jul 08, 2017 4:49 am
by OneCD
No worries. Please post the Plex init script if you need a hand.

Re: QNAP Hangs on Shutdown or Reboot

Posted: Sat Jul 08, 2017 8:26 am
by MRVigliotti
Just got around to this and tested and working!

I ended up pasting this command:

Code: Select all

mv /etc/rcK.d/QK105PlexMediaServer /etc/rcK.d/QK100PlexMediaServer
into/etc/init.d/plex.sh right under the start section.

Sure enough, works like a charm. I may need to do this after every Plex update but right now I am just ecstatic that I can reboot the NAS remotely.

Huge thanks to OneCD, really appreciate the guidance, couldn't have fixed it without you!

Re: QNAP Hangs on Shutdown or Reboot

Posted: Sat Jul 08, 2017 8:53 am
by OneCD
That's great. Nicely done. :D

But I've just noticed - you've hardcoded the original kill symlink name. Please note: this can change depending on what else is being loaded. That's why my original suggestion had those 3 question marks. Each one acts as a wildcard meaning "a single unknown character in this position".

The QPKG creator can specify a preferred start number (and the preferred kill number is 1000 minus the start number) from 100 to 999 but the NAS assigns the next one available. So, your 105 may end up being something else if you add/remove QPKGs.

Maybe try this instead:

Code: Select all

mv /etc/rcK.d/QK???PlexMediaServer /etc/rcK.d/QK100PlexMediaServer

Re: QNAP Hangs on Shutdown or Reboot

Posted: Sat Jul 08, 2017 9:06 am
by MRVigliotti
Good catch, updated and tested. Thanks.