Page 1 of 13

Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Fri Mar 02, 2018 4:48 pm
by jollino
Hello all,
I had set up my TS-431 with 4 disks in a RAID5 array to run a monthly scrub at midnight on the first day of each month, and it started just fine on March 1st, announcing it with a beep. About 26 hours into it, however, the NAS went beeping and disappeared from the network for a few minutes. The NAS is connected to a UPS with no other devices and I'm the only user, and hadn't accessed it at all. I am running QTS 4.3.4.0486 (2018-02-15).

When it came back from the dead, it stated that the machine had rebooted unexpectedly, and the filesystem was not clean so a fsck was in order; I stopped the scrub (which had started again) I ran the check, and thankfully it reported no damage.

Is there any way to find out what happened? I can't find the actual system logs anywhere: /var/log/messages is empty and there are no utmp/wtmp files. Even getting some output from 'last' would be nice to see if it reports anything.

I'm concerned that for some reason the scrub may have triggered a kernel panic, and I'm not sure if it's worth the risk to even just try the scrub again manually.

Any advice would be most welcome. :)

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Sat Mar 03, 2018 2:44 am
by 4ppr3ntice
Hi all,

same problem here: TS-431 with RAID5 array (4 disks), monthly scrub scheduled at midnight on each month's 1st. Scrub started and system rebooted unexpectedly after about 6 hours. I stopped scrubbing, did an fsck (from GUI), and restarted scrubbing: again it rebooted unexpectedly after several hours.
One more time: fsck and restart scrubbing, and it rebooted again. I also tried to lower scrubbing priority, but it did not help.
After last reboot and fsck, I did not restart scrubbing; and I'm waiting to see if it reboots again.
If it won't I'd assume that scrubbing is involved in this behavior.

As already stated:
jollino wrote:Any advice would be most welcome.
Thanks in advance

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Sat Mar 03, 2018 2:50 am
by Don
Open a ticket with QNAP via the help desk app and upload the log files.

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Sat Mar 03, 2018 3:17 am
by jollino
That's my question, how do we get the log files? I can't find any of the traditional Unix/Linux log files on the system.

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Sat Mar 03, 2018 3:21 am
by Don
Use the helpdesk app. It will download and zip the files for upload to QNAP when it creates the ticket.

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Sat Mar 03, 2018 3:26 am
by jollino
Oh, I see. Thank you very much, I never needed to use that until now so I had no idea. :) Will do.

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Fri Mar 09, 2018 4:46 pm
by jollino
4ppr3ntice wrote:Hi all,

same problem here: TS-431 with RAID5 array (4 disks), monthly scrub scheduled at midnight on each month's 1st. Scrub started and system rebooted unexpectedly after about 6 hours. I stopped scrubbing, did an fsck (from GUI), and restarted scrubbing: again it rebooted unexpectedly after several hours.
One more time: fsck and restart scrubbing, and it rebooted again. I also tried to lower scrubbing priority, but it did not help.
After last reboot and fsck, I did not restart scrubbing; and I'm waiting to see if it reboots again.
If it won't I'd assume that scrubbing is involved in this behavior.
Do you have any updates on this? Did you try it again? I opened a ticket and they replied quickly asking to do remote support on my unit, but it's been four days now and I'm still waiting.

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Fri Mar 09, 2018 5:48 pm
by storageman
And what do the SMART checks show?
That the disks are fine?

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Fri Mar 09, 2018 7:04 pm
by jollino
storageman wrote:And what do the SMART checks show?
That the disks are fine?
In my case they are, the check didn't report anything wrong and SMART is fine on all four disks. I didn't try scrubbing again though.

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Fri Mar 09, 2018 9:11 pm
by storageman
I would run it and see what the processor/ram levels show too.
Clearly it should not reboot if the drives are ok.

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Sat Mar 10, 2018 6:42 pm
by 4ppr3ntice
jollino wrote: Do you have any updates on this? Did you try it again? I opened a ticket and they replied quickly asking to do remote support on my unit, but it's been four days now and I'm still waiting.
Once stopped scrubbing, NAS has been running for more than five days, before rebooting again; this happened because scrubbing was restarted by a scheduled activity. I'm still investigating which process did the trick, because regular schedule was monthly...
Now scrubbing is de-scheduled, for an abundance of caution.

I also opened a ticket with QNAP in the meantime, HDD section. Support stated that my configuration is not fully supported, from a storage compatibility point of view.
Although my disks are in the compatibility list (actually they are 4 disks of the same model), they need a firmware update in order to be fully compliant. To be honest, I did not realize that for more than two years after assembling the RAID array, because no problems arose.
Now I'm trying to get support from disk vendor: online automated download services don't show any firmware update available for my HDDs' serials. So I've emailed them asking for a deeper check (also because QNAP states that there is a newer firmware for them, by putting it in their compatibility list). Got a quick answer, with a link to a non-compatible firmware; replied to that, and I'm still waiting.
off-topic start
Just to understand how deep my fault was, two questions for all users.
How many of your preferred storage resellers can guarantee you in advance the firmware version of the disks you are going to purchase?
How many of you check the firmware version before mounting the disk in the bay, once you know that disk model is compatible?
off topic end

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Mon Mar 12, 2018 1:10 am
by katbert
Same issie with TS-431 4.3.4.0486 and unexcepted reboot in RAID Scrubbing
My problem begins from 02 mar 2018
In the end of feb - firmware was updated from 4.3.3.0396 to 4.3.4.0486
In jan and feb 2018 - RAID scrubbing finished successfully
Maybe this is issue of firmware 4.3.4.0486?

My topic on forum:
viewtopic.php?f=73&t=139856

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Mon Mar 12, 2018 1:37 am
by 4ppr3ntice
Forgot to mention that, in all of my posts: firmware 4.3.4.0486 here too.
Looks like QNAP should take our issues into consideration: one coincidence is just a coincidence, two coincidences are a clue, three coincidences are a proof...
To get back at the original question in the topic: to me it looks like that most recent logs are in /mnt/HDA_ROOT/.logs, at least on TS-431.

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Mon Mar 12, 2018 2:24 pm
by katbert
Kernel log analyzer in Qnap diagnostic tool show only one problem, but it was in nov 2015

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Posted: Mon Mar 12, 2018 3:11 pm
by 4ppr3ntice
No errors in the kernel log, no errors in SMART analysis, for me. I'm not that great log reader, but I haven't seen any lines easily identifiable as errors/warnings around reboot events.
As stated in thread title, reboots are "sudden"; the one and only signal, as far as I've seen, are the NAS "beeps".

As soon as I get the FW update (and if I manage not to brick anything... :wink:), I'll force start scrubbing. If it reboots again, I'll go back to QNAP support (assuming that they did not do anything in the meantime, like a QTS update).

FYI, Seagate ST4000VN000 with SC46 firmware aboard HDDs are not fully compatible with TS-431; they need at least SC60.