Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Questions about SNMP, Power, System, Logs, disk, & RAID.
Post Reply
jollino
Starting out
Posts: 22
Joined: Fri Dec 18, 2015 9:22 pm

Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by jollino » Fri Mar 02, 2018 4:48 pm

Hello all,
I had set up my TS-431 with 4 disks in a RAID5 array to run a monthly scrub at midnight on the first day of each month, and it started just fine on March 1st, announcing it with a beep. About 26 hours into it, however, the NAS went beeping and disappeared from the network for a few minutes. The NAS is connected to a UPS with no other devices and I'm the only user, and hadn't accessed it at all. I am running QTS 4.3.4.0486 (2018-02-15).

When it came back from the dead, it stated that the machine had rebooted unexpectedly, and the filesystem was not clean so a fsck was in order; I stopped the scrub (which had started again) I ran the check, and thankfully it reported no damage.

Is there any way to find out what happened? I can't find the actual system logs anywhere: /var/log/messages is empty and there are no utmp/wtmp files. Even getting some output from 'last' would be nice to see if it reports anything.

I'm concerned that for some reason the scrub may have triggered a kernel panic, and I'm not sure if it's worth the risk to even just try the scrub again manually.

Any advice would be most welcome. :)

4ppr3ntice
Starting out
Posts: 14
Joined: Sun Jun 05, 2016 10:05 pm

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by 4ppr3ntice » Sat Mar 03, 2018 2:44 am

Hi all,

same problem here: TS-431 with RAID5 array (4 disks), monthly scrub scheduled at midnight on each month's 1st. Scrub started and system rebooted unexpectedly after about 6 hours. I stopped scrubbing, did an fsck (from GUI), and restarted scrubbing: again it rebooted unexpectedly after several hours.
One more time: fsck and restart scrubbing, and it rebooted again. I also tried to lower scrubbing priority, but it did not help.
After last reboot and fsck, I did not restart scrubbing; and I'm waiting to see if it reboots again.
If it won't I'd assume that scrubbing is involved in this behavior.

As already stated:
jollino wrote:Any advice would be most welcome.

Thanks in advance
TS-431 + 4 x ST4000VN000-1H4168 RAID5

User avatar
Don
Guru
Posts: 12007
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by Don » Sat Mar 03, 2018 2:50 am

Open a ticket with QNAP via the help desk app and upload the log files.
Read the Online Manuals and use the forum search feature before posting.

It is a recommended to use RAID and have external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

Submit bugs and feature requests to QNAP via their Helpdesk app.

NAS: TVS-882BR | F/W: 4.3.6.0895 | 40GB | 2 x M.2 SATA RAID 1 (System/VMs) | 4 x M.2 NMVe QM2-4P-384A RAID 5 (Cache) | 5 x 4TB HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-663 | F/W: 4.4.2.1320 | 16GB | 2 x M.2 NMVe QM2-2P RAID 1 (Cache) | 4 x 4TB RAID 5
Apps: Boinc, Squid, DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS, Entware, DLstation, +others

jollino
Starting out
Posts: 22
Joined: Fri Dec 18, 2015 9:22 pm

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by jollino » Sat Mar 03, 2018 3:17 am

That's my question, how do we get the log files? I can't find any of the traditional Unix/Linux log files on the system.

User avatar
Don
Guru
Posts: 12007
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by Don » Sat Mar 03, 2018 3:21 am

Use the helpdesk app. It will download and zip the files for upload to QNAP when it creates the ticket.
Read the Online Manuals and use the forum search feature before posting.

It is a recommended to use RAID and have external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

Submit bugs and feature requests to QNAP via their Helpdesk app.

NAS: TVS-882BR | F/W: 4.3.6.0895 | 40GB | 2 x M.2 SATA RAID 1 (System/VMs) | 4 x M.2 NMVe QM2-4P-384A RAID 5 (Cache) | 5 x 4TB HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-663 | F/W: 4.4.2.1320 | 16GB | 2 x M.2 NMVe QM2-2P RAID 1 (Cache) | 4 x 4TB RAID 5
Apps: Boinc, Squid, DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS, Entware, DLstation, +others

jollino
Starting out
Posts: 22
Joined: Fri Dec 18, 2015 9:22 pm

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by jollino » Sat Mar 03, 2018 3:26 am

Oh, I see. Thank you very much, I never needed to use that until now so I had no idea. :) Will do.

jollino
Starting out
Posts: 22
Joined: Fri Dec 18, 2015 9:22 pm

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by jollino » Fri Mar 09, 2018 4:46 pm

4ppr3ntice wrote:Hi all,

same problem here: TS-431 with RAID5 array (4 disks), monthly scrub scheduled at midnight on each month's 1st. Scrub started and system rebooted unexpectedly after about 6 hours. I stopped scrubbing, did an fsck (from GUI), and restarted scrubbing: again it rebooted unexpectedly after several hours.
One more time: fsck and restart scrubbing, and it rebooted again. I also tried to lower scrubbing priority, but it did not help.
After last reboot and fsck, I did not restart scrubbing; and I'm waiting to see if it reboots again.
If it won't I'd assume that scrubbing is involved in this behavior.


Do you have any updates on this? Did you try it again? I opened a ticket and they replied quickly asking to do remote support on my unit, but it's been four days now and I'm still waiting.

User avatar
storageman
Ask me anything
Posts: 5511
Joined: Thu Sep 22, 2011 10:57 pm

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by storageman » Fri Mar 09, 2018 5:48 pm

And what do the SMART checks show?
That the disks are fine?

jollino
Starting out
Posts: 22
Joined: Fri Dec 18, 2015 9:22 pm

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by jollino » Fri Mar 09, 2018 7:04 pm

storageman wrote:And what do the SMART checks show?
That the disks are fine?


In my case they are, the check didn't report anything wrong and SMART is fine on all four disks. I didn't try scrubbing again though.

User avatar
storageman
Ask me anything
Posts: 5511
Joined: Thu Sep 22, 2011 10:57 pm

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by storageman » Fri Mar 09, 2018 9:11 pm

I would run it and see what the processor/ram levels show too.
Clearly it should not reboot if the drives are ok.

4ppr3ntice
Starting out
Posts: 14
Joined: Sun Jun 05, 2016 10:05 pm

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by 4ppr3ntice » Sat Mar 10, 2018 6:42 pm

jollino wrote:Do you have any updates on this? Did you try it again? I opened a ticket and they replied quickly asking to do remote support on my unit, but it's been four days now and I'm still waiting.

Once stopped scrubbing, NAS has been running for more than five days, before rebooting again; this happened because scrubbing was restarted by a scheduled activity. I'm still investigating which process did the trick, because regular schedule was monthly...
Now scrubbing is de-scheduled, for an abundance of caution.

I also opened a ticket with QNAP in the meantime, HDD section. Support stated that my configuration is not fully supported, from a storage compatibility point of view.
Although my disks are in the compatibility list (actually they are 4 disks of the same model), they need a firmware update in order to be fully compliant. To be honest, I did not realize that for more than two years after assembling the RAID array, because no problems arose.
Now I'm trying to get support from disk vendor: online automated download services don't show any firmware update available for my HDDs' serials. So I've emailed them asking for a deeper check (also because QNAP states that there is a newer firmware for them, by putting it in their compatibility list). Got a quick answer, with a link to a non-compatible firmware; replied to that, and I'm still waiting.

off-topic start
Just to understand how deep my fault was, two questions for all users.
How many of your preferred storage resellers can guarantee you in advance the firmware version of the disks you are going to purchase?
How many of you check the firmware version before mounting the disk in the bay, once you know that disk model is compatible?
off topic end
TS-431 + 4 x ST4000VN000-1H4168 RAID5

katbert
Starting out
Posts: 25
Joined: Fri Mar 02, 2018 3:48 pm

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by katbert » Mon Mar 12, 2018 1:10 am

Same issie with TS-431 4.3.4.0486 and unexcepted reboot in RAID Scrubbing
My problem begins from 02 mar 2018
In the end of feb - firmware was updated from 4.3.3.0396 to 4.3.4.0486
In jan and feb 2018 - RAID scrubbing finished successfully
Maybe this is issue of firmware 4.3.4.0486?

My topic on forum:
viewtopic.php?f=73&t=139856

4ppr3ntice
Starting out
Posts: 14
Joined: Sun Jun 05, 2016 10:05 pm

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by 4ppr3ntice » Mon Mar 12, 2018 1:37 am

Forgot to mention that, in all of my posts: firmware 4.3.4.0486 here too.
Looks like QNAP should take our issues into consideration: one coincidence is just a coincidence, two coincidences are a clue, three coincidences are a proof...
To get back at the original question in the topic: to me it looks like that most recent logs are in /mnt/HDA_ROOT/.logs, at least on TS-431.
TS-431 + 4 x ST4000VN000-1H4168 RAID5

katbert
Starting out
Posts: 25
Joined: Fri Mar 02, 2018 3:48 pm

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by katbert » Mon Mar 12, 2018 2:24 pm

Kernel log analyzer in Qnap diagnostic tool show only one problem, but it was in nov 2015
You do not have the required permissions to view the files attached to this post.

4ppr3ntice
Starting out
Posts: 14
Joined: Sun Jun 05, 2016 10:05 pm

Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?

Post by 4ppr3ntice » Mon Mar 12, 2018 3:11 pm

No errors in the kernel log, no errors in SMART analysis, for me. I'm not that great log reader, but I haven't seen any lines easily identifiable as errors/warnings around reboot events.
As stated in thread title, reboots are "sudden"; the one and only signal, as far as I've seen, are the NAS "beeps".

As soon as I get the FW update (and if I manage not to brick anything... :wink:), I'll force start scrubbing. If it reboots again, I'll go back to QNAP support (assuming that they did not do anything in the meantime, like a QTS update).

FYI, Seagate ST4000VN000 with SC46 firmware aboard HDDs are not fully compatible with TS-431; they need at least SC60.
TS-431 + 4 x ST4000VN000-1H4168 RAID5

Post Reply

Return to “System & Disk Volume Management”