Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
-
- Starting out
- Posts: 23
- Joined: Fri Dec 18, 2015 9:22 pm
Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
Hello all,
I had set up my TS-431 with 4 disks in a RAID5 array to run a monthly scrub at midnight on the first day of each month, and it started just fine on March 1st, announcing it with a beep. About 26 hours into it, however, the NAS went beeping and disappeared from the network for a few minutes. The NAS is connected to a UPS with no other devices and I'm the only user, and hadn't accessed it at all. I am running QTS 4.3.4.0486 (2018-02-15).
When it came back from the dead, it stated that the machine had rebooted unexpectedly, and the filesystem was not clean so a fsck was in order; I stopped the scrub (which had started again) I ran the check, and thankfully it reported no damage.
Is there any way to find out what happened? I can't find the actual system logs anywhere: /var/log/messages is empty and there are no utmp/wtmp files. Even getting some output from 'last' would be nice to see if it reports anything.
I'm concerned that for some reason the scrub may have triggered a kernel panic, and I'm not sure if it's worth the risk to even just try the scrub again manually.
Any advice would be most welcome.
I had set up my TS-431 with 4 disks in a RAID5 array to run a monthly scrub at midnight on the first day of each month, and it started just fine on March 1st, announcing it with a beep. About 26 hours into it, however, the NAS went beeping and disappeared from the network for a few minutes. The NAS is connected to a UPS with no other devices and I'm the only user, and hadn't accessed it at all. I am running QTS 4.3.4.0486 (2018-02-15).
When it came back from the dead, it stated that the machine had rebooted unexpectedly, and the filesystem was not clean so a fsck was in order; I stopped the scrub (which had started again) I ran the check, and thankfully it reported no damage.
Is there any way to find out what happened? I can't find the actual system logs anywhere: /var/log/messages is empty and there are no utmp/wtmp files. Even getting some output from 'last' would be nice to see if it reports anything.
I'm concerned that for some reason the scrub may have triggered a kernel panic, and I'm not sure if it's worth the risk to even just try the scrub again manually.
Any advice would be most welcome.
-
- Starting out
- Posts: 14
- Joined: Sun Jun 05, 2016 10:05 pm
Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
Hi all,
same problem here: TS-431 with RAID5 array (4 disks), monthly scrub scheduled at midnight on each month's 1st. Scrub started and system rebooted unexpectedly after about 6 hours. I stopped scrubbing, did an fsck (from GUI), and restarted scrubbing: again it rebooted unexpectedly after several hours.
One more time: fsck and restart scrubbing, and it rebooted again. I also tried to lower scrubbing priority, but it did not help.
After last reboot and fsck, I did not restart scrubbing; and I'm waiting to see if it reboots again.
If it won't I'd assume that scrubbing is involved in this behavior.
As already stated:
same problem here: TS-431 with RAID5 array (4 disks), monthly scrub scheduled at midnight on each month's 1st. Scrub started and system rebooted unexpectedly after about 6 hours. I stopped scrubbing, did an fsck (from GUI), and restarted scrubbing: again it rebooted unexpectedly after several hours.
One more time: fsck and restart scrubbing, and it rebooted again. I also tried to lower scrubbing priority, but it did not help.
After last reboot and fsck, I did not restart scrubbing; and I'm waiting to see if it reboots again.
If it won't I'd assume that scrubbing is involved in this behavior.
As already stated:
Thanks in advancejollino wrote:Any advice would be most welcome.
TS-431 + 4 x ST4000VN000-1H4168 RAID5
- Don
- Guru
- Posts: 12289
- Joined: Thu Jan 03, 2008 4:56 am
- Location: Long Island, New York
Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
Open a ticket with QNAP via the help desk app and upload the log files.
Use the forum search feature before posting.
Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.
NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.
NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
-
- Starting out
- Posts: 23
- Joined: Fri Dec 18, 2015 9:22 pm
Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
That's my question, how do we get the log files? I can't find any of the traditional Unix/Linux log files on the system.
- Don
- Guru
- Posts: 12289
- Joined: Thu Jan 03, 2008 4:56 am
- Location: Long Island, New York
Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
Use the helpdesk app. It will download and zip the files for upload to QNAP when it creates the ticket.
Use the forum search feature before posting.
Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.
NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.
NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
-
- Starting out
- Posts: 23
- Joined: Fri Dec 18, 2015 9:22 pm
Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
Oh, I see. Thank you very much, I never needed to use that until now so I had no idea. Will do.
-
- Starting out
- Posts: 23
- Joined: Fri Dec 18, 2015 9:22 pm
Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
Do you have any updates on this? Did you try it again? I opened a ticket and they replied quickly asking to do remote support on my unit, but it's been four days now and I'm still waiting.4ppr3ntice wrote:Hi all,
same problem here: TS-431 with RAID5 array (4 disks), monthly scrub scheduled at midnight on each month's 1st. Scrub started and system rebooted unexpectedly after about 6 hours. I stopped scrubbing, did an fsck (from GUI), and restarted scrubbing: again it rebooted unexpectedly after several hours.
One more time: fsck and restart scrubbing, and it rebooted again. I also tried to lower scrubbing priority, but it did not help.
After last reboot and fsck, I did not restart scrubbing; and I'm waiting to see if it reboots again.
If it won't I'd assume that scrubbing is involved in this behavior.
- storageman
- Ask me anything
- Posts: 5507
- Joined: Thu Sep 22, 2011 10:57 pm
Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
And what do the SMART checks show?
That the disks are fine?
That the disks are fine?
-
- Starting out
- Posts: 23
- Joined: Fri Dec 18, 2015 9:22 pm
Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
In my case they are, the check didn't report anything wrong and SMART is fine on all four disks. I didn't try scrubbing again though.storageman wrote:And what do the SMART checks show?
That the disks are fine?
- storageman
- Ask me anything
- Posts: 5507
- Joined: Thu Sep 22, 2011 10:57 pm
Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
I would run it and see what the processor/ram levels show too.
Clearly it should not reboot if the drives are ok.
Clearly it should not reboot if the drives are ok.
-
- Starting out
- Posts: 14
- Joined: Sun Jun 05, 2016 10:05 pm
Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
Once stopped scrubbing, NAS has been running for more than five days, before rebooting again; this happened because scrubbing was restarted by a scheduled activity. I'm still investigating which process did the trick, because regular schedule was monthly...jollino wrote: Do you have any updates on this? Did you try it again? I opened a ticket and they replied quickly asking to do remote support on my unit, but it's been four days now and I'm still waiting.
Now scrubbing is de-scheduled, for an abundance of caution.
I also opened a ticket with QNAP in the meantime, HDD section. Support stated that my configuration is not fully supported, from a storage compatibility point of view.
Although my disks are in the compatibility list (actually they are 4 disks of the same model), they need a firmware update in order to be fully compliant. To be honest, I did not realize that for more than two years after assembling the RAID array, because no problems arose.
Now I'm trying to get support from disk vendor: online automated download services don't show any firmware update available for my HDDs' serials. So I've emailed them asking for a deeper check (also because QNAP states that there is a newer firmware for them, by putting it in their compatibility list). Got a quick answer, with a link to a non-compatible firmware; replied to that, and I'm still waiting.
off-topic start
Just to understand how deep my fault was, two questions for all users.
How many of your preferred storage resellers can guarantee you in advance the firmware version of the disks you are going to purchase?
How many of you check the firmware version before mounting the disk in the bay, once you know that disk model is compatible?
off topic end
TS-431 + 4 x ST4000VN000-1H4168 RAID5
-
- Starting out
- Posts: 47
- Joined: Fri Mar 02, 2018 3:48 pm
Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
Same issie with TS-431 4.3.4.0486 and unexcepted reboot in RAID Scrubbing
My problem begins from 02 mar 2018
In the end of feb - firmware was updated from 4.3.3.0396 to 4.3.4.0486
In jan and feb 2018 - RAID scrubbing finished successfully
Maybe this is issue of firmware 4.3.4.0486?
My topic on forum:
viewtopic.php?f=73&t=139856
My problem begins from 02 mar 2018
In the end of feb - firmware was updated from 4.3.3.0396 to 4.3.4.0486
In jan and feb 2018 - RAID scrubbing finished successfully
Maybe this is issue of firmware 4.3.4.0486?
My topic on forum:
viewtopic.php?f=73&t=139856
-
- Starting out
- Posts: 14
- Joined: Sun Jun 05, 2016 10:05 pm
Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
Forgot to mention that, in all of my posts: firmware 4.3.4.0486 here too.
Looks like QNAP should take our issues into consideration: one coincidence is just a coincidence, two coincidences are a clue, three coincidences are a proof...
To get back at the original question in the topic: to me it looks like that most recent logs are in /mnt/HDA_ROOT/.logs, at least on TS-431.
Looks like QNAP should take our issues into consideration: one coincidence is just a coincidence, two coincidences are a clue, three coincidences are a proof...
To get back at the original question in the topic: to me it looks like that most recent logs are in /mnt/HDA_ROOT/.logs, at least on TS-431.
TS-431 + 4 x ST4000VN000-1H4168 RAID5
-
- Starting out
- Posts: 47
- Joined: Fri Mar 02, 2018 3:48 pm
Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
Kernel log analyzer in Qnap diagnostic tool show only one problem, but it was in nov 2015
You do not have the required permissions to view the files attached to this post.
-
- Starting out
- Posts: 14
- Joined: Sun Jun 05, 2016 10:05 pm
Re: Sudden reboot during monthly scrub, unclean filesystem; any way to see the logs?
No errors in the kernel log, no errors in SMART analysis, for me. I'm not that great log reader, but I haven't seen any lines easily identifiable as errors/warnings around reboot events.
As stated in thread title, reboots are "sudden"; the one and only signal, as far as I've seen, are the NAS "beeps".
As soon as I get the FW update (and if I manage not to brick anything... ), I'll force start scrubbing. If it reboots again, I'll go back to QNAP support (assuming that they did not do anything in the meantime, like a QTS update).
FYI, Seagate ST4000VN000 with SC46 firmware aboard HDDs are not fully compatible with TS-431; they need at least SC60.
As stated in thread title, reboots are "sudden"; the one and only signal, as far as I've seen, are the NAS "beeps".
As soon as I get the FW update (and if I manage not to brick anything... ), I'll force start scrubbing. If it reboots again, I'll go back to QNAP support (assuming that they did not do anything in the meantime, like a QTS update).
FYI, Seagate ST4000VN000 with SC46 firmware aboard HDDs are not fully compatible with TS-431; they need at least SC60.
TS-431 + 4 x ST4000VN000-1H4168 RAID5