Page 1 of 6

random reboot issues

Posted: Wed Jun 12, 2019 7:05 pm
by Moogle Stiltzkin

made a separate thread for this, from my other one which was originally a qts 4.4.1 review until i got hit by some random reboot issue.

the problem is one of a few things happens

- random reboot
- file system unclean
- raid synchronization
- raid rebuild

this happens between 1-3 days.... i can't keep the TS-877 up and running before i get hit by a random reboot :(

i listed some troubleshooting i've done so far here
just a recap on troubleshoot

- the problem is one or more of these things occurring randomly and regularly ( system unclean check file system, random reboot, raid synchronization, raid rebuilding )
- this trouble started occurring sometime after updating to qts 4.4.1 but in hindsight, prior to this, i did have some unexplained random reboots, but not nearly as often as now, or that i was aware of since i wasn't looking too closely at the logs.
- ram is highly suspected. due to not running memtest86 long enough, we can toss those results and await the new test results running for 72 hours or more. (pending)
- in logs the only hdd that ever got called out was a hdd4 hgst deskstar nas 4tb. this has since been replaced with a wd red 4tb.
- all hdds have long smart test , scandisk, and zero fill formated
- NAS is running QTS 4.4.1 and has already reinitialized cleanly (i did not do a dom recovery since i don't believe i ever got a malware issue, so was unecessary). fyi i got qts 4.4.1 running on ts-653a and it is stable, so i don't believe this issue to be qts 4.4.1 related.
- also the QTS has been installed in 2x256gb samsung ssd raid1.
- the random shutdowns still happen even when qts is run from the ssd's instead of hdds (after the renit). which makes me think that the ssd and hdds are not to blame. and is something else.
- i tried replacing the ram with kingston hyperx fury 16gb, but these were 2 x single sticks. NOT DUAL KIT. and it still had the random reboot issue. Considering it's not dual kit, chances are this is a simple matter of incompatible ram..... I need a different ram i can trust to be fully compatible for ts-877 to switch to, to see if it fixes the random reboots. (pending)
- yes i reported this to helpdesk in detail since last week. so far they said they found nothing. i provided dump logs to them as requested, and prior to renit we had a remote session active. For now all the advise i got is to report when it happens and send them the dump logs. i understand they need the data to analyze. Hopefully there is some more definitive answers as to what to do. Cannot have the NAS unstable and constantly rebooting randomly. NAS is unusable like this. (pending)
- thought it virtual switches could be a possible cause, so i removed them. but i still got the random reboot.

other suspicions (not very sure on this..... but what else is there)
- PSU ?
- CPU ?
- raid controller ?
- ram contact points and slot needs cleaning?

troubles started sometime after upgrading to qts 4.4.1 though i'm unsure if this is related, could just be a coincidence.

the full details of this issue is also in the linked thread, it's a big read but there is a lot of info posted in detail as to this nas instability leading to random reboots (which causes issues for filesystem unclean and raid....)

I'll however point out that this issue persisted even after a reinitialize, and setting up the QTS to switch from 4x4tb HDD, to a 2x256gb SSD. So i doubt this is a hdd/ssd issue because QTS is installed on different drives but still happens.

Toxic17 wrote:
Wed Jun 12, 2019 6:05 pm
I take it you are running memtest from the QNAP NAS? when testing ram modules. a one line answer will suffice.
Version 8.2 3/Jun/2019 pro queud 999 passes. so far 12 hours + no errors yet.

Re: random reboot issues

Posted: Wed Jun 12, 2019 8:56 pm
by dolbyman
if it is a 4.4.1 beta issue..I'd open a ticket..maybe some mix of hardware/software causes reboots

Re: random reboot issues

Posted: Thu Jun 13, 2019 4:00 am
by Moogle Stiltzkin
dolbyman wrote:
Wed Jun 12, 2019 8:56 pm
if it is a 4.4.1 beta issue..I'd open a ticket..maybe some mix of hardware/software causes reboots

yeah already :} first thing i did.

it's looking that way.

I got the memtest86 running , 24+ hours still no errors. i'm waiting for 72 hours before i call it.

After that gonna try cleaning the ram contacts and the slot, test again.

if still don't work, i'll try disable some of the apps. helpdesk and hbs see if that does anything.

then try a dirty downgrade. then try a renit clean downgrade.

i also found this which gives some ideas what to do ... ould-i-do/

by the way, if pool 2 raid5 4x4tb had a bad hdd (i doubt this because i already did the tests and replace hdd4 which is the only that ever got flagged for anything in the logs for the past few months), that would not cause the nas to crash would it? because after the reinit i had already moved the qts to the 2x256gb ssds, but it still happens. thats why i don't think it's hdd related.

also since the reinit, i managed to create raid1 and raid2 and restore from backup. and during that time it did not random reboot. only after i finished setting up the raid and a day or more went by did it start acting up again :S

all i can think of is that perhaps i had started to install some apps from appcenter, maybe that is the cause?

- linux stations
- virtual station
- helpdesk
- qumagie
- multimedia console (already had this installed but i changed from disabled multimedia to enabled. then change it to begin indexing)
- photo station
- video station
- codexpack

Re: random reboot issues

Posted: Thu Jun 13, 2019 2:44 pm
by Moogle Stiltzkin
i read this
I usually run memtest EVERY TIME I change CPU, memory or notice odd issues. Reason is - memtest does not just test memory. It will error out in three other important cases:

1) If there are issues due to overheating of either CPU or memory. Normally, memtest loads CPU for several of the tests to almost 100%.

2) if CPU cannot access memory correctly. Most of the tests read & write at all locations of the memory.

3) if there are problems with power. I.e. a weak power supply or bad capacitors cause too many spikes offsetting random bits.

If I see at least a single red error, I start looking for what should be removed / fixed before using the hardware. ... th-memtest

so can i rule out cpu and psu as well, as suspects if the memtest86 successfully runs 1-3 days no errors?

Re: random reboot issues

Posted: Thu Jun 13, 2019 3:34 pm
by Moogle Stiltzkin
Okay i got some excellent news.

I managed to get as replacement.
Crucial 16GB Kit (2 x 8GB) DDR4-2666 UDIMM

they are 100% sure to work for the qnap ts-877. yes it won't have the heatspread but ram doesn't really need it anyway. i just want something that just works and getting the NAS back up and running asap.

i was also offered an alternative which was an ADATA ram he said was same like mine, but i'm not sure if that would have worked (but they are slower ram :S )

Code: Select all

now i just got to send back the kingston and get the crucial in return.

i'm sure the kingston hyperx fury are fine (since trex and others use it without issue, also having a first hand look at it, it does look like quality ram), just make sure you get the KITversion which are verified working together.

new ram i'll probably just test it for a day max or at minimum 32 passes this time around.
To use all possible data patterns 32 passes are required. This test is quite effective at detecting data sensitive errors but the execution time is long.

I was only thorough with the original ram because i wanted to establish the cause was not the ram as to what is causing these random reboots :/ then i got a trusted stick to go back to in case of future random reboots that get blamed on bad ram (after i upgrade ram)

that said, i don't think i have a ram issue.... memtest running for 34+ hours no errors. so new ram only fixes my low ram capacity which is what i needed to do anyway. but i doubt this will solve my random reboot issues :?


50 hours+ memtest86 no errors still.

1 more day to go.... :( ram testing su-cks, but it probably beats the patience required to farm cosmic wings in diablo 3, so i'm bearing with it. But i won't ever test beyond 3 days max going forward. all anyone does is 4passes, 32 passes, 1 day or 3 days tops afaik. or even better, they just swap in ram they KNOW works, to rule out issues with ram. But i never ran a ram test before (this is my first time) so.... :(

Re: random reboot issues

Posted: Thu Jun 13, 2019 4:25 pm
by GTunney
Like i said my memtest ran fine for nearly 24 hours yet the only thing to solve my reboots was to go with the crucial ram I have now.

Below is a brief overview of my daily uptime, I changed my ram on the 18th May, 30th of may was when I upgraded to QTS 4.4.1 and 8th June was when I rebooted thinking I had an issue but it was actually my AVR.

As you can see as soon as the ram was changed the uptime % has been perfect. For such a small cost it's worth a go to try something completely compatible.

Code: Select all

13th Jun 2019	93	100%
12th Jun 2019	268	100%
11th Jun 2019	251	100%
10th Jun 2019	283	100%
09th Jun 2019	282	100%
08th Jun 2019	282	99.29%
07th Jun 2019	262	100%
06th Jun 2019	276	100%
05th Jun 2019	271	100%
04th Jun 2019	279	100%
03rd Jun 2019	280	100%
02nd Jun 2019	283	100%
01st Jun 2019	285	100%
31st May 2019	281	100%
30th May 2019	281	99.29%
29th May 2019	281	100%
28th May 2019	286	100%
27th May 2019	291	100%
26th May 2019	290	100%
25th May 2019	285	100%
24th May 2019	280	100%
23rd May 2019	284	100%
22nd May 2019	282	100%
21st May 2019	283	100%
20th May 2019	284	100%
19th May 2019	284	100%
18th May 2019	281	95.37%
17th May 2019	274	79.56%
16th May 2019	271	76.01%
15th May 2019	283	97.88%
14th May 2019	280	77.5%
13th May 2019	285	99.65%
12th May 2019	282	100%
11th May 2019	279	99.64%
10th May 2019	283	93.64%

Re: random reboot issues

Posted: Thu Jun 13, 2019 4:31 pm
by Moogle Stiltzkin
thx gtunney, i'll be joining the crucial club soon :} hope to share my results as well.

avr? :' oo u meant ups i see :)

Re: random reboot issues

Posted: Thu Jun 13, 2019 5:41 pm
by GTunney
Moogle Stiltzkin wrote:
Thu Jun 13, 2019 4:31 pm
thx gtunney, i'll be joining the crucial club soon :} hope to share my results as well.

avr? :' oo u meant ups i see :)
Yes I lost all sound from the QNAP and rebooted it thinking it was the nas but it was actually my amp playing up.

Re: random reboot issues

Posted: Thu Jun 13, 2019 5:53 pm
by storageman
Wise words...

One day Qnap might be a bit more sensible on their own RAM pricing!

Re: random reboot issues

Posted: Thu Jun 13, 2019 6:39 pm
by Moogle Stiltzkin
what ram do you use o-O: ?

i've only ever used the stock ram. this is the first time i'm upgrading ram :D

I notice from sigs, a lot of qnappers tend to prefer sourcing their own ram :mrgreen: it kinda helps others know what works or not.

crucial is the only vendor i know that has a compatibility/validation list for QNAP NAS (beside the official qnap store), which adds some peace of mind when you buy sticks from them.

Not doing anything different than what other users are doing 8)

Re: random reboot issues

Posted: Sat Jun 15, 2019 1:31 pm
by Moogle Stiltzkin
72 hours memtest86 completed. no errors....

booted up... first thing i did is disable non essential apps. also considering instability others reporting in regards to indexing, i also disabled indexing and multimedia in multimedia console. gonna wait for the next firmware that fixes it before i enable it back again.

did file system checks for all 3 volumes. then re-ran the qts 4.4.1 firmware. Rebooted and now it's 6 hours+ uptime no issue yet......

Not sure if this is going to be enough :S (still waiting on new ram to arrive)

nas would have to have a 100% uptime for a week to a month before i let down my guard :/ thats why i'm not storing any new on it just yet.


1 day up-time no issues yet.

I'll however mention that i thought i heard a beep from the qnap, but when i checked logs i did not see any event occur. so maybe i misheard :S

i wiped my share data and am recovering again from backup with HBS. no interruptions yet. i'll give it a week up-time before i give the all clear.


less than 2 days uptime before random reboot.

this time around it did so during a hbs recover :(

booted up, it did raid resync (probly faster due to bitmap enabled beforehand), ran file system checks on vol1 and vol2. updated firmware back to qts 4.3.6 to see if this helps.

leaving system running and seeing if the same things happens. if it does, i may have to try a clean reinit.


did a clean reinit instead (using qts 4.3.6 latest stable). because i noticed some left over stuff from qts 4.4.1 like the nvidia drivers that i couldn't remove. so if i'm going to test if it's stable or not, may as well be on a clean renit.

uptime 22 hours no random reboots...yet.... :'

i'm only doing the barest necessary settings that i normally do. and doing the hbs (using latest hbs 2 stable. not the hbs 3) to recover from backup.

when thats done, just gonna be streaming from nas and leaving it up and check that uptime can get over a week or month without random rebooting.

other things i am doing is waiting on new ram (still) and also ordered for electronic motherboard/contact point spray cleaner, so i can try cleaning the ram slots just in case.... (but it ran the memtest86 for 3 days no errors, so i'm very doubtful this is the cause).


helpdesk thinks the problem is ram. so gonna wait for new ram before i can replace :(

Re: random reboot issues

Posted: Mon Jun 17, 2019 7:10 pm
by GTunney
When is the new ram due to arrive? The suspense is killing me :D

Re: random reboot issues

Posted: Mon Jun 17, 2019 7:50 pm
by Moogle Stiltzkin
GTunney wrote:
Mon Jun 17, 2019 7:10 pm
When is the new ram due to arrive? The suspense is killing me :D
funny story with the ram....

i bought the 16gb kingston hyperx fury thinking i was getting the dual kit. but when it arrived i see that these were the single sticks. the seller just put 2 together then called it a kit despite it not being kit. A kit is packaged 2 sticks together with the specific part number to point out they are the kit variant.

i mean they installed fine, and nas booted up. but my NAS random shutdown eventually few hours later... so.... :S

so i sent that back for replacement. supposedly i was to get the crucial ram
16gb (8x2)

but he said the one he is going to send is this


i already clarified i want the kit (CT2K8G4DFS8266) and not the single sticks (CT8G4DFS8266)

i found this on the subject
Dual channel memory works most reliably if you use two identical modules: RAM from the same manufacturer, the same timing specs, and the same manufacturing lot. You can often get it to work without doing that, but the more difference in timing you have between the modules the less reliable it is likely to be.

There is no difference between buying a kit of two modules and buying two identical modules separately. The catch is that there is no guarantee that the two separate modules are identical, even if they are from the same brand and have identical specs. There are multiple manufacturers of RAM chips, and most of the popular brands of RAM use chips from more than one of them. (Crucial is the notable exception; it is owned by Micron and only uses Micron chips.) Chip fabricators also change their designs and process from time to time, so there might be subtle differences between older and newer chips or chips from different fabs.

Usually, if you go to a store and buy two sticks of the same brand with the same specs you will be fine. But buying the two module kit is the surest way to get memory that will be happy together, and the price is usually about the same as buying the two modules separately. If you have the rare system that implements quad channel memory, buying a kit of four modules is the safest bet. Intel had one motherboard chipset that implemented THREE channel memory some years back, but I doubt anybody is packaging kits with three RAM sticks now. ... -32-GB-kit

i want the kit because i don't want to play russian roulette :S

i intend to use the ram in a dual channel configuration.

so anyway right now seller is gonna send ram... i just don't know what i'm getting at this point :( sigh.

tbh i dont often upgrade ram, so maybe i'm being overly paranoid about this, i haven't installed enough ram to know enough. i'm just reading what others are saying on this subject and relying on that.

other qnap upgraders just buy the dual kit and are done, full stop. i have to worry about this seller who is insisting on selling a non kit for the ram :S

i just want what i want, and not being pushed to get something else :roll: i don't need to justify my purchase. next time i'm buying direct when possible so i don't have to put with this nons-ense.

it took less than 3 days to ship back the ram, so i'm estimating i can get my replacement somewhere between 1-2 weeks at this rate :(

Re: random reboot issues

Posted: Mon Jun 17, 2019 8:45 pm
by Toxic17
just a thought, have you run the diagnostics tool app on the NAS and then inspected all the dump files?

also if you haven't ye done this, run the disk and memory stress tests.

Re: random reboot issues

Posted: Mon Jun 17, 2019 9:03 pm
by Moogle Stiltzkin
Toxic17 wrote:
Mon Jun 17, 2019 8:45 pm
just a thought, have you run the diagnostics tool app on the NAS and then inspected all the dump files?
yes i generated the dump and sent it forward to helpdesk. this is before i reinitialized.

they now think the issue was the ram (despite the 3 day no error memtest86 results). I'm not sure if they are just simply saying that (since i mentioned how gtunney solved his similar issue), or whether they did a proper troubleshoot and found something from the dump i sent.

anyway i would not know how to use the dump to perform any meaningful analysis to troubleshoot myself, i'm not an IT expert (main reason i bought a qnap so i don't have to deal with the technical stuff in detail). all i could tell was that a random reboot happened, and looking at the basic log it just simply said filesystem unclean, and just booted back up (essentially a restart) :S and following that, sometimes you also get raid resync (doesn't take too long) or a full raid rebuild (takes 13 hours to complete).

best i did was run that memtest86, and do smart test/scan disk, reinitialize, so far. now i'm just waiting on the new ram and just leaving nas up to see if the problem is still happening. I reinit to downgrade from 4.4.1 to 4.3.6 latest stable clean setup to check whether the issue is not simply (firmware related because of beta status).

QTS 4.3.6 latest build, with 1 day 4 hours up time, no random reboots yet.

Toxic17 wrote:
Mon Jun 17, 2019 8:45 pm
also if you haven't ye done this, run the disk and memory stress tests.
i did run the memory test in the diagnostic but it finished quick and came out no errors. i wasn't sure it was effective because it completed so soon. so thats why i used memtest86 instead.

as for diagnostic stress test no i did not do that. because i donno how to properly use that to check for issue. the disclaimer said to let the helpdesk support use these tools. thats why i didn't use it. all helpdesk wanted from me was dump files, they didn't ask for anything else :S there was 1 remote session at 1 time, but that was about it (i donno what troubleshooting they were doing at the time).

But ok i will take a look at that and try and run it. Just need to wait for my recover job to complete first :)

fyi the random reboot can happen during regular usage e.g. streaming (watching videos), running HBS, or even simply just an idle NAS not doing much.