TVS-873 hangs on 'booting the kernel'

[Barend]

Hi,

This morning I couldn't access my NAS (TVS-873) from the network - and the display on the NAS did not show anything. The volume is quite full so I first assumed it might be that.

I rebooted and the display on the NAS kept saying 'booting...' for an hour.

I removed the 8 drives, and rebooted again, but same result.

I connected an HDMI display and there's a "Wrong EFI loader signature" warning but from what I've read I can ignore that.

It follows with:

"Decompressing Linux... Parsing ELF... done.
Booting the kernel."

The HDMI display shows the cube logo and says "Starting"

And this is where it stays.

Is the next step to attempt a firmware recovery? Or are there other steps I should attempt first?

Thanks for any insight,

Barend

TVS-873
8x WD Red 8TB drives

[Barend]

I grew a little impatient - so I tried to follow the descriptions here: https://wiki.qnap.com/wiki/Firmware_Recovery to see if I can get access to the bios at all, but connecting a keyboard and keeping [Del] down does nothing. The keyboard is recognized because [Alt][Ctrl][Del] does force a reboot. But I haven't been able to get into any kind of boot menu. It's just stuck at 'Booting the kernel'.

[mike85]

I just experienced the same problem with my QNAP (TVS-873, it was running smoothly last 4 years, maybe 1 unexpected reboot per 6 months). I think 4 years shall not be much for NAS device.

Briefly: QNAP TVS-873 hanged 3 days ago (not accessible via LAN/buttons), attempts to restart it lead to a non-bootable state (with and without disks same effect: after powering in 30 sec it show "Booting..." for 10+ minutes. And on HDMI monitor it shows "Booting the kernel" - see attachment.

Side-notes:
- QNAP is not exposed to the internet, in local network there is no PC with uncontrolled installation of Software (that potentially might attack the QNAP NAS ports)
- I don't install any tools/packages that might damage QNAP OS or something in it's config.
- I clean dust on year basis from it to avoid overheating
- temperature of the NAS and fan speeds were stable last year - no spikes, and they rotate fine now (had custom monitoring via Grafana and SNMP).
- NAS has 8 HDDs, last smart checks ~2 months ago were all GREEN (and NAS stopped booting without HDDs too)

Problem/how it appeared in details:
1. QNAP hanged without any reason (become not accessible via LAN, and not reacting on buttons nearby to the display).
2. I pressed the power button for 10 sec to shut the QNAP forcefully.
3. Then waited 10 sec, and pressed power button - QNAP booted normally. I waited maybe 15 minutes being busy, then tried to check via WEB the admin console of QNAP -> was noticed that the RAID is in rebuilding state (was something like "2% done, ETA 9 hours")
4. In one hour noticed that HDD leds are not blinking (so RAID rebuild stopped). So - the QNAP was in hanged state again.
5. Again pressed power button for 10 sec power NAS down.
6. After this - booting is hanging after 30 seconds (tried to remove all HDDs, keep 1 RAM stick, change RAM sticks - same effect - see attached picture of the timeline).

Any hints? Any guidelines what to check/probe (I probed quite some suggestions - none improved/changed the situation, I even was not able to reach it via terminal to check disks statuses or dmsg or anything like this). Are voltages fine on VINs in BIOS?

Thank you for possible ideas/suggestions!

[tvs873_user]

I also have a TVS-873 and started having the same exact issues at the same time. Really odd coincidence if you ask me.

Have been working with support on a fix. They have been very responsive and helpful, but nothing is resolved yet.

Initially I was able to get into the web panel for a minute or two and it was doing a full raid sync. Very shortly after the network dropped completely and was inaccessible. Tried multiple NICs with no success.

One thing I noticed early on was the status light turned red after a minute or so. That plus the freezing lead me to think the CPU might be overheating. With a machine that runs 24x7 after 4 years the thermal paste may have needed to be replaced. I very carefully cleaned off the old paste and applied a dab of new high quality paste. After that the red Status light went away but I'm unable to boot to anything and the keyboard becomes useless after just about 30 seconds.

[mike85]

tvs873_user: looks like it is exactly the same situation! I didn't describe this (as it was already too much details), but when I tried to reboot 2nd time, I was hearing that coolers started to spin to max level and in 1 minute QNAP shut down automatically, after this attempt all following ones were same - as I described and you are describing - after 30 sec device hangs (and even keyboard turns-off leds and stops reacting to caps-lock). And I also thought that it could be overheating, but last 2 years I had graphana with temperatures of CPU, System and HDDs - all were stable (not trends in increasing by degrees every month). So - I believe that the hardware shall be fine - it is more on SW side of QNAP (IMHO).

To QNAP: I assume it will be more people with this effect. That is a disaster, together with update by force feature by pulling updates in an uncontrolled way. QNAP - how to solve this situation?

[dolbyman]

QNAP does not come here..so no need to 'address' them ..open a ticket please

[tvs873_user]

@mike85: I'm also dead in the water but hopeful QNAP support can help. It's helpful to know that I'm not the only one in this strange situation. Have been building computers for decades and haven't run into anything quite like this. I'm also hopeful this is a software-only issue, but at the point we're looking at something that appears to be pretty low level. Complete freeze up and disabling of peripherals after 30 seconds is very odd.

[mike85]

@tv873_user: opened a ticket to the support. Hoping for some help/hints.
What's your progress? Improved or you have more advanced plan (e.g. buy a new similar QNAP model and move HDDs there?)

[mike85]

@tv873_user: any updates to your journey with qnap support. I'm to the moment unhappy with the process (got an ask to pay almost the full price of the device taking into account that the model is from 2016). What I don't like is that first your device was forcefully updated (~Dec2022), then in ~4 months 2+ users have same issue (that looks like as a software side-effect of an unreliable roll-out of a broken fix), and then ask to pay $820 for a fix. If I have device which I don't touch, don't expose to Internet, that simply runs for 4+ years, and then one day if fails with exactly same way as same device of another user - coincidence? don't think so...
What I don't understand - does QNAP think that it's reputation could not be even worth than now? I have an impression that last year QNAP wants to patch QTS "on-the-fly" to bug-fix security holes. But sometimes these updates go wrong and brick the devices. And, no problems, we will fix stuff for you - just send the device to us, pay $800, and hope for a success.

[tvs873_user]

@mike85 - Same as you. They said i could pay $820 for them to take it back and fix it, but wouldn't be able to help further. Obviously this is way too much money for a piece of hardware from 2016. Given the proprietary hardware, operating system, and LVM configuration I can't buy another QNAP with this level of support. I have a 12 year old desktop that took everything i through it for years -- including gaming and a little mining -- and still runs well, but the 3.5 year old QNAP is completely dead.

I agree our devices both dying at the same time with the same firmware is more than a coincidence, but support didn't care. Maybe if more people speak up with the same issue they will, but who knows.

What also bugs me is that the upgraded version of this, the TVS-872x is $2,000 and has a generation 8 inter processor. That is from 4+ years ago.

I'm going to just build a Linux server and use that. At least if something goes I can fix it with commodity hardware. QNAP doesn't sell boards for self repair.

[dolbyman]

Not that the x72 units have no faults...

viewtopic.php?f=50&t=153097