QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Introduce yourself to us and other members here, or share your own product reviews, suggestions, and tips and tricks of using QNAP products.
raidforsaving
Easy as a breeze
Posts: 252
Joined: Mon Jul 14, 2014 9:14 am

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by raidforsaving » Fri Jan 29, 2021 5:55 am

Part of me wonders if this issue is why many NAS devices have had random crashes over the years. It has been about a month and no issues on my new NAS, even though the drives, data, and apps are the same (upgraded admittedly). But I was upgrading that NAS QTS for years.
Model name: TS-x53D
Firmware version: 4.5.1.1495(20201123)
Volumes: EXT4, Encrypted volume
Previous: TS-x51 dead by CPU bug

Kngstnguy
New here
Posts: 7
Joined: Thu Oct 17, 2019 8:25 pm

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by Kngstnguy » Sat Jan 30, 2021 12:09 am

I'm not liking the fact that QNAP continues to advertise the TS-251+ and the TS-451+ for sale on their site, despite this issue.

I own a 251+, and when I submitted a support ticket about remedies like a reduced price on an extended warranty or a discount on another system, they say they're not aware of this issue.

How many who posted they had this issue submitted support tickets for this?
TS-251+: 2x 4Tb WD Red, 16Gb ram
TS-453D: 4x 4 Tb WD Red, 16gb ram

kibkib
Starting out
Posts: 18
Joined: Sun Oct 30, 2016 8:58 pm

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by kibkib » Mon Feb 01, 2021 2:47 am

I have TS-453b (not PRO),just red blinking, not passing the POS, no beep, no display.
Could it be affected by the CLK issue and fixed with the resistor? The LPC_CN1 connector seems to be different.

Brandon_K
New here
Posts: 2
Joined: Sat Jun 27, 2020 4:33 am

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by Brandon_K » Tue Feb 02, 2021 2:19 am

Just wanted to add another reference point. My TS-651 beeped and became unresponsive on the 30th. No status light, all HDD LED's red. Dead.

I wanted to throw a huge thank you to yujungchang for doing the initial leg work and soulcages01 for throwing up the pic. I grabbed a 100ohm out of my resistor drawer, soldered it on and voila, back up and running. I'm in the process of building a ZFS on Unraid NAS (DL80 G9, 24x2.5, 24x3.5" disk shelfs), it would figure that my Qnap dies days before transferring everything over. I'm super happy to get up and running again, at least long enough to copy the data over. You guys are rockstars.

nas_down
First post
Posts: 1
Joined: Wed Feb 03, 2021 7:06 am

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by nas_down » Wed Feb 03, 2021 7:25 am

It happened to me yesterday on my TS-453 PRO. Red status light on NAS and loss of the LAN connection, now it stuck on system booting. This wasn't the result of a power outage as other devices on my UPS have an 11week uptime. I noticed LACP was suspended towards my switch.

I rebooted it several times with and without drives. After playing around with it for hours, I am able to log into the system through HDMI on the linux shell (developers console). However, it still will not finish booting and the NIC is getting a link local address (instead of the static configured). I opened a ticket yesterday but still waiting to hear back. Only my custom user account (with limited permissions) works and not the admin account, strange.

mlinsemier
New here
Posts: 8
Joined: Thu Mar 29, 2012 11:05 pm

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by mlinsemier » Sat Feb 06, 2021 11:49 pm

I find it interesting that I experienced the exact same issue with a TS-453 today at 8:43am, so close to others experiencing failure. This seems less of a coincidence and more of something else going on. I also will open a ticket with QNAP once I can get access to the physical NAS.

AlastairStevenson
Experience counts
Posts: 2315
Joined: Wed Jan 08, 2014 10:34 pm

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by AlastairStevenson » Sun Feb 07, 2021 4:32 am

This seems less of a coincidence and more of something else going on.
The cause of this problem is a reliability issue with a specific series of Intel CPUs.
Some interconnect on the silicon die is running at an excess current density and is therefore subject to a migration problem that shortens the in-service lifetime.
It's not just certain QNAP NAS models that are affected, there are many other manufacturers and devices that have been affected by this failure mode.
TS-431+ for storage and media and a bunch of IP cams under Surveillance Station. TVS-473 as files backup and QVR Pro.

kantzow
New here
Posts: 4
Joined: Thu Feb 21, 2013 9:21 pm
Location: Sweden

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by kantzow » Mon Feb 08, 2021 12:24 am

My TS-453 Pro just conked out, bought it late 2015, been running 24/7 with 4x WD Red drives in RADI5, by now I've replaced 3x of them with recertified drives from WD (warranty replacements). Been reading through the thread and can now happily report I've got my TS-453 Pro back operating normally after soldering in a 100 ohm resistor between pin 1 and 10 on the LPC, good job to everyone involved in finding the solution!
Model: TS-453 Pro FW: 4.5.2.1566
Disks (HDD): [RAID-5] 4 x 3TB WD Red
RAM: Corsair Value Select 8GB 1600MHz DDR3L SDRAM SO-DIMM 204-pin

Model: TVS-673e FW: 4.5.2.1566
Disks (HDD): [RAID-5] 4 x 4TB WD Gold DC / 1 x 8TB WD Ultrastar
RAM: Stock 8GB

alexvan
New here
Posts: 2
Joined: Mon Feb 08, 2021 3:12 am

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by alexvan » Mon Feb 08, 2021 5:59 am

Hi,

I'm sorry for this long message: I just believe it's what many of us have in mind and that it can be useful to help us.

Got exactly the same issue on a TS251(end of 2015) and the resistor fixed it (100 ohms). Thanks to @yujungchang for sharing it, really appreciated! For some readers: yes, it's a pretty easy fix => don't be afraid.

I've understood that it's a temporary solution since the main issue will continue to evolve and then, the problem will happen again. I've updated my "offline backup" that was one month old. Like many others, I'm now considering my options:

Option 1: buy a new NAS now, keep the same disks
I'm in favor of not throwing any device until it's really died just because I believe it's better for the planet. I can live without my NAS for few days/weeks (it's for home purpose) so, even if it dies again, that's ok. Last, getting a new one doesn't prevent me to have another issue in few years. I would then discard option 1.
For someone else who needs guarenteed high availability, I would believe this option is the best/recommended one.

Option 2: keep my 'repaired faulty" NAS, continue "regular offline backup" and wait until next crash
Expectations on the duration?
I've read on Wikipedia that the issue is known since Q4 2016 (roughly 4 years ago). Do we have any feedback on the possible duration of this "fix"? Are we talking about weeks? months? years? Or is it really something we can't even estimate?
Risks?
Is there any risk of this fix? (damaging the harddrive, others?)
New crash workaroung?
When the crash will happen again, is there any chance that changing the resistor value fixes the issue for another time? (putting 150 ohms as an example?)

Option 3: do a more expensive fix (replace the cpu)
I've read a post from another users who changed the CPU. Since it's soldered, it's not an easy fix for someone who is not an expert in soldering. However, is it a durable solution or the new processor will have the issue too after some months?

Thanks for any reply that can help!

Mousetick
Been there, done that
Posts: 949
Joined: Thu Aug 24, 2017 10:28 pm

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by Mousetick » Mon Feb 08, 2021 9:18 am

alexvan wrote:
Mon Feb 08, 2021 5:59 am
Option 3: do a more expensive fix (replace the cpu)
I've read a post from another users who changed the CPU. Since it's soldered, it's not an easy fix for someone who is not an expert in soldering. However, is it a durable solution or the new processor will have the issue too after some months?
Forget about it. The affected CPUs are SoCs (system on a chip) intended for use in embedded systems, mobile devices and appliances by hardware manufacturers and integrators. They are not sold at retail. You, as a consumer, can't buy a brand new one. Hypothetically you might be able to buy from a reseller or distributor, if you order a large quantity.

SunLL404
New here
Posts: 2
Joined: Thu Jul 09, 2020 11:47 am

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by SunLL404 » Tue Feb 09, 2021 2:29 am

After having a TS-653 Pro die, I was looking through the hardware.
The drive backplane appears to be a PCIe x4 connector. The chips on it are Marvell SE889215
Anyone tried to use the backplane on another motherboard?
Might be ugly, but if you could leave the cover off, run a pcie ribbon cable to new motherboard and have 6 sata hotswap slots on the dead chassis. Might be work experimenting.

yujungchang
Starting out
Posts: 25
Joined: Sun Sep 20, 2020 11:54 am

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by yujungchang » Tue Feb 09, 2021 3:39 am

alexvan wrote:
Mon Feb 08, 2021 5:59 am
Option 2: keep my 'repaired faulty" NAS, continue "regular offline backup" and wait until next crash
Expectations on the duration? Do we have any feedback on the possible duration of this "fix"? Are we talking about weeks? months? years? Or is it really something we can't even estimate?
Is there any risk of this fix? (damaging the harddrive, others?)
New crash workaroung? When the crash will happen again, is there any chance that changing the resistor value fixes the issue for another time? (putting 150 ohms as an example?)
From the Synology comments(eeblog youtube video) I've read, I've seen people said few months to over couple years, so YMMV.
Turn it off when not in use is the best way to conserve its life.

The risk to damage the HDD is extremely low, since the LPC is mainly for the communication to the SIO and it's a completely different interface than the SATA used by the HDD. The worst case is the SIO behaves erratically again from further LPC degradation and the system resets itself while writing to the HDD. Most of your data should be intact and the RAID setup should be able to recover most of the failures, just like a random power outage. Since the degradation is happening gradually, hopefully you will notice the fan/temp error before the next failure (reset/failed to boot). When it failed again, you want to lower its value, not up. If you capture the LPC CLK and read the electrical spec, I think that 100 Ohms is already reducing the LPC clock voltage to its limit. However some have tried ~85 Ohm to get their system running, so we know at least you can lower it down further to 80~85 Ohm.

alexvan wrote:
Mon Feb 08, 2021 5:59 am
Option 3: do a more expensive fix (replace the cpu)
I've read a post from another users who changed the CPU. Since it's soldered, it's not an easy fix for someone who is not an expert in soldering. However, is it a durable solution or the new processor will have the issue too after some months?
I guess the new stepping should have fixed the issue, otherwise Qnap will probably stop selling it. The latest is the D1 stepping (SR3V5) and I saw it was sold on Ebay inexpensively, $31.5~$34.5. However the tricky part is to get it replaced and most people won't be able to do it easily or even find one that can do it. I saw a comment that people have to update the ucode as well in the BIOS, see this how to for reference https://www.win-raid.com/t4032f47-GUIDE ... l-Hex.html. Having said that, it's an inexpensive experiment, but the risk is extremely high. Expect to spend time and don't expect the success on your 1st try. Might have to try a few times until you succeed or the board is completely ruined.

If you really want to do it, I think this is the low cost way (relatively speaking) that I've seen ... The set up is not complicate(youtube has some DIY videos) and the actual rework time is less than 10 minutes (not including the prep time, like dry up the board/CPU under heat/oven), but just not sure about the rate of success. Basically you heat the board from the bottom to 180°C for several minutes and then heat the top to 220°C in next couple minutes to remove the CPU, clean up the pads and heat the CPU in place up to 180~220°C. I'm also interested to hear if someone has actually done this and might consider doing this in the future as well (for fun and not expecting it to work).
Electric griddle ($10, next Black Friday or local used one)
Thermal sensor (or $2~15, 3D printer/Arduino controller)
Solid state relay ($5/each) to control the electric griddle. I think the top might not be necessary, since you will be watching the temp carefully not to exceed 220°C.
Harbor Freight heat gun ($17 w/o coupon), or modify with the SSR to control the precise temperature
CPU from Ebay/Aliexpress (might get a used one sold as new). If it's really a new one and not the used pre-ball, then you will need to reball(need $2 stencil and solder paste) as well.
Last edited by yujungchang on Wed Feb 10, 2021 3:50 pm, edited 1 time in total.
[How to] Recover no beep / dead Intel based Qnap x53/x51 (2014~2016?) viewtopic.php?f=45&t=157459
[How to] Debug the SATA backplane power issue (models w/ a brick power supply viewtopic.php?f=50&t=157649
[How to] Enable onboard SATA connectors on the x53/x51 (Total 4+2 SATA drives for a 4 bay model) viewtopic.php?f=97&t=157816

alexvan
New here
Posts: 2
Joined: Mon Feb 08, 2021 3:12 am

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by alexvan » Tue Feb 09, 2021 4:08 am

Hi,

Getting such a complete answer is awesome and it really helps. I will go with option 2 since the risk is low and maybe, it will last for years (fingers crossed).
I've read most of the comments of this video from EEVBlog and it seems totally fair to say it will last at least for many months: https://www.youtube.com/watch?v=Y8_emfoR_MI

If it fails again, I guess I will reconsider option 3 (changing cpu). However, I believe I don't have the skills today and, if I need to go to a repair shop, I'm pretty sure buying a new NAS would be cheaper (I have a TS251 that was sold 240€ 5 years ago). In France, it would cost at least 150€ + new proc.

Personally, my main take away is that neither Synology nor QNAP are good brands for supporting their customers. Even if the faulty piece is an Intel one, they should have done something for their customer, at least communicating.

kibkib
Starting out
Posts: 18
Joined: Sun Oct 30, 2016 8:58 pm

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by kibkib » Wed Feb 10, 2021 2:54 am

I have a different device TS-453b, just red blinking, not passing the POS, no beep, no display.
Could be the same CLK issue, but there are no clock signals on any of the connector pins (should be around 25Mhz, right?).
There is a chip IT8528E nearby which should have PCICLK on pin 13 according to datasheet, but there's no signal too.
So probably the main processor is dead.
ts-453b.jpg
You do not have the required permissions to view the files attached to this post.

meerkat00
New here
Posts: 5
Joined: Wed Jan 13, 2021 10:05 am

Re: QNAP TS-453 Pro stuck on "SYSTEM BOOTING"

Post by meerkat00 » Wed Feb 10, 2021 3:11 am

alexvan wrote:
Tue Feb 09, 2021 4:08 am
... Even if the faulty piece is an Intel one, they should have done something for their customer, at least communicating.
Good point!

Post Reply

Return to “Users' Corner”