NAS not booting - RAMdisk and/or system partition full? Desperately trying to find help, I am fine paying for services

Questions about SNMP, Power, System, Logs, disk, & RAID.
Post Reply
hugo_b
New here
Posts: 5
Joined: Sun May 09, 2021 4:37 am

NAS not booting - RAMdisk and/or system partition full? Desperately trying to find help, I am fine paying for services

Post by hugo_b »

Dear all,

Do you know somebody, or maybe yourself, who can help me with my Qnap NAS tomorrow (Sunday May 9th). Of course, I pay for services. We are a decent small engineering firm. I will also send this message to a few very active members on this forum, I am desperate.

We use on a Qnap TS1277XU-RP running on QTS 4.5.3.1652. I can access it via SSH, but that’s all. I’ve already created an incident by Qnap, but so far my experience with Qnap support is that it will take some time for them to come to an answer.

Current emergency
The NAS is not booting, we cannot access our files. After a soft reboot, because of some errors which I will explain below, the NAS did not come up. Now the status light is permanently red, the network light blinks fast orange. The 10GBE network card is not working, but the build in normal RJ45 ethernet port is working. I cannot login at the web interface, but I can SSH into it. See screenshot 2a for status lights.

History
There were 3 signs things were going wrong:
1. Screenshot 3a Sometimes I got the message "A system failure occured. The device has insufficient system storage. Check the following settings: RAMdisk (/ or /tmp)". The NAS was still accessible via Windows SMB. I was in contact with QNAP about this issue but nothing came out of it yet.
2. Screenshot 3b1, 3b2 and 3b3. After and/or during those error messages the CPU, RAM and network usage dropped to zero. This was incorrect information. Files shares where accessible during those moments and there was network traffic on the selected NIC.
3. Screenshot 3c1 This happened this morning and was for me the reason to reboot it. I got the message “A system error occurred. The device has insufficient system memory/storage. Check the following settings: system partition” and later screenshot 3c2: “System error found”. After this soft reboot it did not come up.

Possible cause of insufficient ramdisk or system partition capacity
The HBS3 backups were not running smoothly and on screenshot 3b3 you see HBS2 consuming almost 1Gb of memory. The system has in total 64GB memory. Maybe HBS3 is the cause?

Data
There are a huge amount of small files on the NAS with a complicated user and usergroup design. Due to the backup issues of HBS our latest backup is a few weeks old. So far I believe our data is fine. I did not press the reset button for 10 seconds.

No Qlocker
The NAS is not exposed to the internet, UPNP is off, the NAS is only via VPN accessible for homeworkers. All fancy stuff is turned off as much as possible.

Begging for help
Please assist. I am of course willing to pay for support. Do you know somebody who could help me. My phone number is +31 6 50 500 822. My email address is hugo@breuerscobelens.nl.

Thanks for reading this and thanks in advance for your response?
You do not have the required permissions to view the files attached to this post.
User avatar
Toxic17
Ask me anything
Posts: 6469
Joined: Tue Jan 25, 2011 11:41 pm
Location: Planet Earth
Contact:

Re: NAS not booting - RAMdisk and/or system partition full? Desperately trying to find help, I am fine paying for servic

Post by Toxic17 »

you could have numerous issues, but can you tell me is the 64GB of ram on the compatible list for your NAS?

also is there any file in this folder:

/mnt/HDA_ROOT/update
Regards Simon

Qnap Downloads
MyQNap.Org Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following


NAS: TS-673A QuTS hero h5.1.2.2534 • TS-121 4.3.3.2420 • APC Back-UPS ES 700G
Network: VM Hub3: 500/50 • UniFi UDM Pro: 3.2.9 • UniFi Network Controller: 8.0.28
USW-Aggregation: 6.6.61 • US-16-150W: 6.6.61 • 2x USW Mini Flex 2.0.0 • UniFi AC Pro 6.6.62 • UniFi U6-LR 6.6.62
UniFi Protect: 2.11.21/8TB Skyhawk AI • 3x G3 Instants: 4.69.55 • UniFi G3 Flex: 4.69.55 • UniFi G5 Flex: 4.69.55
hugo_b
New here
Posts: 5
Joined: Sun May 09, 2021 4:37 am

Re: NAS not booting - RAMdisk and/or system partition full? Desperately trying to find help, I am fine paying for servic

Post by hugo_b »

Dear Simon,

Thanks for your amazing quick reply!

- The memory was supplied with the NAS and is installed already the lifetime of the NAS. However, if you have an indication that this might be an broken memory module, I can test and replace them.

- The update directory is empty (I assume, since I never worked in the console of a linux based OS before).

Hugo.
You do not have the required permissions to view the files attached to this post.
Skwor
Know my way around
Posts: 247
Joined: Thu Feb 27, 2020 1:38 am

Re: NAS not booting - RAMdisk and/or system partition full? Desperately trying to find help, I am fine paying for servic

Post by Skwor »

hugo_b wrote: Sun May 09, 2021 7:42 am Dear Simon,

Thanks for your amazing quick reply!

- The memory was supplied with the NAS and is installed already the lifetime of the NAS. However, if you have an indication that this might be an broken memory module, I can test and replace them.

- The update directory is empty (I assume, since I never worked in the console of a linux based OS before).

Hugo.
Where did you buy your NAS? I had a QNAP authorized vendor ship a QNAP NAS with the wrong ram and then on a second RMA send ram that was not compatible. Do not assume a third party vendor will get it right even if they are an authorized vendor.
NAS:
TS-453Be
2-4 Gig QNAP ram sticks
1x12 TB Seagate Iron Wolf and 3x12 TB Seagate Exos
Mainly used as a Plex Server and Photo manager (QuMagie is actually pretty good)

WD 12 TB Elements for each hard drive - External HD BU to the NAS movie database and Photos
User avatar
Toxic17
Ask me anything
Posts: 6469
Joined: Tue Jan 25, 2011 11:41 pm
Location: Planet Earth
Contact:

Re: NAS not booting - RAMdisk and/or system partition full? Desperately trying to find help, I am fine paying for servic

Post by Toxic17 »

have you tried reinstalling the firmware again manually via SSH? https://wiki.qnap.com/wiki/Manually_Updating_Firmware

I did notice however your /mnt/HDA_ROOT is 80% usage, slightly high but not sure if thats effecting it. have you dont anythin just before it started playing up?

Also can you post the results of this command too:

cat /proc/mdstat
Regards Simon

Qnap Downloads
MyQNap.Org Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following


NAS: TS-673A QuTS hero h5.1.2.2534 • TS-121 4.3.3.2420 • APC Back-UPS ES 700G
Network: VM Hub3: 500/50 • UniFi UDM Pro: 3.2.9 • UniFi Network Controller: 8.0.28
USW-Aggregation: 6.6.61 • US-16-150W: 6.6.61 • 2x USW Mini Flex 2.0.0 • UniFi AC Pro 6.6.62 • UniFi U6-LR 6.6.62
UniFi Protect: 2.11.21/8TB Skyhawk AI • 3x G3 Instants: 4.69.55 • UniFi G3 Flex: 4.69.55 • UniFi G5 Flex: 4.69.55
hugo_b
New here
Posts: 5
Joined: Sun May 09, 2021 4:37 am

Re: NAS not booting - RAMdisk and/or system partition full? Desperately trying to find help, I am fine paying for servic

Post by hugo_b »

Toxic17 wrote: Sun May 09, 2021 8:14 am have you tried reinstalling the firmware again manually via SSH? https://wiki.qnap.com/wiki/Manually_Updating_Firmware

I did notice however your /mnt/HDA_ROOT is 80% usage, slightly high but not sure if thats effecting it. have you dont anythin just before it started playing up?

Also can you post the results of this command too:

cat /proc/mdstat
Dear Simon,

Thank you! Attached the result of cat /proc/mdstat.

I reinstalled the firmware yesterday via Qfinder, this was successfull. However it did not change anything. Will it make a difference if I do this without or without the hard disks installed? I will try again now and do it first with hard drives installed and if it doesnt change anything again without hard disks.

Hugo.
You do not have the required permissions to view the files attached to this post.
hugo_b
New here
Posts: 5
Joined: Sun May 09, 2021 4:37 am

Re: NAS not booting - RAMdisk and/or system partition full? Desperately trying to find help, I am fine paying for servic

Post by hugo_b »

Skwor wrote: Sun May 09, 2021 8:07 am
hugo_b wrote: Sun May 09, 2021 7:42 am Dear Simon,

Thanks for your amazing quick reply!

- The memory was supplied with the NAS and is installed already the lifetime of the NAS. However, if you have an indication that this might be an broken memory module, I can test and replace them.

- The update directory is empty (I assume, since I never worked in the console of a linux based OS before).

Hugo.
Where did you buy your NAS? I had a QNAP authorized vendor ship a QNAP NAS with the wrong ram and then on a second RMA send ram that was not compatible. Do not assume a third party vendor will get it right even if they are an authorized vendor.
Thank you Skwor,

To be extremely sure it’s not a faulty or incompatible memory module I replaced my 4 memory modules with the 2 that Qnap factory installed (instead of the ones that were installed by the supplier). It would have been great if something this simple was the solution, unfortunately there is no change whatsoever. Thanks anyway for your tip!

Hugo.
User avatar
Toxic17
Ask me anything
Posts: 6469
Joined: Tue Jan 25, 2011 11:41 pm
Location: Planet Earth
Contact:

Re: NAS not booting - RAMdisk and/or system partition full? Desperately trying to find help, I am fine paying for servic

Post by Toxic17 »

hugo_b wrote: Sun May 09, 2021 3:42 pm
Toxic17 wrote: Sun May 09, 2021 8:14 am have you tried reinstalling the firmware again manually via SSH? https://wiki.qnap.com/wiki/Manually_Updating_Firmware

I did notice however your /mnt/HDA_ROOT is 80% usage, slightly high but not sure if thats effecting it. have you dont anythin just before it started playing up?

Also can you post the results of this command too:

cat /proc/mdstat
Dear Simon,

Thank you! Attached the result of cat /proc/mdstat.

I reinstalled the firmware yesterday via Qfinder, this was successfull. However it did not change anything. Will it make a difference if I do this without or without the hard disks installed? I will try again now and do it first with hard drives installed and if it doesnt change anything again without hard disks.

Hugo.

I cannot see installing it without disks will make a difference, there is something no right with your setup I guess. might be worth stopping all apps, see if the issue gos away, if not start removing apps to see if that helps. failing this I really dont know.
something has taken up spce on your system, but what it is could be anything.

make sure you have good recoverable backups before attempting anything.
Regards Simon

Qnap Downloads
MyQNap.Org Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following


NAS: TS-673A QuTS hero h5.1.2.2534 • TS-121 4.3.3.2420 • APC Back-UPS ES 700G
Network: VM Hub3: 500/50 • UniFi UDM Pro: 3.2.9 • UniFi Network Controller: 8.0.28
USW-Aggregation: 6.6.61 • US-16-150W: 6.6.61 • 2x USW Mini Flex 2.0.0 • UniFi AC Pro 6.6.62 • UniFi U6-LR 6.6.62
UniFi Protect: 2.11.21/8TB Skyhawk AI • 3x G3 Instants: 4.69.55 • UniFi G3 Flex: 4.69.55 • UniFi G5 Flex: 4.69.55
hugo_b
New here
Posts: 5
Joined: Sun May 09, 2021 4:37 am

Re: NAS not booting - RAMdisk and/or system partition full? Desperately trying to find help, I am fine paying for servic

Post by hugo_b »

Thanks all for the tips, Simon in particular. For future reference people running towards a simular problem: I finally managed to login again by pressing the reset for 10+ seconds. I was scared to do so, but the recreation of shares to find my data back was not really needed since they recovered by restoring a backup of my settings.
Post Reply

Return to “System & Disk Volume Management”