Storage & Snapshots - (possibly) causes system to go unresponsive during snapshot

[M4ngo]

I have a (fairly new) TS-1886-XU-RP-64G put into production 3 months ago. I like it. Snapshots are on and using Smart Versioning. Working normally for several weeks.

But... yesterday - the machine was completely unreachable - except to ping. I had to drive there, confirmed there was no way in I could get into it (and it did not appear on QFinder). Lights all green on the face - so held power down (for ~3 sec) and it shut off instantly. I booted it again and it came right back to available and working normally.

The logs only show it was right about to finish creating a snapshot when I believe it stopped responding. There are no other clues. The next entry in QuLog center is about the power was not shut down properly last time. I am fairly certain it has something to do with Storage & Snapshots > make snapshot, because - whenever a snapshot is created on this system, there appear four entries in QuLog center (see screenshot). These entries all occur within 5 seconds of the first event and always in the same order. They are expected.

Untitled2.png

The last logged event before the incident is very suspiciously event #3 in this series. Which leads me to believe something in the write operations of the snapshot functionality caused the system to hang. (total speculation of course)

Since the reboot it has been happily making snapshots as per the schedule every hour+ with no problem. As it did before the incident - repeating the same four events in the QuLog.

I scanned for malware, viruses, and scanned the disks for health - all green. There is tons of space (see below). There is tons of guaranteed snapshot space too, and as expected - it is just using a fraction.

It has a 21.8 TB (RAID10) storage pool with 1 Data Volume at 3.97 TB (used 747 GB only) - see screenshot:

Untitled1.png

Cache acceleration is on 715 GB over provisioned tp 20% as per profiler. All green.

Right now, this is a single, unexplained incident - but I fully expect it to happen again. I would like to understand what I can do to analyze what happened/diagnose this and be ready if it happens again.

TIA

[dolbyman]

tons of posts about the same thing on qnap forums (here and German one) and reddit

seems to be a bug log it with qnap please (they do not come here)

[bokr71]

This sounds a lot like what I have been experiencing for months (viewtopic.php?f=25&t=164199)... I wonder if my issues are snapshot related...

According to my log, the last thing that happened this morning at 00:05:00, that possibly locked up the NAS, until I pulled power at 08:34:07, and it came back on telling me I had File System Not Clean:

22066, 0,2022-01-09,00:05:00,System,127.0.0.1,localhost,[Storage & Snapshots] Started creating scheduled snapshot. Volume: BMK-vo1.,222,1641708300,A002,Storage & Snapshots,C012,Snapshot,
22067, 1,2022-01-09,08:34:07,System,127.0.0.1,localhost,[Storage & Snapshots] File system not clean. Volume: BMK-vo2, Storage pool: 2. Run a file system check.,835,1641738847,A002,Storage & Snapshots,C001,Volume,

[dosborne]

https://www.qnap.com/en-uk/support/con_ ... cation_bar

Solution for NAS Disconnection Caused by Malware Remover Update
A recent Malware Remover service update might cause system disconnection on certain QNAP NAS devices. This issue mainly affects devices where users have enabled auto update in Malware Remover > Settings. We have updated our service policies to fix this issue.

If your NAS stops responding or cannot be accessed after this service update, we recommend following these steps to manually restart the NAS.

Press and hold the power button until the NAS powers off.
Press the power button again to power on the NAS.
Most affected devices should return to normal after a manual restart. If this issue still persists, please contact QNAP Technical Support for further assistance.

[bokr71]

That doesn’t sound at all like my issue at least. I have had this happen for well over 2 years, throughout several FWs, and several versions of Malware Remover. Also, my latest lockup happened at 00:08:54, and Malware Remover updates at 04:00.

[swilson]

Three of my four qnaps had the same thing on Jan 9th.

All froze on snapshot of System Volume.
I suspect a bug.

[bokr71]

Three of my four qnaps had the same thing on Jan 9th.

All froze on snapshot of System Volume.
I suspect a bug.

It's possible, but it is a "funny" one then. I have had these freezes happen for a long long time, and though they almost all have happened at night. The last entries in my Sys log have varying times, and they do not correlate to my designated Snapshot time. They also do not correlate to the time when Malware Remover is updating.

I have forwarded all info to QNAP, and will await them returning with suggestions. They have looked at it before and have had me remove applications I really didn't need. Change a few settings, and eventually replaced my MoBo. All to no avail...

[manfredell]

Same here, freeze on Jan 9th after a scheduled snapshot. Never happened before.

[Moogle Stiltzkin]

i use snapshots on a ys-877 using qts 5, i didn't notice issues in regards to snapshots. my settings is take snapshots daily and keep for a week.

are u using quts hero perhaps?

[bokr71]

I finally heard back from QNAP, and according to them, my crashes was not because if snapshots... They say:

It looks like the 3 times the system locked up in the past, it was due to the memory being insufficient.
The RD team suggests you either use normal volumes instead of encrypted volumes (meaning you'd have to move the data off the current volumes and recreate normal ones instead), or just add RAM to increase the amount of memory available.

Can that really be true, that because I have encrypted volumes, my 16GB RAM TVS-872XT is running out of RAM? Really?

[Moogle Stiltzkin]

no idea. but i use 32gb of crucial ram :/ on a ts-877

but last time i used way less ram but didn't have an issue. but i wasn't using the current qts version back then so :/ ...

[bokr71]

I think it sounds like **... 16GB RAM is not enough to run two encrypted volumes... hmmm... Sounds like they don't know and are just looking to close the ticket.

If I was to upgrade the RAM, should I go to 32GB or 64GB?

[Moogle Stiltzkin]

i encrypted 3 shares. but i don't encrypt entire volumes :/