Out Of Memory (OOM) with 40GB RAM

Q'center app, Helpdesk app
Post Reply
knebb
Starting out
Posts: 20
Joined: Wed Oct 21, 2015 5:06 pm

Out Of Memory (OOM) with 40GB RAM

Post by knebb »

Hi all,

this morgning I encountered an oom situation in my brandnew TS-473A. I would like to get the root cause of the issue if possible. So if you have any ideas, let me know.
It is a TS-473A with 40G of RAm, QuTS hero 5.0.0.1856 (latest).
I have only Virtualization Station installed, doing malware scan and sharing folders to Windows. As virtualization I have a single VM up and running which has 4GB of configured memory.
Two RAID1 ZFS volumes (2TB and 6TB) and a SSD cache from my two m2-SSDs. One share is configured to do compression and deduplication.

This morning I got an alert from my NAS about having to few memory ressources and Virtualization Station was stopped and some pyhton task was killed:

Code: Select all

Warnung	2021-12-03	07:07:52	System	127.0.0.1	---	---	[System Message] QKVM has been automatically disabled by QTS to free up system memory.
I restarted VirtStation and the VM and took a look on the system. When the issue occured I got this in kernel log:

Code: Select all

2021-12-03 07:08:18 +01:00 <4> [137885.065372] Mem-Info:
2021-12-03 07:08:18 +01:00 <4> [137885.067744] active_anon:233180 inactive_anon:362759 isolated_anon:0
2021-12-03 07:08:18 +01:00 <4> [137885.067744]  active_file:7615 inactive_file:2330 isolated_file:0
2021-12-03 07:08:18 +01:00 <4> [137885.067744]  unevictable:3988 dirty:0 writeback:52
2021-12-03 07:08:18 +01:00 <4> [137885.067744]  slab_reclaimable:21309 slab_unreclaimable:6402434
2021-12-03 07:08:18 +01:00 <4> [137885.067744]  mapped:11407 shmem:6834 pagetables:30234 bounce:0
2021-12-03 07:08:18 +01:00 <4> [137885.067744]  free:1206021 free_pcp:1335 free_cma:0

I am fully aware of Linux memory management but not of QuTS hero. Anyone having a hint how to read these entries? Has there really been a memory lack? Where do I have to take a look to identify the process which consumed memory?

When I do a top currently I see:

Code: Select all

Mem: 35042300K used, 6003108K free, 43240K shrd, 1924K buff, 121732K cached
CPU:  7.7% usr 13.5% sys  0.6% nic 77.9% idle  0.0% io  0.0% irq  0.1% sirq
Load average: 9.52 9.37 9.33 7/2883 11134
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
18860     1 admin    S    8593m 21.4   4  0.8 /QVS/usr/bin/qemu-system-x86_64 -name guest=
I am really wondering as 40GB of RAM ought to be enough.... (decades ago I had a PC which had 65MB of HDD which was plenty... :))


Thanks& Greetings

/KNEBB
User avatar
Cbrad01
Know my way around
Posts: 245
Joined: Fri Jan 15, 2016 9:17 pm

Re: Out Of Memory (OOM) with 40GB RAM

Post by Cbrad01 »

I am running into the same type of problem but on a NAS with 128 GB of RAM. It appears that the ZFS caches are using in my case over 100GB of ram and will not release any back to the OS or any other app.
What is odd is that Qnap says the cache should only be 50% of the RAM not try to use 100%


Sent from my iPhone using Tapatalk
User avatar
dolbyman
Guru
Posts: 35024
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: Out Of Memory (OOM) with 40GB RAM

Post by dolbyman »

Already a thread here about hero memory issues

viewtopic.php?f=73&t=163886
knebb
Starting out
Posts: 20
Joined: Wed Oct 21, 2015 5:06 pm

Re: Out Of Memory (OOM) with 40GB RAM

Post by knebb »

Thanks for ppointing out to the existing thread.
Indeed this is the same issue I had here.

Meanwhile I reinstalled my NAS with QuTS hero from scratch and I noticed I am not getting offered any QuTS 5.x firmware updates.
Looks like QNAP itself decided the QuTS firmware 5 is still beta. :)

However, I will stay away from the 5.x version at least for a while!

/KNEBB
n.hristov
New here
Posts: 3
Joined: Thu Jan 20, 2022 8:52 pm

Re: Out Of Memory (OOM) with 40GB RAM

Post by n.hristov »

I resolved this problem by setting the ZFS ARC maximum to 16GB out of 64GB of memory and minimum to 4GB.
I did that by adding two lines to autorun.sh script on startup:

Code: Select all

sysctl -w vfs.zfs.arc_min=4294967296
sysctl -w vfs.zfs.arc_max_set=17179869184
Post Reply

Return to “NAS Management”