qulogd using all RAM and swap

Questions about SNMP, Power, System, Logs, disk, & RAID.
Post Reply
desmodue
New here
Posts: 2
Joined: Mon Jun 15, 2015 1:28 am

qulogd using all RAM and swap

Post by desmodue »

Hi

Firmware: 4.5.2.1630
Qu-Log: 1.2.0.553
16gbRAM

Since applying the last firmware update last week, I am getting very high RAM usage. On investigation qulogd is using most of our RAM and there is 36GB of swap.
Trying to stop the service '/etc/init.d/qulog.sh stop' does nothing, '/etc/init.d/qulog.sh force-stop' kills the service and all of the memory is freed up.
This has happened a couple of times, I have now stopped any logging from the GUI and deleted all previous logs to see if that helps

Checking the logs '/logs/qulog/mariadb.err' I have these entries corresponding to the high memory usage. Maybe related to my forced stop of the service.

210425 20:35:46 [Note] Plugin 'InnoDB' is disabled.
210425 20:35:46 [Note] Plugin 'FEEDBACK' is disabled.
210425 20:35:46 [Note] Event Scheduler: Loaded 0 events
210425 20:35:46 [Note] /usr/local/sbin/qulogdb: ready for connections.
Version: '5.5.57-MariaDB' socket: '/var/run/qulog/mariadb.sock' port: 0 MariaDB Server
210425 20:35:46 [Note] Event Scheduler: scheduler thread started with id 1
210425 20:35:49 [ERROR] qulogdb: Table './qulog/local_access' is marked as crashed and should be repaired
210425 20:35:49 [Warning] Checking table: './qulog/local_access'

'/logs/qulog/qulogd.log' for the restart

2021-04-25 20:34:55,302 [Information] [qulog] service pid(4551) stop.
2021-04-25 20:34:55,308 [Information] [SyslogClientManager] stop threading
2021-04-25 20:35:49,621 [Information] [Database] initialize
2021-04-25 20:35:49,714 [Information] [Database] initialize database
2021-04-25 20:35:49,728 [Information] [Database] initialize function
2021-04-25 20:35:49,728 [Information] [Database] initialize procedure
2021-04-25 20:35:49,755 [Information] [Database] initialize device table
2021-04-25 20:35:49,877 [Information] [Database] initialize filter tables
2021-04-25 20:35:49,885 [Information] [Database] initialize highlight tables
2021-04-25 20:35:49,897 [Information] [Database] initialize receiver table
2021-04-25 20:35:49,907 [Information] default volume = /share/CACHEDEV1_DATA
2021-04-25 20:35:49,907 [Information] [Database] initialize local_event log table
2021-04-25 20:36:17,365 [Information] [qulog] service pid(20241) start.
2021-04-25 20:36:17,366 [Information] [SyslogClientManager] load system default adapter mac address [24:5e:be:0f:8b:ab]
2021-04-25 20:36:17,366 [Information] [SyslogClientManager] start threading
2021-04-25 20:46:20,438 [Information] [Database] truncate local_event
2021-04-25 20:46:45,505 [Information] [Database] truncate local_access
2021-04-25 20:46:58,514 [Information] [Database] initialize local_access access table

Only qulogd.log error I can find is this, from the day before

2021-04-24 08:30:35,661 [Fatal] worker.cpp:120, virtual void QuLog::ProcessWorker::dispatch(QuLog::Endpoint*, json_t*, const char*, qs_io*, int) [ProcessWorker] fork first child process fail
2021-04-24 08:31:00,228 [Information] [qulog] service pid(4551) start.
2021-04-24 08:31:00,238 [Information] [SyslogClientManager] load system default adapter mac address [24:5e:be:0f:8b:ab]
2021-04-24 08:31:00,238 [Information] [SyslogClientManager] start threading

There are no corresponding entries in the mariadb.err log.

To me it looks like it has choked on some json data and started a new qulogd which has no connection to mariadb and hence stores logs in memory.

Our NAS only provides SMB and NFS shares and does not run any other applications. so all logging data is generated from itself.

Any ideas\comments on what else I can do to stop this memory issue ?
User avatar
Toxic17
Ask me anything
Posts: 6469
Joined: Tue Jan 25, 2011 11:41 pm
Location: Planet Earth
Contact:

Re: qulogd using all RAM and swap

Post by Toxic17 »

are you getting alerts about memory usage or is it a case of https://www.linuxatemyram.com/

if you think its not normal then raise a ticket with QNAP.
Regards Simon

Qnap Downloads
MyQNap.Org Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following


NAS: TS-673A QuTS hero h5.1.2.2534 • TS-121 4.3.3.2420 • APC Back-UPS ES 700G
Network: VM Hub3: 500/50 • UniFi UDM Pro: 3.2.9 • UniFi Network Controller: 8.0.28
USW-Aggregation: 6.6.61 • US-16-150W: 6.6.61 • 2x USW Mini Flex 2.0.0 • UniFi AC Pro 6.6.62 • UniFi U6-LR 6.6.62
UniFi Protect: 2.11.21/8TB Skyhawk AI • 3x G3 Instants: 4.69.55 • UniFi G3 Flex: 4.69.55 • UniFi G5 Flex: 4.69.55
desmodue
New here
Posts: 2
Joined: Mon Jun 15, 2015 1:28 am

Re: qulogd using all RAM and swap

Post by desmodue »

Yes I am getting alerts about memory in the GUI. The little QBoost guy is red and memory is showing 98% used in the Dashboard.
QBOOST is unable to free memory due to the high SWAP usage.

Under normal operation it uses 14% RAM and 0% SWAP. qulogd uses about 200MB.

Everything seems quite stable now I have turned off as much logging as is possible in the GUI and also cleared any logs.

Next step is maybe turn logging on one bit at a time over the next few weeks.

I'll also raise a ticket, see where that gets me.

Thanks
User avatar
MikeO3
Starting out
Posts: 13
Joined: Sat May 20, 2017 8:46 am

Re: qulogd using all RAM and swap

Post by MikeO3 »

Keep us posted as mine is almost dead in the water with this same issue.
pmcjones
New here
Posts: 4
Joined: Sun Aug 16, 2015 9:22 am

Re: qulogd using all RAM and swap

Post by pmcjones »

Was there a resolution to this issue? My /var/log/qulog/mariadb.err is showing those same messages. I was running an app on my PC that made heavy SMB access to the QNAP, and access logging of SMB was turned on in QuLog Center, and that was causing my QNAP to run out of RAMdisk on the / filesystem. For now, I've turned off access logging for SMB, but the error messages are still being written to mariadb.err a few times a minute.
Ron1963
Been there, done that
Posts: 530
Joined: Wed Apr 28, 2010 9:22 pm

Re: qulogd using all RAM and swap

Post by Ron1963 »

Fixed in new QulogCenter released today according to release notes.
1x TS251, 1x TS251D, 1x TS253
pmcjones
New here
Posts: 4
Joined: Sun Aug 16, 2015 9:22 am

Re: qulogd using all RAM and swap

Post by pmcjones »

I just installed that. It now links /var/log/qulog/mariadb.err to /dev/null, so the file doesn't grow. But the underlying problem with the bad database for qulog hasn't been solved: /var/log/qulog/qulogd.log contains these entries:

2021-08-12 10:22:42,027 [Information] [Database] initialize local_access access table
2021-08-12 10:22:42,031 [Fatal] worker.cpp:53, void QuLog::AbstractWorker::invoke(QuLog::Function, json_t*, const char*, qs_io*, int) [AbstractWorker
] Table './qulog/local_access' is marked as crashed and last (automatic?) repair failed: ALTER TABLE local_access ADD mac CHAR(17) NOT NULL DEFAULT '
' AFTER ip, ADD INDEX (mac, level, id DESC), ADD INDEX (action, id DESC);
pmcjones
New here
Posts: 4
Joined: Sun Aug 16, 2015 9:22 am

Re: qulogd using all RAM and swap

Post by pmcjones »

I filed a support ticket, and quickly got this response:

"Sorry for any inconvenience. We're aware of the issue and currently working on the solution. This issue will be fixed in the next firmware version. Please update the firmware once it's available."
sunnyl
Starting out
Posts: 28
Joined: Sat Oct 22, 2016 11:07 am

Re: qulogd using all RAM and swap

Post by sunnyl »

Good stuff - I'd been ripping my hair out for weeks (after one particular firmware update) when I logged into the NAS and every once a week or so the UI was nerfed because of / filling up.

Great to see Qnap going for a band aid fix instead of fixing the underlying problem....

I remember reading the release notes saying QulogCenter had fixed it, but now for the life of me find where those notes were..
Post Reply

Return to “System & Disk Volume Management”