Hi
Firmware: 4.5.2.1630
Qu-Log: 1.2.0.553
16gbRAM
Since applying the last firmware update last week, I am getting very high RAM usage. On investigation qulogd is using most of our RAM and there is 36GB of swap.
Trying to stop the service '/etc/init.d/qulog.sh stop' does nothing, '/etc/init.d/qulog.sh force-stop' kills the service and all of the memory is freed up.
This has happened a couple of times, I have now stopped any logging from the GUI and deleted all previous logs to see if that helps
Checking the logs '/logs/qulog/mariadb.err' I have these entries corresponding to the high memory usage. Maybe related to my forced stop of the service.
210425 20:35:46 [Note] Plugin 'InnoDB' is disabled.
210425 20:35:46 [Note] Plugin 'FEEDBACK' is disabled.
210425 20:35:46 [Note] Event Scheduler: Loaded 0 events
210425 20:35:46 [Note] /usr/local/sbin/qulogdb: ready for connections.
Version: '5.5.57-MariaDB' socket: '/var/run/qulog/mariadb.sock' port: 0 MariaDB Server
210425 20:35:46 [Note] Event Scheduler: scheduler thread started with id 1
210425 20:35:49 [ERROR] qulogdb: Table './qulog/local_access' is marked as crashed and should be repaired
210425 20:35:49 [Warning] Checking table: './qulog/local_access'
'/logs/qulog/qulogd.log' for the restart
2021-04-25 20:34:55,302 [Information] [qulog] service pid(4551) stop.
2021-04-25 20:34:55,308 [Information] [SyslogClientManager] stop threading
2021-04-25 20:35:49,621 [Information] [Database] initialize
2021-04-25 20:35:49,714 [Information] [Database] initialize database
2021-04-25 20:35:49,728 [Information] [Database] initialize function
2021-04-25 20:35:49,728 [Information] [Database] initialize procedure
2021-04-25 20:35:49,755 [Information] [Database] initialize device table
2021-04-25 20:35:49,877 [Information] [Database] initialize filter tables
2021-04-25 20:35:49,885 [Information] [Database] initialize highlight tables
2021-04-25 20:35:49,897 [Information] [Database] initialize receiver table
2021-04-25 20:35:49,907 [Information] default volume = /share/CACHEDEV1_DATA
2021-04-25 20:35:49,907 [Information] [Database] initialize local_event log table
2021-04-25 20:36:17,365 [Information] [qulog] service pid(20241) start.
2021-04-25 20:36:17,366 [Information] [SyslogClientManager] load system default adapter mac address [24:5e:be:0f:8b:ab]
2021-04-25 20:36:17,366 [Information] [SyslogClientManager] start threading
2021-04-25 20:46:20,438 [Information] [Database] truncate local_event
2021-04-25 20:46:45,505 [Information] [Database] truncate local_access
2021-04-25 20:46:58,514 [Information] [Database] initialize local_access access table
Only qulogd.log error I can find is this, from the day before
2021-04-24 08:30:35,661 [Fatal] worker.cpp:120, virtual void QuLog::ProcessWorker::dispatch(QuLog::Endpoint*, json_t*, const char*, qs_io*, int) [ProcessWorker] fork first child process fail
2021-04-24 08:31:00,228 [Information] [qulog] service pid(4551) start.
2021-04-24 08:31:00,238 [Information] [SyslogClientManager] load system default adapter mac address [24:5e:be:0f:8b:ab]
2021-04-24 08:31:00,238 [Information] [SyslogClientManager] start threading
There are no corresponding entries in the mariadb.err log.
To me it looks like it has choked on some json data and started a new qulogd which has no connection to mariadb and hence stores logs in memory.
Our NAS only provides SMB and NFS shares and does not run any other applications. so all logging data is generated from itself.
Any ideas\comments on what else I can do to stop this memory issue ?
qulogd using all RAM and swap
- Toxic17
- Ask me anything
- Posts: 6469
- Joined: Tue Jan 25, 2011 11:41 pm
- Location: Planet Earth
- Contact:
Re: qulogd using all RAM and swap
are you getting alerts about memory usage or is it a case of https://www.linuxatemyram.com/
if you think its not normal then raise a ticket with QNAP.
if you think its not normal then raise a ticket with QNAP.
Regards Simon
Qnap Downloads
MyQNap.Org Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following
NAS: TS-673A QuTS hero h5.1.2.2534 • TS-121 4.3.3.2420 • APC Back-UPS ES 700G
Network: VM Hub3: 500/50 • UniFi UDM Pro: 3.2.9 • UniFi Network Controller: 8.0.28
USW-Aggregation: 6.6.61 • US-16-150W: 6.6.61 • 2x USW Mini Flex 2.0.0 • UniFi AC Pro 6.6.62 • UniFi U6-LR 6.6.62
UniFi Protect: 2.11.21/8TB Skyhawk AI • 3x G3 Instants: 4.69.55 • UniFi G3 Flex: 4.69.55 • UniFi G5 Flex: 4.69.55
Qnap Downloads
MyQNap.Org Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following
NAS: TS-673A QuTS hero h5.1.2.2534 • TS-121 4.3.3.2420 • APC Back-UPS ES 700G
Network: VM Hub3: 500/50 • UniFi UDM Pro: 3.2.9 • UniFi Network Controller: 8.0.28
USW-Aggregation: 6.6.61 • US-16-150W: 6.6.61 • 2x USW Mini Flex 2.0.0 • UniFi AC Pro 6.6.62 • UniFi U6-LR 6.6.62
UniFi Protect: 2.11.21/8TB Skyhawk AI • 3x G3 Instants: 4.69.55 • UniFi G3 Flex: 4.69.55 • UniFi G5 Flex: 4.69.55
-
- New here
- Posts: 2
- Joined: Mon Jun 15, 2015 1:28 am
Re: qulogd using all RAM and swap
Yes I am getting alerts about memory in the GUI. The little QBoost guy is red and memory is showing 98% used in the Dashboard.
QBOOST is unable to free memory due to the high SWAP usage.
Under normal operation it uses 14% RAM and 0% SWAP. qulogd uses about 200MB.
Everything seems quite stable now I have turned off as much logging as is possible in the GUI and also cleared any logs.
Next step is maybe turn logging on one bit at a time over the next few weeks.
I'll also raise a ticket, see where that gets me.
Thanks
QBOOST is unable to free memory due to the high SWAP usage.
Under normal operation it uses 14% RAM and 0% SWAP. qulogd uses about 200MB.
Everything seems quite stable now I have turned off as much logging as is possible in the GUI and also cleared any logs.
Next step is maybe turn logging on one bit at a time over the next few weeks.
I'll also raise a ticket, see where that gets me.
Thanks
- MikeO3
- Starting out
- Posts: 13
- Joined: Sat May 20, 2017 8:46 am
Re: qulogd using all RAM and swap
Keep us posted as mine is almost dead in the water with this same issue.
-
- New here
- Posts: 4
- Joined: Sun Aug 16, 2015 9:22 am
Re: qulogd using all RAM and swap
Was there a resolution to this issue? My /var/log/qulog/mariadb.err is showing those same messages. I was running an app on my PC that made heavy SMB access to the QNAP, and access logging of SMB was turned on in QuLog Center, and that was causing my QNAP to run out of RAMdisk on the / filesystem. For now, I've turned off access logging for SMB, but the error messages are still being written to mariadb.err a few times a minute.
-
- Been there, done that
- Posts: 530
- Joined: Wed Apr 28, 2010 9:22 pm
Re: qulogd using all RAM and swap
Fixed in new QulogCenter released today according to release notes.
1x TS251, 1x TS251D, 1x TS253
-
- New here
- Posts: 4
- Joined: Sun Aug 16, 2015 9:22 am
Re: qulogd using all RAM and swap
I just installed that. It now links /var/log/qulog/mariadb.err to /dev/null, so the file doesn't grow. But the underlying problem with the bad database for qulog hasn't been solved: /var/log/qulog/qulogd.log contains these entries:
2021-08-12 10:22:42,027 [Information] [Database] initialize local_access access table
2021-08-12 10:22:42,031 [Fatal] worker.cpp:53, void QuLog::AbstractWorker::invoke(QuLog::Function, json_t*, const char*, qs_io*, int) [AbstractWorker
] Table './qulog/local_access' is marked as crashed and last (automatic?) repair failed: ALTER TABLE local_access ADD mac CHAR(17) NOT NULL DEFAULT '
' AFTER ip, ADD INDEX (mac, level, id DESC), ADD INDEX (action, id DESC);
2021-08-12 10:22:42,027 [Information] [Database] initialize local_access access table
2021-08-12 10:22:42,031 [Fatal] worker.cpp:53, void QuLog::AbstractWorker::invoke(QuLog::Function, json_t*, const char*, qs_io*, int) [AbstractWorker
] Table './qulog/local_access' is marked as crashed and last (automatic?) repair failed: ALTER TABLE local_access ADD mac CHAR(17) NOT NULL DEFAULT '
' AFTER ip, ADD INDEX (mac, level, id DESC), ADD INDEX (action, id DESC);
-
- New here
- Posts: 4
- Joined: Sun Aug 16, 2015 9:22 am
Re: qulogd using all RAM and swap
I filed a support ticket, and quickly got this response:
"Sorry for any inconvenience. We're aware of the issue and currently working on the solution. This issue will be fixed in the next firmware version. Please update the firmware once it's available."
"Sorry for any inconvenience. We're aware of the issue and currently working on the solution. This issue will be fixed in the next firmware version. Please update the firmware once it's available."
-
- Starting out
- Posts: 28
- Joined: Sat Oct 22, 2016 11:07 am
Re: qulogd using all RAM and swap
Good stuff - I'd been ripping my hair out for weeks (after one particular firmware update) when I logged into the NAS and every once a week or so the UI was nerfed because of / filling up.
Great to see Qnap going for a band aid fix instead of fixing the underlying problem....
I remember reading the release notes saying QulogCenter had fixed it, but now for the life of me find where those notes were..
Great to see Qnap going for a band aid fix instead of fixing the underlying problem....
I remember reading the release notes saying QulogCenter had fixed it, but now for the life of me find where those notes were..