Low performance and high CPU after FW upgrade to 4.3.3

Introduce yourself to us and other members here, or share your own product reviews, suggestions, and tips and tricks of using QNAP products.
james_66
New here
Posts: 3
Joined: Mon Aug 21, 2017 8:36 pm

Low performance and high CPU after FW upgrade to 4.3.3

Postby james_66 » Thu Aug 24, 2017 5:32 pm

Greetings,

we have an TS-1679U-RP with 16 WD4000F9YZ-09N20L0 HDs used as a respository for our veeam backup.

Last week I upgraded from 4.1.0 to 4.3.3, since then we have performace (throughput not higher than 7 MB/s) problems and a high CPU IO WAIT (~33%) and CPU usage (~35%)

after running "top" I found that kswapd0 is resonsable for the high cpu rate. After rebooting the NAS the CPU goes down to normal but as soon as the backup starts the cpu goes up and stays that way.

Any Ideas?

Thx,
James

dolbyman
Ask me anything
Posts: 6191
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: Low performance and high CPU after FW upgrade to 4.3.3

Postby dolbyman » Thu Aug 24, 2017 9:31 pm

what raid level and storage configuration are you using?(thick or thin ..etc)

User avatar
Toxic17
Experience counts
Posts: 3778
Joined: Tue Jan 25, 2011 11:41 pm
Location: Planet Earth
Contact:

Re: Low performance and high CPU after FW upgrade to 4.3.3

Postby Toxic17 » Thu Aug 24, 2017 11:13 pm

perhaps Raid Scrubbing is running?
Regards Simon

QTS 4.3/4.2 MANUAL

Submit a ticket - QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following


Intel NUC DN2820FYKH - Debian 9.1 Stretch - CumulusMX 3.0.0 Build 3043
NAS: TVS-463/QM2-2P 4.3.3.0299 - TS-121 4.3.3.0299 - APC Back-UPS ES 700G
QPKG's: TwonkyMedia 8.4.1 - QApache 2.4.28.7110 - QSonarr 2.0.0.4949 - QNBZGet 19.1 - phpMyAdmin v4.7.4 - Piwigo 2.9.1 - Qmono 4.8.1.0
Network: VM Hub 3.0 (200/10) - Sophos UTM 9.503-4 - Ubiquiti CloudKey - UniFi AP AC Pro - TL-SG1016DE - Cisco SLM2008 - RPi 3(PiAware) - RPi 3(Dump1090-mutability)
Retiring - TS-459 Pro-2G 4.2.6 Build 20170729

dolbyman
Ask me anything
Posts: 6191
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: Low performance and high CPU after FW upgrade to 4.3.3

Postby dolbyman » Thu Aug 24, 2017 11:30 pm

Toxic17 wrote:perhaps Raid Scrubbing is running?


I guess you could be right .. is the scrubbing invoking the kswap ?

User avatar
Trexx
Experience counts
Posts: 2646
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota
Contact:

Re: Low performance and high CPU after FW upgrade to 4.3.3

Postby Trexx » Thu Aug 24, 2017 11:38 pm

from googling:

kswapd is managing swap space in response to memory demands greater than physically available for all processes.

It is process agnostic, it is only interested in what pages are access and when (it is more complex than this of course but to keep things simple we may as well view it this way).

So the real question is "what processes have the greatest burden on memory that are causing kswapd to need to page all the time".

That is most easily answered using 'top' and switching to memory usage sort mode.


How much memory do you have? Please paste the output of the following:

Code: Select all

free
Paul

Model: TVS-673 32GB FW: 4.3.3.0299 Build: 20170901
Disks: [RAID-5] 6 x 3TB HGST DeskStar NAS SSD Cache: [RAID-1] 2 x 525GB Crucial MX300 m.2's
UPS: CyberPower AVR1350 Ext. Backup: USB 3.0 Seagate 5TB
Media Boxes: Nvidia ShieldTV Pro (4K), AppleTV 4
-----------------------------------------------------------------------------------------------------------------------------------------
Model: TS-453 Pro 16GB FW: 4.3.x (varies)

Father Mande's Kodi 17.4
Information needed when you ask for HELP
| QNAP Links, Tutorials, etc. |Moogle's QNAP Faq | Plex NAS Compatibility Guide | QNAP Plex FAQ

james_66
New here
Posts: 3
Joined: Mon Aug 21, 2017 8:36 pm

Re: Low performance and high CPU after FW upgrade to 4.3.3

Postby james_66 » Fri Aug 25, 2017 2:38 pm

what raid level and storage configuration are you using?(thick or thin ..etc)
We are using RAID 5, 30 TB thin

perhaps Raid Scrubbing is running?
no

[~] # free
total used free shared buffers cached
Mem: 1859532 1579680 279852 251036 575024 261376
-/+ buffers/cache: 743280 1116252
Swap: 530108 1172 528936

User avatar
Trexx
Experience counts
Posts: 2646
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota
Contact:

Re: Low performance and high CPU after FW upgrade to 4.3.3

Postby Trexx » Fri Aug 25, 2017 10:21 pm

You aren't short of memory, as you have roughly 1GB free of your 2GB of memory. And your minimal swap usage also confirms that as well.

Thin LUNs are by their very nature causing some overhead/performance loss do to allocating blocks on demand. And that also would correspond to CPU IO Wait (waiting for disk I/O) as new blocks are being allocated/mapped.

Have you run SMART tests on your drives lately or a check filesystem?
Paul

Model: TVS-673 32GB FW: 4.3.3.0299 Build: 20170901
Disks: [RAID-5] 6 x 3TB HGST DeskStar NAS SSD Cache: [RAID-1] 2 x 525GB Crucial MX300 m.2's
UPS: CyberPower AVR1350 Ext. Backup: USB 3.0 Seagate 5TB
Media Boxes: Nvidia ShieldTV Pro (4K), AppleTV 4
-----------------------------------------------------------------------------------------------------------------------------------------
Model: TS-453 Pro 16GB FW: 4.3.x (varies)

Father Mande's Kodi 17.4
Information needed when you ask for HELP
| QNAP Links, Tutorials, etc. |Moogle's QNAP Faq | Plex NAS Compatibility Guide | QNAP Plex FAQ

fichtenelch
New here
Posts: 3
Joined: Fri Sep 01, 2017 10:05 pm

Re: Low performance and high CPU after FW upgrade to 4.3.3

Postby fichtenelch » Fri Sep 01, 2017 10:13 pm

Same problem here on a QNAP TS-670 Pro since FW 4.3.3. CPU load is around 25 % (kswapd0).

Code: Select all

             total       used       free     shared    buffers     cached
Mem:       1865400    1694744     170656     250156      77872    1137180
-/+ buffers/cache:     479692    1385708
Swap:     17307320       1032   17306288

Raid scrubbing is enabled on this device.

User avatar
Trexx
Experience counts
Posts: 2646
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota
Contact:

Re: Low performance and high CPU after FW upgrade to 4.3.3

Postby Trexx » Fri Sep 01, 2017 10:44 pm

fichtenelch wrote:Same problem here on a QNAP TS-670 Pro since FW 4.3.3. CPU load is around 25 % (kswapd0).

Code: Select all

             total       used       free     shared    buffers     cached
Mem:       1865400    1694744     170656     250156      77872    1137180
-/+ buffers/cache:     479692    1385708
Swap:     17307320       1032   17306288

Raid scrubbing is enabled on this device.


Have you opened a helpdesk ticket with QNAP? You don't say which build of 4.3.3 you are using.
Paul

Model: TVS-673 32GB FW: 4.3.3.0299 Build: 20170901
Disks: [RAID-5] 6 x 3TB HGST DeskStar NAS SSD Cache: [RAID-1] 2 x 525GB Crucial MX300 m.2's
UPS: CyberPower AVR1350 Ext. Backup: USB 3.0 Seagate 5TB
Media Boxes: Nvidia ShieldTV Pro (4K), AppleTV 4
-----------------------------------------------------------------------------------------------------------------------------------------
Model: TS-453 Pro 16GB FW: 4.3.x (varies)

Father Mande's Kodi 17.4
Information needed when you ask for HELP
| QNAP Links, Tutorials, etc. |Moogle's QNAP Faq | Plex NAS Compatibility Guide | QNAP Plex FAQ

fichtenelch
New here
Posts: 3
Joined: Fri Sep 01, 2017 10:05 pm

Re: Low performance and high CPU after FW upgrade to 4.3.3

Postby fichtenelch » Fri Sep 01, 2017 10:59 pm

It's build 4.3.3.0262. No ticket opened yet. I will investigate if raid scrubbing is triggering this issue.

sogg
New here
Posts: 5
Joined: Mon Oct 31, 2011 9:14 am

Re: Low performance and high CPU after FW upgrade to 4.3.3

Postby sogg » Thu Sep 07, 2017 12:23 pm

Updated my TS-879u-rp yesterday to 4.3.3.0262 and also experiencing the same issues with kswapd0 using 25% CPU constantly and the NAS rebooting randomly.
Anyone found a solution to this yet?
I Just opened a helpdesk ticket with qnap but seeinging if anyone has a quick fix for it

james_66
New here
Posts: 3
Joined: Mon Aug 21, 2017 8:36 pm

Re: Low performance and high CPU after FW upgrade to 4.3.3

Postby james_66 » Fri Sep 08, 2017 3:24 pm

I opened a Helpdesk ticket on 8/12 and no real help except the suggestion to upgrade to 4.3.3.0299 which didn't help at all. I downgraded to 4.2.5. and everything is back to normal. Will wait an see if QNAP offers a working solution. Based on the experience with the helpdesk so far I`m not very optimistic...

sogg
New here
Posts: 5
Joined: Mon Oct 31, 2011 9:14 am

Re: Low performance and high CPU after FW upgrade to 4.3.3

Postby sogg » Mon Sep 11, 2017 6:33 am

james_66 wrote:I opened a Helpdesk ticket on 8/12 and no real help except the suggestion to upgrade to 4.3.3.0299 which didn't help at all. I downgraded to 4.2.5. and everything is back to normal. Will wait an see if QNAP offers a working solution. Based on the experience with the helpdesk so far I`m not very optimistic...


Did you downgrade by just downloading older firmware and uploading it via the web gui?
Any loss of data or configuration when downgrading?

fichtenelch
New here
Posts: 3
Joined: Fri Sep 01, 2017 10:05 pm

Re: Low performance and high CPU after FW upgrade to 4.3.3

Postby fichtenelch » Tue Sep 12, 2017 1:01 am

RAID scrubbing is definitely not the cause of the problem. Got this log entry (no task was running at this time):

Code: Select all

Type   Date   Time   Users   Source IP   Computer name   Content   
Information   10.09.2017   20:51:16   System   127.0.0.1   localhost   [Swap Memory] Swap Memory is currently been used to free out space for physical memory. When it is required to unplug disk(s) where Swap Partition(s) are located, please unplug them one by one.   

On my other QNAP device (TVS-882 - same firmware, 4.3.3.0299) no problems so far. This machine has 8 GB of RAM.

User avatar
Trexx
Experience counts
Posts: 2646
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota
Contact:

Re: Low performance and high CPU after FW upgrade to 4.3.3

Postby Trexx » Tue Sep 12, 2017 2:42 am

james_66 wrote:I opened a Helpdesk ticket on 8/12 and no real help except the suggestion to upgrade to 4.3.3.0299 which didn't help at all. I downgraded to 4.2.5. and everything is back to normal. Will wait an see if QNAP offers a working solution. Based on the experience with the helpdesk so far I`m not very optimistic...


Make sure and keep QNAP updated on any changes you have made and bugging them for a fix.
Paul

Model: TVS-673 32GB FW: 4.3.3.0299 Build: 20170901
Disks: [RAID-5] 6 x 3TB HGST DeskStar NAS SSD Cache: [RAID-1] 2 x 525GB Crucial MX300 m.2's
UPS: CyberPower AVR1350 Ext. Backup: USB 3.0 Seagate 5TB
Media Boxes: Nvidia ShieldTV Pro (4K), AppleTV 4
-----------------------------------------------------------------------------------------------------------------------------------------
Model: TS-453 Pro 16GB FW: 4.3.x (varies)

Father Mande's Kodi 17.4
Information needed when you ask for HELP
| QNAP Links, Tutorials, etc. |Moogle's QNAP Faq | Plex NAS Compatibility Guide | QNAP Plex FAQ


Return to “Users' Corner”

Who is online

Users browsing this forum: No registered users and 4 guests