Low performance and high CPU after FW upgrade to 4.3.3

[james_66]

Greetings,

we have an TS-1679U-RP with 16 WD4000F9YZ-09N20L0 HDs used as a respository for our veeam backup.

Last week I upgraded from 4.1.0 to 4.3.3, since then we have performace (throughput not higher than 7 MB/s) problems and a high CPU IO WAIT (~33%) and CPU usage (~35%)

after running "top" I found that kswapd0 is resonsable for the high cpu rate. After rebooting the NAS the CPU goes down to normal but as soon as the backup starts the cpu goes up and stays that way.

Any Ideas?

Thx,
James

[dolbyman]

what raid level and storage configuration are you using?(thick or thin ..etc)

[Toxic17]

perhaps Raid Scrubbing is running?

[dolbyman]

perhaps Raid Scrubbing is running?

I guess you could be right .. is the scrubbing invoking the kswap ?

[Trexx]

from googling:

kswapd is managing swap space in response to memory demands greater than physically available for all processes.

It is process agnostic, it is only interested in what pages are access and when (it is more complex than this of course but to keep things simple we may as well view it this way).

So the real question is "what processes have the greatest burden on memory that are causing kswapd to need to page all the time".

That is most easily answered using 'top' and switching to memory usage sort mode.

How much memory do you have? Please paste the output of the following:

Code: Select all

free

[james_66]

what raid level and storage configuration are you using?(thick or thin ..etc)
We are using RAID 5, 30 TB thin

perhaps Raid Scrubbing is running?
no

[~] # free
total used free shared buffers cached
Mem: 1859532 1579680 279852 251036 575024 261376
-/+ buffers/cache: 743280 1116252
Swap: 530108 1172 528936

[Trexx]

You aren't short of memory, as you have roughly 1GB free of your 2GB of memory. And your minimal swap usage also confirms that as well.

Thin LUNs are by their very nature causing some overhead/performance loss do to allocating blocks on demand. And that also would correspond to CPU IO Wait (waiting for disk I/O) as new blocks are being allocated/mapped.

Have you run SMART tests on your drives lately or a check filesystem?

[fichtenelch]

Same problem here on a QNAP TS-670 Pro since FW 4.3.3. CPU load is around 25 % (kswapd0).

Code: Select all

             total       used       free     shared    buffers     cached
Mem:       1865400    1694744     170656     250156      77872    1137180
-/+ buffers/cache:     479692    1385708
Swap:     17307320       1032   17306288

Raid scrubbing is enabled on this device.

[Trexx]

Same problem here on a QNAP TS-670 Pro since FW 4.3.3. CPU load is around 25 % (kswapd0).

Code: Select all

             total       used       free     shared    buffers     cached
Mem:       1865400    1694744     170656     250156      77872    1137180
-/+ buffers/cache:     479692    1385708
Swap:     17307320       1032   17306288

Raid scrubbing is enabled on this device.

Have you opened a helpdesk ticket with QNAP? You don't say which build of 4.3.3 you are using.

[fichtenelch]

It's build 4.3.3.0262. No ticket opened yet. I will investigate if raid scrubbing is triggering this issue.

[sogg]

Updated my TS-879u-rp yesterday to 4.3.3.0262 and also experiencing the same issues with kswapd0 using 25% CPU constantly and the NAS rebooting randomly.
Anyone found a solution to this yet?
I Just opened a helpdesk ticket with qnap but seeinging if anyone has a quick fix for it

[james_66]

I opened a Helpdesk ticket on 8/12 and no real help except the suggestion to upgrade to 4.3.3.0299 which didn't help at all. I downgraded to 4.2.5. and everything is back to normal. Will wait an see if QNAP offers a working solution. Based on the experience with the helpdesk so far I`m not very optimistic...

[sogg]

I opened a Helpdesk ticket on 8/12 and no real help except the suggestion to upgrade to 4.3.3.0299 which didn't help at all. I downgraded to 4.2.5. and everything is back to normal. Will wait an see if QNAP offers a working solution. Based on the experience with the helpdesk so far I`m not very optimistic...

Did you downgrade by just downloading older firmware and uploading it via the web gui?
Any loss of data or configuration when downgrading?

[fichtenelch]

RAID scrubbing is definitely not the cause of the problem. Got this log entry (no task was running at this time):

Code: Select all

Type	Date	Time	Users	Source IP	Computer name	Content	
Information	10.09.2017	20:51:16	System	127.0.0.1	localhost	[Swap Memory] Swap Memory is currently been used to free out space for physical memory. When it is required to unplug disk(s) where Swap Partition(s) are located, please unplug them one by one.	

On my other QNAP device (TVS-882 - same firmware, 4.3.3.0299) no problems so far. This machine has 8 GB of RAM.

[Trexx]

I opened a Helpdesk ticket on 8/12 and no real help except the suggestion to upgrade to 4.3.3.0299 which didn't help at all. I downgraded to 4.2.5. and everything is back to normal. Will wait an see if QNAP offers a working solution. Based on the experience with the helpdesk so far I`m not very optimistic...

Make sure and keep QNAP updated on any changes you have made and bugging them for a fix.