Poor NFS performance 3..5MB/s - 100% CPU usage

[Nick57]

Hi guys,

I am having a problem and was wondering if any of you have some experience in this. My nfs speed is low, the cpu utilises at a 100% and I am not getting any more speed out of this nfs share then between 3..5MB/s I am trying to move a 300GB backup around and it seems to take at least 2 days now with this speed which is really not acceptable option. I know I am losing some speed because I use the -o sync option but 3..5 seems to be very low also this 100% CPU worries me. Is it possible something is set wrong in the configuration or something wrong with hardware? Is it possible that because of the RAID 5 the CPU is just to busy with calculating? because I believe this is on this model a software raid!

I have this TS-410U [512mb mem] with a Raid 5 [3 DISK + 1 SPARE]
I know Jumbo packets can take load of cpu away but I can't set everything to same size of [9000] jumbo packets, so jumbo packets is no option.

On the Xenserver:
mkdir /mnt/VMbackup
mount xxx.xxx.xxx.xxx:/VMbackup /mnt/VMbackup -o sync

Command to export the backup to the NFS share.
xe vm-export vm=b759625c-eab5-4e0f-be5e-a05bcbad869a filename=/VMbackup/filename.xva

Any help much appreciated.

[geowrian]

That kind of speed is not likely typical. I use the x79 series personally, but basic file transfer speeds should be able to exceed 10MB/sec even on an entry level home NAS on a single drive. A 410U is well above that, so I'm thinking something else is going on here. What kind of speeds do you get with other files on the device? What speeds do you get within the VM? Normally I would point towards the drives having an issue, but if you're seeing 100% CPU load then something else is going on.

[Nick57]

Thanks for the reply, the 100%cpu worries me, no idea what is going on, when looking in the resource monitor under the process i see a few nfs deamons but they don't take up much, memory usage is very low as well. You would think it is the software raid 5 what is so busy with the writings [calculating] ? I will do some testing with writing files to the qnap and come back.

-Question, is it possible to give more priority to the nfs deamons/process to speed things up? The combination low memory usage high CPU load is giving me thoughts about tweaking this a lot better, maybe there is to much optimization going on to free memory usage?

Maybe good to mention that I have put the 2 nics in a bond [trunking] Balance-alb mtu=1500, so there should no issue with this, well I hope!

Testing result:

Writing a file of size: 6,268,150,272 takes 32:35 minutes.
Reading the same file back takes only a: 5:08 minutes.
Hardly any CPU usage when reading!

[forkless]

First of all I would start with checking the disk subsystem and take any network transport out of the equation to see if there are any other bottlenecks there.

If you have Optware installed you can either use Midnight Commander to check the throughput on a local file copy, or install pv and use something like;

#pv -p -a infile > outfile

This will read and write from the array so figures won't be fully accurate obviously but it should be a good enough indication whether the CPU/Disk subsystem is or is not the root issue.

Cheers,
fork

[Nick57]

Hi,

Thanks for the reply, how do I get root access to do this?
Would you mind to give some more in detail instruction to test the disk/subsystem with pv?

The read speed is always fine, the bottleneck is this writing it seems.
The CPU load tops to 100% even if i do a remote replication to copy files internal from 1 folder to another.
The problem is the writing to disk, no matter if it is NFS, FTP or internal copy...

Ok, here some info, I really hope some wizzkids can help me with this problem.
http://wiki.qnap.com/wiki/How_to_identi ... ance_issue

[~] # hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 576 MB in 2.01 seconds = 287.13 MB/sec
Timing buffered disk reads: 216 MB in 3.03 seconds = 71.40 MB/sec
[~] # hdparm -tT /dev/sdb

/dev/sdb:
Timing cached reads: 568 MB in 2.00 seconds = 283.93 MB/sec
Timing buffered disk reads: 234 MB in 3.01 seconds = 77.75 MB/sec
[~] # hdparm -tT /dev/sdc

/dev/sdc:
Timing cached reads: 564 MB in 2.00 seconds = 281.59 MB/sec
Timing buffered disk reads: 232 MB in 3.01 seconds = 76.98 MB/sec
[~] # hdparm -tT /dev/sdd

/dev/sdd:
Timing cached reads: 576 MB in 2.01 seconds = 286.22 MB/sec
Timing buffered disk reads: 234 MB in 3.01 seconds = 77.79 MB/sec
[~] #

[~] # ethtool eth0
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbag
Wake-on: d
Link detected: yes
[~] # ethtool eth1
Settings for eth1:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 8
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbag
Wake-on: d
Link detected: yes
[~] #

ifconfig
bond0 Link encap:Ethernet HWaddr 00:08:99:66:55
inet addr:192.168.181.14 Bcast:192.168.181.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:619495188 errors:851 dropped:0 overruns:0 frame:0
TX packets:315981918 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3190527211 (2.9 GiB) TX bytes:3027777531 (2.8 GiB)

eth0 Link encap:Ethernet HWaddr 00:08:9B:99:66:67
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:508906458 errors:386 dropped:0 overruns:0 frame:0
TX packets:290149842 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:532
RX bytes:1714316301 (1.5 GiB) TX bytes:2994367799 (2.7 GiB)
Interrupt:15

eth1 Link encap:Ethernet HWaddr 00:08:9B:99:66:78
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:110588730 errors:465 dropped:0 overruns:0 frame:0
TX packets:25832076 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:532
RX bytes:1476210910 (1.3 GiB) TX bytes:33409732 (31.8 MiB)
Interrupt:11

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:14371290 errors:0 dropped:0 overruns:0 frame:0
TX packets:14371290 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:342504447 (326.6 MiB) TX bytes:342504447 (326.6 MiB)

So, it seems some errors on my LAN.
Reading of disk seems ok, but the problem is the writing speed.
The charts showing writing speed of 22.6MB are based on a ext4 system: http://wiki.qnap.com/wiki/NFS_Performance_Report
My system is however formatted with the ext2 system.

# cat /etc/fstab
# /etc/fstab: static file system information.
#
# <file system> <mount pt> <type> <options> <dump> <pass>
/dev/ram / ext2 defaults 1 1
proc /proc proc defaults 0 0
none /dev/pts devpts gid=5,mode=620 0 0

Anyone else some suggestions?

[Nick57]

ok, it seems this raid 5 is putting to much strain on the CPU!

Question, I have this raid 5 [3 disk + 1 global spare]
Is it possible to remove disk4 as a global spare and copy all data to that drive?

Then create a new raid 1 with 2 disk and 1 global spare [disk 1+ 2 + 3]?

Then copy all data back to new raid 1?

Is this possible?