Performance lost with >100.000ms Latency :(

[icecoke]

Hello everyone!

we're using a TS-EC1279U-RP with 4.1.1 firmware over a LCAP (2x10Gbe) Network to be the Storage of a Xenserver Pool (currently 4 machines).

We experience really slow performance for simple directory or file listings in the VMs:

[root@foxxxhe log]# time ls /usr/local/www/apache24/noexec
index.html info.php mydir userdocs
0.000u 0.007s 0:05.02 0.0% 0+0k 13+0io 0pf+0w

As you can see, the IPOS are really low (mostly <100) and the Latency is extremly high (up to 150.000ms !!!):

http://www.screencast.com/t/3sOsipWIDQ5

We are using 10 fast SATA Drives in a RAID6 and additional two SSDs for caching, while ext4 write cache is disabled as recommended.

The Latency goes up at the same time, when the LAN bond (IEEE 802.3ad) traffic goes higher (15-25MB/s). But, that is not a really hugh bandwith (~25MBit/s) for a 2x10Gbit/s bonding - is it?


Does anyone has a hint to find out, what might be wrong?

[pwilson]

Hello everyone!

we're using a TS-EC1279U-RP with 4.1.1 firmware over a LCAP (2x10Gbe) Network to be the Storage of a Xenserver Pool (currently 4 machines).

We experience really slow performance for simple directory or file listings in the VMs:

[root@foxxxhe log]# time ls /usr/local/www/apache24/noexec
index.html info.php mydir userdocs
0.000u 0.007s 0:05.02 0.0% 0+0k 13+0io 0pf+0w

As you can see, the IPOS are really low (mostly <100) and the Latency is extremly high (up to 150.000ms !!!):

http://www.screencast.com/t/3sOsipWIDQ5

We are using 10 fast SATA Drives in a RAID6 and additional two SSDs for caching, while ext4 write cache is disabled as recommended.

The Latency goes up at the same time, when the LAN bond (IEEE 802.3ad) traffic goes higher (15-25MB/s). But, that is not a really hugh bandwith (~25MBit/s) for a 2x10Gbit/s bonding - is it?


Does anyone has a hint to find out, what might be wrong?

Nope. Insufficient information provided. Please review: When you're asking a question, please include the following.

[icecoke]

Ah, ok. Sorry for that!

Comparing the list you gave and my given Information, I missed the drive Manufacturer, Model and Firmware revision: WDC WD4000FYYT-01UL1B0, Firmware 01.01K01.
The two SSDs are Samsung EVO840 250GB, Firmware EXT0BB6Q

Many thanks in advance!
Jimmy

[icecoke]

In the meantime I already played around with enabling/disabling tcp offloading with ethtool, but it even did not changed the (overall low) CPU Load on the qnap. As we have the 10GBe Cards installed, I expect their CPU to be helpful, but even with no offloading to the card, the CPU load does not change at all.

There must be something really silly I miss here, when we get such IOPS/Latency values.

Again, any help is welcome and when I missed some more relevant information, please let me know. I will provide them asap.

Thanks again,
Jimmy