Asymmetric network performance in VM on TS-563

[supervortex]

Hi. I have a QNAP TS-563 running 4.3.3.0262 Build 20170727. I'm running a Fedora Linux 26 VM on this machine through Virtualization Station 3.0 and using iperf3 to test network bandwidth. With this setup, I'm seeing asymmetric network performance that I don't understand. Namely, all networking performance seems fine except traffic to the VM, which is about 10 times slower than other traffic. Is this a known issue? Any way to fix it? As is, this asymmetry makes running any sort of service on the VM that receives data really slow and VMs on this system not too valuable as a result.

Thanks in advance for any help or insight you might be able to provide!

Summary of my tests:

1/ Laptop Client => TS-563 (non-virtualized): 374 Mb/s
2/ TS-563 (non-virtualized) => Laptop Client: 486 Mb/s
3/ Laptop client => Fedora VM on TS-563: 32.9 Mb/s
4/ Fedora VM on TS-563 => Laptop client: 393 Mb/s

Test details:

1/ Laptop Client => TS-563 (non-virtualized):

Code: Select all

laptop ~ $ iperf3 -c 192.168.1.22 -t 30 -i 30 
Connecting to host 192.168.1.22, port 5201
[  4] local 192.168.1.157 port 36962 connected to 192.168.1.22 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-30.00  sec  1.31 GBytes   375 Mbits/sec    0   3.62 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-30.00  sec  1.31 GBytes   375 Mbits/sec    0             sender
[  4]   0.00-30.00  sec  1.31 GBytes   374 Mbits/sec                  receiver

iperf Done.

2/ TS-563 (non-virtualized) => Laptop Client:

Code: Select all

laptop~ $ iperf3 -c 192.168.1.22 -t 30 -i 30 -R
Connecting to host 192.168.1.22, port 5201
Reverse mode, remote host 192.168.1.22 is sending
[  4] local 192.168.1.157 port 37066 connected to 192.168.1.22 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-30.00  sec  1.70 GBytes   486 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-30.00  sec  1.70 GBytes   487 Mbits/sec  736             sender
[  4]   0.00-30.00  sec  1.70 GBytes   486 Mbits/sec                  receiver

iperf Done.

3/ Laptop client => Fedora VM on TS-563

Code: Select all

laptop ~ $ iperf3 -c 192.168.1.29 -t 30 -i 30
Connecting to host 192.168.1.29, port 5201
[  4] local 192.168.1.157 port 38390 connected to 192.168.1.29 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-30.00  sec   118 MBytes  33.0 Mbits/sec  5837   1.41 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-30.00  sec   118 MBytes  33.0 Mbits/sec  5837             sender
[  4]   0.00-30.00  sec   118 MBytes  32.9 Mbits/sec                  receiver

iperf Done.

4/ Fedora VM on TS-563 => Laptop client

Code: Select all

laptop ~ $ iperf3 -c 192.168.1.29 -t 30 -i 30 -R
Connecting to host 192.168.1.29, port 5201
Reverse mode, remote host 192.168.1.29 is sending
[  4] local 192.168.1.157 port 38612 connected to 192.168.1.29 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-30.00  sec  1.37 GBytes   393 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-30.00  sec  1.37 GBytes   393 Mbits/sec  1232             sender
[  4]   0.00-30.00  sec  1.37 GBytes   393 Mbits/sec                  receiver

iperf Done.

[Trexx]

How do you have your VM settings setup for NW, CPU, VM Disk, etc?

[supervortex]

VM settings:

• 2 cores
• 4.13 GB
• Adapter 1: VirtIO Virtual Switch
• Disk: 250 GB VirtIO disk

[Trexx]

What is your QNAP network configuration? Do you have port trunking enabled?

What is your Cache Mode under VM Settings/Storage?

[supervortex]

My QNAP network config is as follows. The iperf tests I did directly to the QNAP show that there's no issue with network performance directly to the QNAP, only to the underlying VM:

• Single 1GigE on QNAP attacted to virtual switch: 192.168.1.22/24
• Virtual switch with VM interface: 192.168.1.29/24
• no port trunking

adapter1.png

virtual-switch-a.png

virtual-switch.png

virtual-switch-b.png

no-port-trunking.png

Under VM Settings / Storage, my cache settings are set to none. However, the iperf3 network bandwidth test I'm doing doesn't use storage - it's network to memory only - so I'm not sure how this would have an impact.

cache-mode.png

[Trexx]

I would try setting it to writeback just to be safe anyway. It shouldn't make a difference but stranger things have happened.

The other thought is trying a different Linux VM (Debian/Ubuntu) and see if you see similar results. I am assuming you have applied all updates to Fedora and such.

One other thing that is making me nervous is that you are showing alot of packet RETR (retransmissions) which points toward nw errors or congestion.

How are your QNAP-> Router/Switch -> laptop connected from a cabling/nw setup?

[supervortex]

Thanks for all your replies!

- Setting storage cache to writeback had no impact on performance. After making that change, and with the same tests as previously described, it's still 10x slower to the VM vs. from from the VM or to the QNAP.
- I have applied all updates to Fedora 26.
- I noticed the retransmissions in the iperf output too. I found that this is somehow due to the laptop client being wirelessly connected. When wireless is in the mix, network throughput to the QNAP is fine, but throughput to the VM on the QNAP is very bad. In contrast, when I remove wireless from the mix, throughput to both the QNAP and its hosted VM is fine. Something's going on with the VM receiving packets that have transited a wireless network.
- My physical setup was laptop = wifi => port on wireless-ap => qnap, but I've since taken the wifi out for testing, connecting the laptop directly to the wireless-ap.
- I'm in the process of testing CentOS 6 and Ubuntu.
- I'd still like to make this work with wireless as a component.

[supervortex]

A few more updates:

- The problem doesn't exist with a CentOS6 VM!
- The problem on the the Fedora 26 VM isn't exclusively related to wireless. The Fedora 26 VM is also asymmetric to/from the Internet when everything is wired. I.e., traffic to the Fedora 26 VM from the Internet is slow.

Still investigating...

[Trexx]

Sounds like that Fedora version has some weird quirk in it's virtio drivers or something. You can use wireless, but your performance will be hit or miss just too inconsistent and too many outside factors can influence performance. At least for now, lets keeps the laptop hooked via wired until we are done.

Lets run the iperf3 tests again using all hardwired and the centos VM and then post the results. I am hoping we see pretty consistent performance and the RETR's near 0 or at 0.