Win10 network performance to Qnap worse than Win7.

[samohare]

I've been trying to figure out this problem, and have had no luck finding any information about it. Perhaps somebody here has an idea.

We have a TS-EC1279U-SAS-RP with an REXP1 extension unit. Firmware 4.3.3.0299 Build 20170901.
4x Samsung 850 1TB SSDs for cache (currently disabled as they now negatively impact performance)
8x WD 4TB in the main unit
8x Seagate 4TB in the expansion unit.

Back when I got this, we would hit up to 800MB/s across 10GbE with the cache enabled, from a Win7 workstation (this was before the expansion unit). Currently, we can get around 500-600 MB/s with cache disabled.
We have 4 new Win10 workstations, i7-7820X, 32GB RAM, each with an Intel X540-T1 10GbE cards, same as the ones in the Win7 machines.

The Win10 machines will maybe get to 400MB/s read on a good day, and 240MB/s write. Right now, I'm getting 270MB/s read, 101MB/s write, with nothing else accessing the Qnap. This varies greatly without any noticeable pattern, sometimes dropping to 25-30 MB/s each way for no apparent reason.

Same switch (Netgear XS712T), same Cat6 cables as we used on the Win7 machines.

In trying to trouble shoot this, I have:

Turned SMB 3.0 on on the Qnap - previously this was set to 2.0
Tried turning on jumbo frames at 9000 on the Qnap, test workstation NIC, and all the ports on the switch.
Added a second 10GbE cable to the Qnap, and set up Link Aggregation on both it and the switch. This is working (although I can only get it working in RR, not 802.3ad).
Disabled Large Send Offload V2 on the NICs. This makes the speeds worse.

None of these have made any difference to the Win10 machines. Does anyone have any idea what would be causing this, or how I can try and troubleshoot it? SMB3.0 should increase network performance, not impact it. One other thing that might or might not be relevant - the domain controller is a Win2008 Server which doesn't support SMB3.0. This isn't used for filesharing etc though, just DNS, DHCP, license servers, etc.

Thanks,
Sam

[Trexx]

Are you on the most current firmware on Netgear switch?
Are you seeing port/protocol errors reported on Netgear or win10 box?
How are you transferring files between Win10 box and qnap? My computer, robocopy, etc.?
Did you try hooking the win10 box up using the win7 cable to rule out port/cable issue?
Did you try moving files between win10 & win7 box to see what transfer speed that achieved?
What type of storage is on the Win10 box? Just because the NW can do 10GbE, doesn't mean your PC can necessarily especially if using spinning disk.


Did you try the following to resolve your 802.3AD issues (https://community.netgear.com/t5/Smart- ... 7448#M2336)

[dolbyman]

are the Win7 and Win10 machines both using the same SMB protocol? Are Win10 machines using SMB encryption ?

[storageman]

What RAIDs are you using?
How is the expansion unit connected by card or using onboard SAS ports?
Can you build a temporary pool on the SSDs and report the performance? I want to see the performance without the expansion unit.
Run a switch bypass test between NAS and a server to check performance without switch.

[samohare]

Hi Paul,

Thanks for the response. I will check the firmware on the switch. I haven't seen any errors, but I will take a look at where to find those logs.
Speed tests done by using BlackMagic Disk Speed test, and by copying files in explorer.
The Win10 machines have M.2 SSDs that test at 1800-2200 MB/s
The Win7 machines have SSDs that test at ~500 MB/s

The Win10 boxes are hooked up to the same cables / ports as the Win7. Machine was swapped out, same infrastructure behind it. There are still some Win7 workstations remaining which I am using for comparison. Previously, all the win7 machines behaved similarly.

I set up a shared folder on each local drive of a Win7 and Win10 machine to test direct connections between those.

From Win7 Machine connecting to the Win10 share with Disk Speed Test, Win7 gets 400MB/s write, 250MB/s read.
From Win10 Machine connecting to the Win7 share with Disk Speed Test, Win10 gets 82MB/s write, 50MB/s read.

Copying 19 video files totalling 14.5GB via explorer:
From Win7 Machine copying to/from Win10 share, Win7 gets 500MB/s write, and 220MB/s at the start of the read, dropping to average around 100MB/s by the end.
From Win10 Machine copying to/from Win7 share, Win10 gets 380MB/s write, then dropping to 60MB/s approx halfway through, and staying there. 540MB/s read, with some rapid dips lower.
Copying internally on the Win10 machine, it starts at 1200MB/s, and it drops shortly afterwards to 600-700MB/s

I'll check the other pieces after I get some work out.
Thanks,
Sam

[samohare]

Dolbyman:
Win7 is using 2.1, Win10 is using 3.1.1.
I haven't enabled encryption anywhere, as far as I understand it, this has to be done server side.

Storageman:
I'm using RAID 5 for the main array. The expansion unit is using the onboard 6Gbps connection.
I have indeed built a temporary pool with the SSDs in RAID 0 to test - that was something I tried out to see if I could fix the performance issue. The Win10 machines connect to it at much the same speed as they connect to the main array. Right now I'm getting 130MB/s write, 295MB/s read. The Win7 machines hit around 650MB/s - slightly faster than the main array.
I had tried the cache in both read only and read write mode, and both RAID 10 and RAID 0, before turning it off.

I can't run a bypass test right this moment as work is happening, but I will take a look at doing that.

Thank you both for your responses,
Sam

[samohare]

Hey Skypx,

I did reboot the switch, too. Unfortunately that didn't make a difference. Mine isn't assigned a static IP at the moment, although it has held the same DHCP lease since it was installed, pretty much.

Thanks for the suggestion, I'm starting to ponder if there is something between the Win10 machines and the switch which is unhappy.
Sam

[Trexx]

Hi Paul,

I set up a shared folder on each local drive of a Win7 and Win10 machine to test direct connections between those.

From Win7 Machine connecting to the Win10 share with Disk Speed Test, Win7 gets 400MB/s write, 250MB/s read.
From Win10 Machine connecting to the Win7 share with Disk Speed Test, Win10 gets 82MB/s write, 50MB/s read.
Sam

Since WIn7 only supports up to SMB 2.1, regardless of which way you are copying, you are still limited to SMB 2.1 performance levels.

Yet when WIN10 is the TRANSMITTER/SENDER, performance tanks by 5x. That pretty much rules out QNAP NAS as the issue right there.

Points to either a WIn10 bug/config issue, cabling or NW Switch. Easy way to eliminate NW switch is direct connect the 2 nodes together and repeat the test using 2 different cables which rules out cable issue too.

Now there are some things you could due with getting LAG/Port Trunking working that will improve aggregate performance. But a single host to QNAP will be limited to a single 10GbE connection.

[storageman]

If the 1GbE performance also differs between win7 and win10 clients it will not be a 10GbE driver/config issue.
Are you able to create a small thick LUN and connect it ISCSI to one of the win10 clients. I'm interested to see whether this is an win10 SMB issue.
If ISCSI performance is faster/different it means its SMB related.

[samohare]

Sorry for the slow response here - work got in the way of me testing.

I checked with a win10 machine connected directly into one of the 10GbE ports on the Qnap, using a RAID0 of 4 SSDs in the main chassis, writing up to 38GB or so in a few video files and got these results:

Read:

Write:


So the read speed is great - hitting about the limit you'd expect from 10GbE. Writing is a different story, it starts off quickly and then the speed drops to 0, jumps back up, slows down, repeats a few times and eventually evens out at 120Mbps or so, as you see in the image. Looking at resource monitor on the Qnap when writing, I notice that the IO Wait graph jumps as it starts going - this does not happen when reading from the drives.



Looking at task manager as I'm writing you can clearly see the speed go up, then drop to 0, wait a couple of seconds, and repeat.

So the speed issues I am getting appear to be a mix of the cable (I have tried changing some cables, my guess is that the one running from the patch panel to the port is the problem here), and the Qnap. Note that writing from one Win10 machine to another does NOT show the issue above - I can write at a consistent speed of 325 Mbps, just limited right now by the cable issue.

Any ideas what would be causing that IO wait to kick in?

Also, I tried changing some of the settings on the network card - we had similar to what you had skypx - these did improve performance somewhat on reading at the high end compared to the default settings, but did nothing about the write timeouts, etc.

[Trexx]

IO Wait be due to delays from the SSD itself depending on brand, if you are hitting thermal throttling, fill rate of the SSD's, If you have TRIM regularly running not the SSD's, etc.

You are much better off testing with 1 VERY large file vs. multiple files .... as differing file sizes will also have impacts as SSD performance is different also depending on block size, etc.

If you have another QNAP, I would recommend installing QCenter on that box and then QCenter agent on this one. You can get more detailed performance metrics in the QCenter reports.

At the end of the day though, this is a community support forum and your best bet would be to open a helpdesk ticket with QNAP.

[storageman]

Test it direct and bypass the switch.
You didn't try the ISCSI test

[samohare]

Trexx, thanks for the feedback. I opened a ticket the same day I posted this - it took a week to get a preliminary response, but I'm feeding back there the results I'm getting here.

Storageman - the test posted above was bypassing the switch and connecting directly. I haven't set up ISCSI before - right now I am slammed with work so haven't had a chance, but I will take a look into it when I have a moment.

[Don]

Make sure the Win 10 box has the latest drivers from the manufacturer for the nic card. I've seen many instances where a driver update fixed the issue.