[SOLVED] 10GbE Speeds slower than expected

[onesmallstep1966]

I'm not getting expected 10GbE speeds from my setup and wonder if anyone can offer some suggestions as to where the bottleneck might be.

My hardware consists of the following...

QNAP TVS-672XT 16GB RAM (10Gbe LAN Jumbo 9000)
2 x Samsung M.2 1TB EVO 970 RAID 0 Static Volume
3 x Samsung SATA 1TB EVO 960 SSD RAID 5 Qtier Thick Volume
3 x Seagate 10TB IronWolf RAID 5 Qtier Thick Volume

Netgear GS110EMX (2-Port 10G Multi-Gig)

Apple Late 2018 MacBook Pro + Sonnet SOLO10G Thunderbolt 3 Adapter

All benchmark tests are to the M.2 RAID 0 static volume, and whilst the Thunderbolt 3 connection does perform as expected, the 10GbE connection (particularly read) seems a little slow.

The QNAP and MacBook adapters both have their Jumbo Frames set to 9000, so I'm at a loss where else to look.

MacBook Pro via Thunderbolt 3

Thunderbolt 3.png

MacBook Pro via Sonnet 10GbE

10Gbe Jumbo Frame 9000.png

[Bob Zelin]

put in SIX matching drives -
I don't care if they are six Seagate Ironwolf 10TB drives, or six Samsung 960 1TB SSD's. Turn off SSD Caching.
Diable SMB signing on your MacBook Pro. Bypass your cheap Netgear GS100EXM for now while you are doing these tests. (why would you buy this
switch - there are only 2 10G ports, and the rest are 1G connections - just plug directly from the Solo 10G directly into the computer.).

Create a single STATIC volume. And with all that said, you are never going to get 2 people to work reliably (I assume you are doing video editing because I see you are
using Blackmagic Disk Speed Test) - with only 6 drives, and you are NEVER EVER going to get 2 people working reliably on the same box with only 3 drives.

SO - once you follow my instructions (getting 6 matching drives, and creating a single static RAID volume - for only 6 drives, you better only do RAID 5, not 6) -
and you disable SMB signing on your Mac (do you know how to do this ? it's in the /etc/nsmb.conf file) - and you plug directly into the QNAP from the Solo 10G -
if you are still getting slow speeds, then write back.

Bob Zelin

[Bob Zelin]

by the way - I just expanded your images, so I could see them clearly. Even in a fully optimized system, you are never ever going to get more than 1000 MB/sec Read and Write with 10G.
The very definition of 10GbE is 1250 MB/sec in theory - and no one can achieve this. So if you think that you are going to everything "properly" and get the speeds you are getting with thunderbolt 3 on 10G - well, you are not.
People use 10G because it is more reliable, and you are not stuck with a 2 meter cable max (you drop down to thunderbolt 2 speeds the minute you exceed a 2 meter cable with thunderbolt 3).

And with all of that said - 8K RAW from Red is under 200 MB/sec, so your speeds are just fine. But for MULTIPLE USERS, your current configuration is unacceptable, and you will get stuttering in FCP X.
Your system will not be "Super-Fast".

Bob Zelin

[onesmallstep1966]

Thanks for your very informative reply Bob.

I should have made it clear in the original post that my system is not a professional multi-user setup. I’m a FXP X hobbyist and the QNAP Thunderbolt 3 connection works perfectly for me and has been 100% reliable (my QNAP sits on the same desk as my MacBook Pro so 2m cable limit is fine).
The only reason I played with 10GbE was out of interest (hence the cheap switch), but since posting earlier today I rebooted the QNAP and MacBook Pro and the speeds are now where I expected them to be, going via the cheap switch (see image).

20FA27CB-38BB-40DF-BAF5-9EADFB3600D8.jpeg

[Colourspace]

I'd be interested to know if the faster 10g speeds stay like that, or if it slows down to where it was earlier.

I have the same model QNAP but the eight bay version rather than the six (QNAP TVS-872XT). I've had a similar experience to the OP - fast over thunderbolt and slower than expected over ethernet.

A restart or reset makes the 10g go fast, but for me, it always slows down again eventually (it could last hours or just minutes). Further testing suggests Jumbo Frames may cause instability on these units, including erratic speed test results.

[onesmallstep1966]

I'd be interested to know if the faster 10g speeds stay like that, or if it slows down to where it was earlier.

I am seeing this too. Most of the time it works at full speed (940MBs), but it can drop as low as 400MBs. Restarting the QNAP fixes it.

Jumbo frames might be the culprit. I’ve read that enabling Jumbo frames is an all-or-nothing procedure and I have several devices on my LAN using the LAN 1 10GbE adapter that don’t support Jumbo frames, so for now, I’ve put the MTU on the QNAP and MacBook Pro back to 1500. Speeds have dropped to 750MBs Write / 630MBs Read (can’t understand why Read speeds on my setup are always lower with 10GbE), but I’ll see if this stays stable and report back.

[rock2me]

Slow 10GBe performance from TVS-672XT attached to Netgear XS716T. Could never iperf3 results greater than 200MB/sec when the QNAP NAS was set to Jumbo Frames (9000). Setting the QNAP NAS to use Jumbo Frame = 1500 and performance doubled, to what I would expect. Changing the value on the switch side, had no impact on performance, so the switch port is set to 9000. Tried each of the various QNAP Jumbo Frame settings and each were suboptimal including dropped packets. Setting the QNAP Jumbo Frame =1500 was most stable and fastest.

[mrpafu]

Let me understand your setup and workflow.

You are using the m.2 slots as RAID 0, how about the other 6 slots? How do you use them?

[Bob Zelin]

Hello -
when you observe the slowdown (after you see the fast speeds, and then the speeds drop) - you state that you are plugged into a Netgear XS716T, which I use all the time. The next time this happens, don't restart the QNAP. Simply unplug the computer (via it's 10G adapter) - and plug this directly into the QNAP 10G port (in your case with the TVS-672XT, this is the top ethernet port, or ETH 1).
Now, do a speed test. I bet you get full speeds.

I have recently joined community.netgear.com to complain about this. I see this problem now with XS708T's and XS716T more than ever (with the new Netgear 7.0.0.20 firmware) - you either reset the switch, or do a factory default to the switch (which reboots anyway) - reset MTU 9198 on the Netgear XS series switch, and the speeds come back. Then a couple of days later - the speeds slow down again. Reset the switch - speeds back to normal. It sure looks like a Netgear 10G switch issue.

I CANNOT WAIT for QNAP to release the QSW-M1208-8C (I know, it's not big enough if you need a 12 port or 16 port 10G switch) - but it will be at least an alternative to Netgear at a reasonable price.

Bob Zelin

[RudieCph]

I just had the same problems with my new TVS-872-xt. Write speeds was around 800 MB/s and read was only 250-300 MB/s.
My QNAP had Jumbo Frames set to 1500 from the factory and my network adapter is a 10G Asus xg-c100c and that had Jumbo Packets disabled as a standard.
I had the option to put my 2040 bytes, but that didnt change the speed for me.

WHAT SOLVED IT FOR ME:
QNAP to 9000 Jumbo Frames
ASUS c100 to 9014 Bytes Jumbo Packets.

Now my read speed is 1050 MB/s and write speed is 850MB/s in Black Magic speed test and in real life use.

In IPERF3 it now performs a little worse now than before, but has more than tripled it's performance in real life use. I dont know much about iperf3 so I have no idea why in there it dropped from around 8,7 Gbits/sec to 7,4. But again it works much better when i transfer files to and from the computer.

[mkbean]

I wish this worked for me but it isn't. I have a TS-932PX. It has a two 10GB fiber connections. I have changed the Jumbo Frames to all the options (1500/4074/7418/9000) and nothing helps. Seems I only get around 240MBs transfer. My NIC on my computer has 4088/9014/16348 and it doesn't seem to help. In fact when I put them both at the 9K number the speed drops drastically and times out.

[onesmallstep1966]

My original post is no longer relevant as Apple appear to have improved macOS.
I now use QUTSHero and get approx 800MB write / 1020MB read on my 672XT over it’s built-in 10GbE adapter without using jumbo frames to a 6x10TB RAID 5 (900/1020 to SSD raid 1).

I suggest you install iPerf on your NAS and run an iPerf test from your computer to verify the link speed. If it’s working as expected You should see roughly 9.4Gbits/sec.
What is the disk configuration in your NAS?

[ver2ual]

I apologize but I am rookie user.

I've done my best discover what is the cause of slow transfer rates, but I am stuck. I have a TVS-672XT with 6x 10TB iron wolf drives, I am connected to the 10gbe using a 2022 MacBook Pro M1 with a Belkin Thunderbolt adapter. My highest speeds using Black Magic Speed test are low 90's MB/s. I am currently transferring files between my Macbook and the Qnap and my speed is 89.95 MB/sec. I followed the above instructions regarding SMB but that didn't seem to help. I just received this Qnap back from an editor who set it up, but I am certain something is wrong. This is the first time I have used this Qnap ever.

For reference, I connected to an Alienware Laptop using the 10gbe and got the same speed results using CrystalDisk.

Additionally, back on the MacBook, Qfinder Pro shows both Thunderbolt connections as 'disconnected' and Thunderbolt is red (not connected) on my Mac and I can't seem to get the bridge connected in any way using a number of Thunderbolt cables.

Lastly, Qfinder will not let me update anything- firmware, antivirus, etc.- and some times it will not even find the Qnap but will still let me transfer at the slow rate even when not showing in Qfinder. I just got this back from an editor and the log shows nothing has been updated since Oct/21. The QNAP was new in May 21. Methinks this editor did not know what he was doing when setting this up- or perhaps the error is mine.

Any help is greatly appreciated.

Thanks

[onesmallstep1966]

Your QNAP configuration is identical to mine so you should be seeing similar speeds to me.
The speeds you are getting are inline with what I would expect to see when using 1GbE and you say you are using a Belkin USB C Ethernet adapter. I think your Belkin adapter might just be a 1GbE device and that is your problem.

[ver2ual]

Okay, I ditched the Belkin and connected directly to TB3 and now getting similar to your 800MB write/1020MB read - so solved thanks- and that's with the 6 x 10tb IronWolf in RAID 5. I'm also thinking of using the M.2 bays but my project files are often 5-6 TB so it's limiting. BTW, do you use heat sinks on the NVMe? How fast are the 3xEVO's in the bays?