Performance of 10GbE vs SFP+ vs Thunderbolt 2 vs Thunderbolt 3

[jds580s]

I am about to retire (actually re-purpose) my old QNAP NAS and want to replace it with something that will be significantly faster.
I use it in an animation studio setting to serve up 3D assets, renders, and playback large video files. The gigabit network I currently have is the bottleneck.

My goal is to have performance that feels more like local or direct attached storage, but also have it redundant and available to the rest of the network.
I really only need one primary workstation to have that kind of speed. The rest are mostly headless render nodes that can access the files slower.

I've never used 10Gb or greater networking equipment, or it's Thunderbolt equivalent, so I'm torn as to which way I should go. I would love to see real world speeds approaching 1000MB/s read and write, and a relatively problem free and reliable storage solution. I am willing to pay for these features, but I don't want to overspend either. My purchase thoughts so far have been:

1. Wait for the TVS-1282T3 and connect via Thunderbolt 3 over USB-C
Pros:

• Theoretically should be the fastest connection @ 40Gb/s
• Latest generation i5 / i7 are options
• Could make use of QTIER setup with 2 tiers (SSDs & spinning disks) with M.2 Caching
• Low cost of upgrading existing network, though the TVS-1282T3 itself will likely be the most expensive NAS.

Cons:

• Unsure how production ready TB3 networking will be.
• I would lose one entire TB bus on the workstation since it would be dedicated to the QNAP, assuming it works like the TB2 version
• High cost of TVS-1282T3

2. Go with the existing TVS-1282T and connect via Thunderbolt 2
I include this option to reduce cost. TB2 should have sufficient bandwidth. I assume it would perform close to the TVS-1282T3, but I don't have any real world first hand experience with it, so if anyone does know what transfer speeds they are able to sustain I would be interested in hearing.
Online I see benchmarks of 950MB/s write, 750MB/s read for the TVS-871T if I was in that neighborhood it could be enough.

3. Go with a non-Thunderbold unit that has 10GbE out of the box, or add on a 10GbE / SFP+ card
Pros:

• Lower cost for the QNAP (I'm thinking TVS-882-i5-16G)
• Potential to connect more systems at high speeds down the road if I want to by just adding a network card in each
• Seems more industry standard to use traditional networking, rather than doing IP over Thunderbolt. Potentially less problems would pop up?

Cons:

• Less theoretical bandwidth than TB2 or TB3
• Requires add-on 10Gb/s networking card for QNAP
• Requires add-on 10Gbps networking card for workstation
• Requires new switch with at least 2 10Gb ports
• I'm uncertain of the real-world sustained speeds over these connections

Questions

1. If anyone has experience to share regarding Thunderbolt on the QNAP I would appreciate that. Is it reliable? Would it be a better option than traditional 10GbE or SFP+ if you are not trying to get an entire network of devices connected at those speeds?
2. For those using 10GbE, what sustained transfer rates are you seeing on a regular basis?
3. Finally, is 6th or 7th gen i5 processor enough to push 1000MB/s read and write? Should I only be considering i7 models?

Thanks for any community input to help me come to a decision!

[Dan Wells]

I've got my TVS-682T sitting here next to me, and it is extremely fast connected via Thunderbolt 2... I've seen speeds as high as half a gigabyte (4 gigabits) per second sustained over significant periods of time (provided you've got something fast enough to feed it - realize that almost nothing ELSE is that fast, so it's hard to get data to it at that speed -I suspect the NAS would actually go even faster with a fast enough source). I saw those speeds copying off a fast (but SATA) Samsung SSD on my Mac's other Thunderbolt bus, and I suspect I may have actually hit the limit of the SSD or the SATA protocol, rather than the QNAP. My configuration is QTier with two Samsung 512 GB SSDs as a speed tier, with four 8 TB Hitachi Ultrastars as the capacity tier (no M.2 cache, although I may install it someday).

For perspective on how fast half a gigabyte per second is, that's wire speed for USB 3.0 (but no variant of USB ever comes CLOSE to its rated wire speed - I've yet to see USB 3 go over 200-250 megabytes per second, and that's only possible with a fast SSD or a RAID as a source drive). It's ALMOST SATA 3 wire speed, which is why I suspect I may have been limited by the source, not the QNAP destination. The only two possible drives that go significantly faster than half a gigabyte per second are a PCIe SSD (either directly connected or over Thunderbolt 2 or 3) or a RAID composed of either two or more SATA SSDs or five or more hard drives (connected via PCIe, Thunderbolt or Fibre Channel).

10 Gb Ethernet should have a wire speed of about a gigabyte per second, significantly too fast to feed with any but the most outrageous disk configurations (4 SSDs in RAID 0, for example) - but I've seen plenty of benchmarks that show 10 Gb Ethernet under most normal conditions being closer to 4Gb/second - still very fast, and faster than most devices can feed it, but only barely fast enough for the QNAP. (remember that half a gigaBYTE is 4 gigaBITS). Ethernet has a lot of overhead, especially at 10 Gb - and USB of any sort has a ridiculous amount of overhead, at least on a Mac (FireWire 400 was always a lot faster than USB 2.0, even though USB 2.0 was theoretically 480 Mb/second)

I've never used either 10 Gb Ethernet or Thunderbolt 3, but I suspect the performance of all three would be pretty similar, since there are so few devices that can feed anything faster than 10 gigabits. I've seen a few devices around the half gigabyte mark, but I've used a lot of high-end storage configurations and never seen anything except PCIe internal SSDs really beat that (I've never gone from one PCIe SSD to another, which has the highest possible wire speed). The most outrageous configuration I've personally used is an 8 bay Thunderbolt 2 RAID tower copying from a 16 bay Fibre Channel (8 gigabit) RAID, and that was right around the speed of the 682T. This thing's FAST!

While my 682T is only a week old, so I (by definition) have no comments on anything longer term than setup and initial transfer of files to it, it compares favorably so far to other high speed storage devices I have used. In contradiction to the comments in the other x82T thread, I don't find my 682T any louder than I'd expect for a similar storage device (4+ bay NAS or multi-bay high-speed DAS). It's not any different to set up than any other NAS (enabling the Thunderbolt is trivial) - although it is considerably more complex than a self-contained (hardware RAID)DAS.

Yes, it's expensive, but all really high-speed storage is... At high capacities, much of the cost is the disks themselves - it's easy to end up with $1800 or more worth of disks (4 8TB drives and a couple of SSDs) in a $1800 682T enclosure, and the ratio is higher for the units with more bays. ANY NAS with a connection faster than Gigabit Ethernet is going to be in that price range (the discontinued Synology 2015xs had 10Gb Ethernet, but ran around $1500 empty, and the Netgears with 10 Gb are in the same range once you subtract the included disks). Nobody other than QNAP makes a Thunderbolt NAS.

Fast (non-USB) RAID DAS boxes also tend to be well over $1000 for 4-6 bay units after you subtract the cost of any included drives, especially if they have SSD caching (the exception is the new Drobo 5Dt, which is around $700 - I haven't found a real review, and Drobos have previously tended to be convenient, but slow). Most other Thunderbolt DAS units under $1000 (the OWC ThunderBay units, for example) don't have a serious processor on board, and use soft RAID software on your computer (if they have enough bays to run RAID 5) - inconvenient to move from one computer to another, uses a significant amount of processing power and RAM, and not the easiest thing to restore if something goes wrong...). Fibre Channel is the other possible high-speed DAS interface, and that's a LOT of money, and trickier than a NAS to handle.

It's a shame QNAP doesn't make these things with one high-speed interface rather than two. How many people will actually use both Thunderbolt and 10 Gigabit? If each high speed interface adds a couple of hundred dollars or more to the price, they could produce a Thunderbolt only version around $1500 (maybe cutting one or two other features as well).

[jds580s]

Thank you for sharing all that wonderful information, and your insights / personal experience with the 682T.

One of my hopes is that I can hang onto my next NAS for a long time and it will still feel relatively speedy, so having bandwidth and horsepower to spare is a huge bonus. In the 6.5 years that I've owned my TS-459 Pro, technology has certainly marched on, and that NAS was unable to keep up because it's spinning drives were able to saturate it's 1GbE port on day one. While today I only have one system that has an M.2 NVME drive and Thunderbolt 3 and no systems with 10GbE cards, I suspect in the coming years everything will move in that direction around here. It will be nice to have connection speeds that are a close match to both the maximum speed of the internal drives in the NAS, and the drives in the connected computers.

The TVS-1282T3 demo QNAP posted on Youtube shows a Blackmagic Speed Test score of 1095 Write / 1325 Read over TB3.
I seem to be reading on the forum that 10GbE will result in around 800MB/s real world read & write when paired with SSDs, or enough spinning disks in RAID to keep up with that.

I think I am leaning towards waiting for one of the Thunderbolt 3 devices to ship at this point. If nothing else it will give me lots of connection options.

As you have more time using your 682T I would love to know how stable it has been for you.

Thanks!

[sterny72]

Did you get the TB3 model? How were the speeds?

[jds580s]

I ended up going with the vanilla TVS-1282 and will be adding in a 10GbE card once I get a matching 10gig switch and NIC.
I have installed NVMe cards in two of the three PCIE slots. I can only test their speeds internally since the 1GbE interface is the obvious bottleneck for now, but it is able to achieve 1707.42MB/s write and 3544.22MB/s read.