Speed comparison: comparing disk #, raid, CPU, encryption

Introduce yourself to us and other members here, or share your own product reviews, suggestions, and tips and tricks of using QNAP products.
qpio
Getting the hang of things
Posts: 61
Joined: Wed May 11, 2011 6:51 am

Speed comparison: comparing disk #, raid, CPU, encryption

Post by qpio »

I have been able to do some structured testing of different RAID levels, different number of disks, and using different CPU models. I find these results quite interesting myself, so I decided to post them here!

Testing setup:
1. NAS with 8 bays with Atom D2700 dual core @ 2.13 GHz (not a QNAP model, but that 'other' brand, wink wink, psst. the 1813+)
2. QNAP TS-870 with Celeron G550 dual core CPU @ 2.6 GHz
3. QNAP TS-870 with i7 3770t quad core CPU @ 2.5 GHz
4. Disks used (always using disk 1-n when testing with 'n' disks):
disk 1-4: Hitachi Deskstar 1TB 7200 rpm
disk 5-7: WD GREEN 1TB
disk 8: WD GREEN 2TB
5. Disable syncing (echo idle > /sys/block/md1/md/sync_action) -- avoid 10% performance hit
6. Disable thin provisioning -- avoid 15% performance hit

I have tested with a very simple dd command on the NAS itself, and I have looked only at the write performance:
time dd if=/dev/zero of=/share/MD0_DATA/homes/admin/blah bs=4M count=2048 conv=sync
After this, I looked at the 'real' time spent (wall clock time elapsed). For instance with 30.00 seconds of 'real' time:
4*2048/30.00
273.06666666666666666666
I would write down 273.1 MB/s as write speed. All tests were performed at least twice, and if unexpectedly low performance occurred (e.g. measuring 167.0 MB/s after a 273.1 MB/s) then it's fair to make another run and ignore the bad run. That has only happened on very few occasions.

So here we go...

1. NAS with 8 bays with Atom D2700 dual core @ 2.13 GHz

3 disks in raid-5 72.11 MB/s 76.35 MB/s
4 disks in raid-5 167.1 MB/s 176.2 MB/s
5 disks in raid-5 210.5 MB/s 176.2 MB/s
6 disks in raid-5 194.1 MB/s 198.3 MB/s
7 disks in raid-5 199.4 MB/s 215.6 MB/s
8 disks in raid-5 229.4 MB/s 200.6 MB/s

4 disks in raid-6 113.6 MB/s 110.8 MB/s
5 disks in raid-6 154.8 MB/s 135.3 MB/s
6 disks in raid-6 175.8 MB/s 165.3 MB/s
7 disks in raid-6 190.5 MB/s 192.6 MB/s
8 disks in raid-6 196.0 MB/s 177.9 MB/s

4 disks in raid-10 155.2 MB/s 147.7 MB/s
6 disks in raid-10 227.4 MB/s 218.8 MB/s
8 disks in raid-10 238.9 MB/s 249.5 MB/s

2. QNAP TS-870 with Celeron G550 dual core CPU @ 2.6 GHz

3 disks in raid-5 135.3 MB/s 134.8 MB/s
4 disks in raid-5 202.1 MB/s 203.4 MB/s
5 disks in raid-5 275.4 MB/s 268.0 MB/s
6 disks in raid-5 319.8 MB/s 325.3 MB/s
7 disks in raid-5 364.7 MB/s 373.2 MB/s
8 disks in raid-5 356.4 MB/s 382.8 MB/s

4 disks in raid-6 133.9 MB/s 133.3 MB/s
5 disks in raid-6 207.8 MB/s 211.7 MB/s
6 disks in raid-6 270.1 MB/s 271.4 MB/s
7 disks in raid-6 314.6 MB/s 317.6 MB/s crypted: 123.1 MB/s 125.5 MB/s
8 disks in raid-6 338.1 MB/s 353.1 MB/s crypted: 124.5 MB/s 124.7 MB/s

4 disks in raid-10 147.2 MB/s 150.4 MB/s
6 disks in raid-10 229.3 MB/s 234.8 MB/s
8 disks in raid-10 295.2 MB/s 298.6 MB/s

3. QNAP TS-870 with i7 3770t quad core CPU @ 2.5 GHz

3 disks in raid-5 155.6 MB/s 156.1 MB/s
4 disks in raid-5 241.8 MB/s 230.9 MB/s
5 disks in raid-5 325.2 MB/s 316.5 MB/s
6 disks in raid-5 403.8 MB/s 364.8 MB/s
7 disks in raid-5 479.0 MB/s 434.9 MB/s
8 disks in raid-5 497.6 MB/s 491.0 MB/s crypted: 424.5 MB/s 400.1 MB/s

4 disks in raid-6 155.2 MB/s 159.5 MB/s crypted: 134.5 MB/s 127.9 MB/s
5 disks in raid-6 245.9 MB/s 221.9 MB/s crypted: 184.1 MB/s 189.1 MB/s
6 disks in raid-6 324.2 MB/s 315.8 MB/s crypted: 253.4 MB/s 236.3 MB/s
7 disks in raid-6 354.5 MB/s 350.6 MB/s crypted: 297.7 MB/s 293.3 MB/s
8 disks in raid-6 407.4 MB/s 419.8 MB/s crypted: 352.6 MB/s 351.3 MB/s

4 disks in raid-10 166.0 MB/s 159.4 MB/s
6 disks in raid-10 254.1 MB/s 257.4 MB/s
8 disks in raid-10 347.9 MB/s 316.2 MB/s


Some careful conclusions:

(1) when comparing RAID-5 and RAID-6:
RAID-5 and RAID-6 are comparable when you look at the same number of data disks (e.g. 2+1 in RAID-5 compares to 2+2 in RAID-6)

(2) when comparing the number of data disks:
increasing from 2 to 3, and again from 3 to 4 data disks, gives you a dramatic improvement in write speed (i.e. more significant than, say, WD red versus WD green?)

(3) comparing CPU's:
(a) the Atom is overall slower and runs out of steam beyond roughly 4 data disks in RAID-5 and RAID-6
(b) the G550 and i7 are much faster, both run out of steam beyond roughly 6 data disks in RAID-5 and RAID-6
(c) the i7 has a major advantage with encryption, due to its instruction set with AES support

(4) when considering RAID-10
The G550 and i7 CPU's are more than capable to perform the XOR operations at high speeds. In the numbers you can see that RAID-10 only gives a slight advantage when running on the Atom CPU. Also, note that only write speeds where compared. The striping advantage of RAID-10 is only effective when reading.

I have also looked at power consumption during these various tests. It looks to me that the G550 and i7 have a very much comparable power usage footprint in real life. Eventhough in theory the TDP of the G550 is higher, that doesn't mean that the real-life power usage (with RAID, in a NAS) is so much different between the i7 and G550. Of course if you look at power usage per unit of work, instead as of per unit of time, then the i7 wins because it's faster. At idle they're both comparable as well.

The numbers for a QNAP TS-870 with i3 CPU (the 870 Pro) would be interesting to add to this list. If anyone feels like wasting a full weekend doing such tests, please post your results here. :-)
TS-872, TS-870 Pro, TS-459 Pro II, TS-112
numgis
New here
Posts: 3
Joined: Tue Nov 26, 2013 9:31 am

Re: Speed comparison: comparing disk #, raid, CPU, encryptio

Post by numgis »

Thanks for posting this! Just setting up a new TVS-871U-RP (i3 processor) with 8x 4TB WD Red Pro (7200rpm) drives and was wondering whether to go RAID6 or RAID10 for performance. Looking at your number it seems RAID10 is actually slower than RAID6 which is great news, some other posts I've read are really slamming RAID6 for being slow. Since I have an i3 which is better than the G550 I assume I should get similar numbers to you where RAID6 is faster than RAID10. Thanks for helping me make the decision!! :)
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Speed comparison: comparing disk #, raid, CPU, encryptio

Post by P3R »

numgis wrote:Looking at your number it seems RAID10 is actually slower than RAID6...
Yes but only when comparing an equal (and also higher) number of disks. If looking at equal usable storage capacity:
4*RAID 6 should be compared with 4*RAID 10
6*RAID 10 should be compared with 5*RAID 6
8*RAID 10 should be compared with 6*RAID 6

and as qpio noted
4*RAID 6 should be compared with 3*RAID 5
5*RAID 6 should be compared with 4*RAID 5
6*RAID 6 should be compared with 5*RAID 5
7*RAID 6 should be compared with 6*RAID 5
8*RAID 6 should be compared with 7*RAID 5
...some other posts I've read are really slamming RAID6 for being slow.
Not only is RAID 6 undeservingly put down because of it's rumoured bad performance. Many also think RAID 10 is safer than RAID 6, when in reality it's the other way around (at least when talking about "small" disk arrays like 8 or less disks).

RAID 6 is a great choice for home/SMB servers with 5-12 disks!

Some things to remember about this test is that it is purely "synthetic" and doesn't include network performance and different types of loads that, among other things, also affects real world numbers. And as already mentioned, this test only measures write performance, which is only a part of the picture.

@qpio,
Thank you for taking the time to do this testing! It's very interesting data now that 6-8 bay models with these faster CPUs have become very common among advanced home/SMB users.

Regarding that you say RAID 10 have a striping advantage when reading. RAID 5 and 6 also benefit from striping so that's not really an advantage for RAID 10.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
sebastienbo
Know my way around
Posts: 123
Joined: Tue Jul 14, 2015 5:44 am

Re: Speed comparison: comparing disk #, raid, CPU, encryption

Post by sebastienbo »

What baffeled me was the raid 10 performance, I thought it would be so much more faster

I was just deciding between having 4 disks in Raid 10 instead of 4 disks in raid 5, but in raid 5the 4th disk really increases the throughput... plus I get more net capacity
I find raid 6 uselsess because the idea is that you have two disks that may fail before the raid dies instead of 1 with raid 5, but if you really think that way, why are you only thinking about the disk failure? the hole nas could be gone...
SO what I do is just have another nas on another site with a cheap disk and that is getting online sync data, that way I can setup raid 5 with 4 disks and still be sure that any kind of faults with my primary nas, my secondary nas will have a copy!

Thanks for the comparissons

I'm gonnna set it up as raid 5 with 4 disks, actually it makes much sense that the 4th disk suddenly increases speed: all three disks are used when you write something 2 for data + 1 for checksum that means that your nas is locked for waiting.
But with a 4th disk, the nas can get some blocks from the 4th disk that is doing nothing at that moment, so this extra disk can fill the "waiting time" that the raid has during a raid 5 write action
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Speed comparison: comparing disk #, raid, CPU, encryption

Post by P3R »

sebastienbo wrote:What baffeled me was the raid 10 performance, I thought it would be so much more fast...
Yes that's the internet myth that so many are blinded by... :roll:

It's not RAID 10 that is slow, it's RAID 5 and 6 that are much faster than their reputation on most modern NAS models.
I find raid 6 uselsess because the idea is that you have two disks that may fail before the raid dies...
RAID 6 offer the same storage capacity as RAID 10 (with 4 disks, with more disks it's better than RAID 10), have better redundancy, and almost the same performance. Hmm, I wonder which RAID level is useless of the two...
...instead of 1 with raid 5...
Yes I agree that in a 4 bay NAS, RAID 6 isn't the first option for most home users but in larger NASes and with very large disks, RAID 5 is risky and RAID 6 is the logical next step for those that need better availability on their system.

During the rebuild of a disk being replaced, all disks need to be read completely and a read/write error on a second disk will destroy the whole RAID volume. It does happen, especially when all disks are bought and installed at the same time. Example from today here. RAID 6 will protect from that.
...but if you really think that way, why are you only thinking about the disk failure? the hole nas could be gone...
Yes of course it can but disks are by very, very far what fails most often.

The NAS models that have model name ending with -RP have redundant power supplies for those that need even better system redundancy.
...But with a 4th disk, the nas can get some blocks from teh 4th disk that is doing nothing at that moment, so this extra disk can fill the "waiting time" that the raid has during a raid 5 write action
I've read this several times and still don't follow you here.

To begin with all disks in a RAID 5 (regardless of 3, 4 or more disks) physically contain both data and checksums. A 4 disk RAID 5 is a bit faster than a 3 disk RAID 5 because there are more disks to spread the data over so less data have to be read from each disk and therefore each disk become less of a bottleneck (in a fast system). This data striping is also effective when writing but normally less visible because the checksum calculation give some overhead.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
User avatar
a13antichrist
Know my way around
Posts: 142
Joined: Tue Apr 17, 2018 10:06 pm
Location: Amsterdam, NL

Re: Speed comparison: comparing disk #, raid, CPU, encryption

Post by a13antichrist »

P3R wrote:
sebastienbo wrote:What baffeled me was the raid 10 performance, I thought it would be so much more fast...
Yes that's the internet myth that so many are blinded by... :roll:

It's not RAID 10 that is slow, it's RAID 5 and 6 that are much faster than their reputation on most modern NAS models.
Sorry but your/OP's original premise here is totally misguided.

"Number of Drives" is a stupid basis for comparison. Of course R5 will be faster in writes than a similarly-spindled R10. R10 means that only half the drives are being used for effective data throughput (doubled-up, everything is written twice).
If you compare on a capacity basis, you will find R10 thrashing R5.

Further, the critical advantage of R10 is in the sequential & random READS, where (in a proper implementation) you can read data from both sides of the mirror and both sides of the stripe together, which compounds your performance. You will see that R10 sequential reads utterly destroy any R5 setup, both on a spindle-count basis and (especially) on a capacity basis (where R10 naturally has effectively (almost) twice the number of spindles).
The other critical difference is that if a disk should be lost, R10 suffers --no-- performance penalty. Whereas a R5 is practically crippled, and this is without even mentioning the increased risk of a secondary failure during rebuild. On the other hand an R10 has --0-- impact to likelihood of failure on the other spindles during a rebuild.

The takeaway should be: if you -can- afford R10, it's by far the best option, all things considered; BUT if absolute write speeds over high numbers of spindles is your first and foremost priority then you should INVESTIGATE whether your workload might perform better under R5.

So, while the OP's numbers are not wrong, per se, the conclusion is faulty at best, and tending towards deception.
TVS-871 @ 58tb formatted R6, Mellanox MCX312b 10Gbe,
R9 5950x || RTX3090 || 64Gb 3600Mhz || Intel X520 10Gbe
MbP M1 Max || 32C GPU || 64Gb || QNAP T310G1S SFP+ 10Gbe
R3 2200G mini HTPC
Mikrotik switches & routers
OpenWRT & DD-WRT wifi
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Speed comparison: comparing disk #, raid, CPU, encryption

Post by P3R »

a13antichrist wrote:"Number of Drives" is a stupid basis for comparison.
If you'd spend some time in this forum you'll notice that a very frequent question is: I have X disks (X is usually the max number of 3.5" drive bays in the NAS like 4, 6 or 8), what RAID should I use for them?

Stupid or not, a fixed set of spindles is usually what we have to give advice on.
Of course R5 will be faster in writes than a similarly-spindled R10. R10 means that only half the drives are being used for effective data throughput (doubled-up, everything is written twice).
I wish you'd tell the RAID 10 fans that instead of me. It was always the write performance being pushed as the argument for RAID 10 in this forum before this nice comparison was posted.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
User avatar
a13antichrist
Know my way around
Posts: 142
Joined: Tue Apr 17, 2018 10:06 pm
Location: Amsterdam, NL

Re: Speed comparison: comparing disk #, raid, CPU, encryption

Post by a13antichrist »

P3R wrote:
a13antichrist wrote:"Number of Drives" is a stupid basis for comparison.
If you'd spend some time in this forum you'll notice that a very frequent question is: I have X disks (X is usually the max number of 3.5" drive bays in the NAS like 4, 6 or 8), what RAID should I use for them?

Stupid or not, a fixed set of spindles is usually what we have to give advice on.
Yes that's fair enough I guess. In terms of a user's question, agreed. In terms of principles of RAID, not so much.

Of course R5 will be faster in writes than a similarly-spindled R10. R10 means that only half the drives are being used for effective data throughput (doubled-up, everything is written twice).
I wish you'd tell the RAID 10 fans that instead of me. It was always the write performance being pushed as the argument for RAID 10 in this forum before this nice comparison was posted.
I think that's more of a miscommunication than a deliberate misunderstanding of the situation. I do think though that it's important to stress that R10 does have a very very real read performance advantage across the board; in this thread where "disproving the myth" is the aim, it should also present the actual truth so as to fully clarify the reasons for the myth in the first place - especially since the OP doesn't even mention "writes" at all (until figures are posted), simply "speed". I can see some people reading the thread (before my post) and going away thinking that "R10 actually isn't faster than R5 at all", which of course would be a huge mistake.
TVS-871 @ 58tb formatted R6, Mellanox MCX312b 10Gbe,
R9 5950x || RTX3090 || 64Gb 3600Mhz || Intel X520 10Gbe
MbP M1 Max || 32C GPU || 64Gb || QNAP T310G1S SFP+ 10Gbe
R3 2200G mini HTPC
Mikrotik switches & routers
OpenWRT & DD-WRT wifi
LittleBird
Starting out
Posts: 22
Joined: Thu Nov 29, 2012 10:38 am

Re: Speed comparison: comparing disk #, raid, CPU, encryption

Post by LittleBird »

Congratulations to @qpio for producing some hard data. Always nice to see your gut feel config. confirmed. @sebastienbo I've had maybe 7 or 8 hard drives fail over the years but only one QNAP expansion unit. An important consideration when updating drives is the "bathtub" failure curve. I had 2 drives fail within a day of updating a RAID 5 on my Server NAS. Normally not a big deal but I then realised I had updated my Backup NAS using disks from the same batch the previous day. So my only copy of 2 months of work resided on a NAS with disks just starting on their "bathtub" failure curve journey. The result was called "breaking out in a cold sweat". Fortunately the Backup NAS didn't fail. But I learnt 2 important lessons:
1 Use RAID6 inlieu of RAID5
2 When upgrading always allow a "burn in" time of at least 1 week (preferably 2) before upgrading your second data repository.
@P3R I agree completely with your three tips at the end of your posts. Perhaps you might consider my humble offering re the upgrade "bathtub" problem as worth adding to your tips.
User avatar
Spider99
Experience counts
Posts: 1951
Joined: Fri Oct 21, 2011 11:14 pm
Location: UK

Re: Speed comparison: comparing disk #, raid, CPU, encryption

Post by Spider99 »

As you are so convinced R10 is faster - care to provide some facts along with you assertions?
Tim

TS-853A(16GB): - 4.3.4.0483 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
User avatar
a13antichrist
Know my way around
Posts: 142
Joined: Tue Apr 17, 2018 10:06 pm
Location: Amsterdam, NL

Re: Speed comparison: comparing disk #, raid, CPU, encryption

Post by a13antichrist »

Contradicts everything we've said so far.

Gets the result on one and concludes the other must be bung ( also his R0 is faster than his R10 so obviously his controller doesn't know what it's doing).

Finds R10 faster, R5 dead last but R1 and even stand-alone drives faster than both.

RAID 10 destroying everything in sight.

Focusses on the theory and concludes per-spindle, R10 is "faster".

So, It's clear that the implementation actually matters at lot more than the RAID level, and so it doens't make sense to talk about which one is "actually" faster unless you're talking about raw principle, in which case the answers are much easier.

It's also -essential- to reference what scope you're talking about - Capacity, or Disk count, because the answers differ greatly depending on which is in play.
TVS-871 @ 58tb formatted R6, Mellanox MCX312b 10Gbe,
R9 5950x || RTX3090 || 64Gb 3600Mhz || Intel X520 10Gbe
MbP M1 Max || 32C GPU || 64Gb || QNAP T310G1S SFP+ 10Gbe
R3 2200G mini HTPC
Mikrotik switches & routers
OpenWRT & DD-WRT wifi
User avatar
dolbyman
Guru
Posts: 35024
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: Speed comparison: comparing disk #, raid, CPU, encryption

Post by dolbyman »

actual speed results would be a test like above, not links to decade old articles with different hardware .. we are talking QNAP here
User avatar
Spider99
Experience counts
Posts: 1951
Joined: Fri Oct 21, 2011 11:14 pm
Location: UK

Re: Speed comparison: comparing disk #, raid, CPU, encryption

Post by Spider99 »

Yes i was hoping you had actual facts/data to backup you claims - not convinced at all yet
Tim

TS-853A(16GB): - 4.3.4.0483 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
User avatar
a13antichrist
Know my way around
Posts: 142
Joined: Tue Apr 17, 2018 10:06 pm
Location: Amsterdam, NL

Re: Speed comparison: comparing disk #, raid, CPU, encryption

Post by a13antichrist »

dolbyman wrote:actual speed results would be a test like above, not links to decade old articles with different hardware .. we are talking QNAP here
I would trust professional thorough reviews far far greater above one user's anecdotal reports. Besides which, the OP is testing not just QNAP, so, that's invalid. And how are other sites & reviews any less "actual facts/data" than our OP's?

I only have a 431 here which isn't going to win awards for anything, but if someone sends me some test devices and some proper NAS drives I'll happily run them through the same tests as the OP.
TVS-871 @ 58tb formatted R6, Mellanox MCX312b 10Gbe,
R9 5950x || RTX3090 || 64Gb 3600Mhz || Intel X520 10Gbe
MbP M1 Max || 32C GPU || 64Gb || QNAP T310G1S SFP+ 10Gbe
R3 2200G mini HTPC
Mikrotik switches & routers
OpenWRT & DD-WRT wifi
User avatar
dolbyman
Guru
Posts: 35024
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: Speed comparison: comparing disk #, raid, CPU, encryption

Post by dolbyman »

as mentioned before, reviews done on hardware controllers vs. software RAID are not comparable, also the whole speed point gets mute when people are blowing through a 1GbE line (or Wifi) on a halfway decent current NAS (and are not using local containers or VM's).. 90% of users here are on a budget and RAID5 (+ regular backups) fits their budget perfectly and if they want extra redundancy they should go RAID6 (on a 4 bay unit undeniably better fault tolerance)

Why would you want to retest the OP's results ? If you really want to .. invest some $$$ and test away
Locked

Return to “Users' Corner”