RAID 60 vs RAID 10 - Initialization Time and Performance

[kolakolakola]

Hi all, I've just purchased a TS-1685 w/ 32GB ram and (12) Seagate Exos 14TB drives to populate it with.
The firmware has been updated to 4.3.6.0993 and all systems and disk report as good.

Using the HDD Performance Analyzer in QNAP Diagnostics tool, each of the twelve disks report throughput between 235-260 MB/s. That said, I'm struggling to find a configuration that gives me the performance I'd expect.

My original plan was to run all drives at RAID 60. It took about 24 hours to finish "synchronizing" after initialization.
However, in RAID 60 the File System Performance Analyzer in QNAP Diagnostics tool reports write performance at 380 MB/s and read performance at 785 MB/s. I was expecting read performance of at least 1300 MB/s or so.

Disappointed with this performance, I tried setting up a RAID 10. This reported about 60 hours to finish "synchronizing" after initialization, and still only reported write performance at 850 MB/s and read performance at 1300 MB/s. Again, here I was expecting 2000 MB/s or more.

These were all configured as static volumes, and synchronizations were done set at "Sync Priority."

Are my expectations simply too high, or does it seem like things are running slower than they should?
Also, shouldn't RAID 10 synchronization be significantly faster than RAID 60? I should also note that CPU and RAM utilization never exceeds 10% even while synchronizing.
Are there other things I need to look at or should consider? I would still consider RAID 6, but am weary of it with this many drives of this high capacity.
Any thoughts or suggestions would be appreciated!

[P3R]

Are my expectations simply too high, or does it seem like things are running slower than they should?

Had the RAID arrays finished synchronizing in both cases when testing performance?

I would still consider RAID 6, but am weary of it with this many drives of this high capacity.

With 12 disks, RAID 6 have better reliability than a RAID 10.

Performance with RAID 6 should be way better than RAID 60 and I would expect it to be faster than RAID 10 also.

[kolakolakola]

Had the RAID arrays finished synchronizing in both cases when testing performance?

Yup, they had finished synchronizing and optimizing.

With 12 disks, RAID 6 have better reliability than a RAID 10.

I keep hearing mixed things about this, and one of the failure calculators indicated a pretty significant chance of failure during rebuild for a RAID 6 volume of this size. But I'll try RAID6 next and see how it fares. Truthfully, that was my original plan, but from my reading everyone seemed like failure during rebuild could be a serious problem. Then I thought about RAID 60 and thought that might solve all my problems!

Performance with RAID 6 should be way better than RAID 60 and I would expect it to be faster than RAID 10 also.

I was under the impression that with a 12 disk array RAID 6 would theoretically provide a read performance of 10x (n-2) and RAID 10 would provide a read performance of 12x (n). With RAID 60 I was expecting 8x. Is that not the case?

[P3R]

I keep hearing mixed things about this, and one of the failure calculators indicated a pretty significant chance of failure during rebuild for a RAID 6 volume of this size.

Well of course the RAID 6 will be more vulnerable than normal during the rebuild but it will still tolerate a second failure of ANY disk and it's not until the third disk fails that the array is lost.

It may not be by a huge margin but RAID 6 should still be more reliable than RAID 10 with 12*14 TB Exos I'd say.

But I'll try RAID6 next and see how it fares. Truthfully, that was my original plan, but from my reading everyone seemed like failure during rebuild could be a serious problem.

Well the hate against RAID 5 took off with this incorrect article
12 years ago. With a lot of less educated people repeating the myth, it have snowballed and now spill over on RAID 6 as well...

RAID 10 have applications where it's superior (mainly very IOPS-intensive) but reliability isn't a strong point for it with a smaller number (12 or less) of disks compared to RAID 6. For high throughput like you're testing, RAID 5/6 are usually faster with the same number of disks and a decent CPU.

Then I thought about RAID 60 and thought that might solve all my problems!

Reliability will be fantastic with RAID 60 but 12 disks is too few to get great performance from it.

I was under the impression that with a 12 disk array RAID 6 would theoretically provide a read performance of 10x (n-2) and RAID 10 would provide a read performance of 12x (n). With RAID 60 I was expecting 8x. Is that not the case?

I guess it should be in theory but since I've focused on testing write performance, I really don't know much about actual results from read testing.

I bet that the RAID 6 will be faster when writing and I'm pretty sure the read speed will be decent as well. Please report your findings.

[kolakolakola]

Thanks P3R - the RAID6 pool finished synchronizing and optimized last night - took about 24 hours.
This morning the new RAID6 pool only reports 620 MB/s write and 965 MB/s read. So you're right - it did end up even faster than the RAID 10, but still seems low, right?
Each of the individual disks is still reporting 230-255 MB/s and healthy.

[P3R]

My 8-disk RAID 6 test give 200.24 MB/s write and 431.17 MB/s read. Five of my disks are old and slow so only perform at 121-122 MB/s but it's clear that you can't simply expect the disk test result multiplied by the number of data disks.

As we don't know what that Qnap test does there's not much point in discussing the numbers it produces. I think that test can be useful to do comparisons between systems of a similar configuration but when it comes to real performance testing, nothing beats having a workstation writing and reading to/from the NAS over the network. That's when testing become relevant.

[kolakolakola]

nothing beats having a workstation writing and reading to/from the NAS over the network.

Agreed, but my problem there is I'm already basically saturating a 10gbe link, so don't have a great way to measure performance to/from a client beyond about 900 MB/s.
#firstworldproblems

I ended up talking with QNAP, and they did confirm that the QNAP diagnostics tool is pretty old, and the better way to do a local throughput test is using CLI tools via SSH:
qcli_storage -t (reports the throughput of an overall volume)
qcli_storage -T (reports the throughput of each individual disk)

When clocking my disks, they were still all reporting between 230-255 MB/s. When clocking my volume, it was clocking at only 830 MB/s. QNAP confirmed that this was pretty slow, and was going to do some more testing on their end to see what they found.

I also did some more poking around, and may have found a big part of my issue... 8 of my disks were running SN02 firmware, while 4 of them were running SN01. I wouldn't have guessed it would matter, since they were still clocking the same, but I went ahead and pulled those 4 disks and upgraded them to SN02.

After upgrading the firmware, qcli_storage -t reports jumped from 830 MB/s to 1.5 GB/s. Still a bit slower than I'd like, but definitely a more realistic expectation.

So for now, I'm relatively satisfied and can consider this resolved. Thanks for your help!!

[P3R]

After upgrading the firmware, qcli_storage -t reports jumped from 830 MB/s to 1.5 GB/s.

I wouldn't have thought that different disk firmwares could make such a huge difference!

So for now, I'm relatively satisfied and can consider this resolved.

Great!