Page 1 of 2

Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 1:27 am
by robske_110
Hi!
I got a TS-873 with 6 Seagate Exos 14TB installed. I was planning to use them in a RAID5 configuration with a 10GbE card and was expecting speeds of at least 900MB/s for seq. reads.
However, over SMB I seem to only be able to reach maximums of 600MB/s seq. read and at most 500MB/s seq. write.
I already tested the 10GbE connection with iperf3, and it reports it being stable around 9.9Gbit/s. (Even with simultaneous bidirectional testing) (I also used 9000 Jumbo MTU)

Is there anything that comes up on your minds that could cause this? qnap reports a RAID5 of 8 SSDs was able to reach up to 2300MB/s (using a dual 10GbE connection). Since the HDDs I am using are well able to do sequential 230MB/s (rated for 260) I expected that a RAID5 of 6 HDDs should at least be able to reach speeds around 10Gbit/s.
I also, for testing, made a RAID0 out of the 6 drives but I am still getting the same speeds, which leads me to think that the bottleneck is defiantly not the drives.

I'm using QTS 4.3.6.1050
dv2.png
dv2.png

Re: Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 1:45 am
by Toxic17
you could run the internal tests as well.

SSH to your NAS.

then type in the following commands, cut and paste the results into a post (use the BBcode quotes/code) to make them readable in a post.

Code: Select all

qcli_storage -d

Code: Select all

qcli_storage -T force=1

Code: Select all

qcli_storage -t force=1
is there any raid syncing or scrubbing going on?

Re: Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 1:54 am
by Toxic17
I take it your using a new CAT6 or higher cable? if the PC directly connected to the 10GbE port or through a switch?

Re: Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 2:05 am
by Trexx
Make sure you have SMB 3.x enabled, which is not the default setting on the QNAP. Which 10GbE card did you put in your QNAP? Also would need more details about the client machine (HW, 10GbE card, OS, etc.)

Also check that your HDD FW is the same on all drives. I remember seeing a post in the forum recently related to slow performance and it was due to a FW mismatch between the drives.

Re: Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 2:22 am
by robske_110
Thank you for your great responses!
Toxic17 wrote: Fri Sep 06, 2019 1:45 am you could run the internal tests as well.

SSH to your NAS.

then type in the following commands, cut and paste the results into a post (use the BBcode quotes/code) to make them readable in a post.

Code: Select all

qcli_storage -d

Code: Select all

qcli_storage -T force=1

Code: Select all

qcli_storage -t force=1
is there any raid syncing or scrubbing going on?
very great that one can run internal tests, here are the results:

Code: Select all

[~] # qcli_storage -d
Enclosure  Port  Sys_Name     Type      Size      Alias          Signature   Partitions  Model  
NAS_HOST   1     /dev/sda     SSD:data  931.51 GB M.2 SSD 1      QNAP FLEX   5           ATA CT1000MX500SSD4 
NAS_HOST   3     /dev/sde     HDD:data  12.73 TB  Disk 1         QNAP FLEX   5           Seagate ST14000NM0018-2H4101
NAS_HOST   4     /dev/sdd     HDD:data  12.73 TB  Disk 2         QNAP FLEX   5           Seagate ST14000NM0018-2H4101
NAS_HOST   5     /dev/sdg     HDD:data  12.73 TB  Disk 3         QNAP FLEX   5           Seagate ST14000NM0018-2H4101
NAS_HOST   6     /dev/sdf     HDD:data  12.73 TB  Disk 4         QNAP FLEX   5           Seagate ST14000NM0018-2H4101
NAS_HOST   7     /dev/sdc     HDD:data  12.73 TB  Disk 5         QNAP FLEX   5           Seagate ST14000NM0018-2H4101
NAS_HOST   8     /dev/sdb     HDD:data  12.73 TB  Disk 6         QNAP FLEX   5           Seagate ST14000NM0018-2H4101
[~] # qcli_storage -T force=1                                                                                                       
fio test command for physical disk: /sbin/fio --filename=test_device --direct=1 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
fio test command for RAID: /sbin/fio --filename=test_device --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
Start testing!
Performance test is finished 100.000%...
Enclosure  Port  Sys_Name      Throughput    RAID        RAID_Type    RAID_Throughput   Pool  
NAS_HOST   1     /dev/sda      537.02 MB/s   /dev/md1    Single       538.50 MB/s       1     
NAS_HOST   3     /dev/sde      258.38 MB/s   /dev/md2    RAID 0       1.02 GB/s         288   
NAS_HOST   4     /dev/sdd      245.01 MB/s   /dev/md2    RAID 0       1.02 GB/s         288   
NAS_HOST   5     /dev/sdg      254.98 MB/s   /dev/md2    RAID 0       1.02 GB/s         288   
NAS_HOST   6     /dev/sdf      244.61 MB/s   /dev/md2    RAID 0       1.02 GB/s         288   
NAS_HOST   7     /dev/sdc      240.62 MB/s   /dev/md2    RAID 0       1.02 GB/s         288   
NAS_HOST   8     /dev/sdb      249.80 MB/s   /dev/md2    RAID 0       1.02 GB/s         288   
[~] # qcli_storage -t force=1
fio test command for LV layer: /sbin/fio --filename=test_device --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
fio test command for File system: /sbin/fio --filename=test_device/qcli_storage --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 --size=128m &>/tmp/qcli_storage.log
Start testing!
Performance test is finished 100.000%...
VolID   VolName             Pool     Mapping_Name            Throughput      Mount_Path                    FS_Throughput
1       DataVol1            1        /dev/mapper/cachedev1   538.30 MB/s     /share/CACHEDEV1_DATA         492.31 MB/s
2       DataVol2            288      /dev/mapper/ce_cachedev21.09 GB/s       /share/CE_CACHEDEV2_DATA      977.00 MB/s
[~] # 
Looks the RAID (DataVol2) is able to do 1.02GB/s according to the first test command, and at least 977MB/s according to the second.

probably didn't clarify that enough, but at the moment I have the 6 drives in a RAID0 until I figure out what slows performance. (but when i tested the RAID5 the syncing was completed)
Toxic17 wrote: Fri Sep 06, 2019 1:54 am I take it your using a new CAT6 or higher cable? if the PC directly connected to the 10GbE port or through a switch?
yes I am using a CAT6A cable, and the Computer is directly connected.
Trexx wrote: Fri Sep 06, 2019 2:05 am Make sure you have SMB 3.x enabled, which is not the default setting on the QNAP. Which 10GbE card did you put in your QNAP? Also would need more details about the client machine (HW, 10GbE card, OS, etc.)

Also check that your HDD FW is the same on all drives. I remember seeing a post in the forum recently related to slow performance and it was due to a FW mismatch between the drives.
I don't know where to enable SMB 3.x, but will figure that out and test it later.
How do I check the HDD FW in the NAS, I couldn't find anything on that yet. (Or do I need to pull the drives and stick them into a computer for that?)
10GbE card is the QXG-10G1T.
The client machine is a MacBook Pro (2016) with macOS with a Thunderbolt OWC 10GbE adapter.

Re: Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 2:34 am
by robske_110
SMB3 was enable (Highest was set to SMB3), and I also tested again after setting lowest to SMB3 too, but I still only get 450 write and 560 read.
Still don't know where to check HDD firmware though.
And I don't think it's the network connection itself, since an iperf3 test between the macbook and the nas says this:

Code: Select all

bash-3.2$ iperf3 --bidir -c 192.168.5.1
Connecting to host 192.168.5.1, port 5201
[  5] local 192.168.5.2 port 56433 connected to 192.168.5.1 port 5201
[ ID][Role] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.15 GBytes  9.86 Gbits/sec                  
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec                  
[  5]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec                  
[  5]   3.00-4.00   sec  1.15 GBytes  9.90 Gbits/sec                  
[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec                  
[  5]   5.00-6.00   sec  1.15 GBytes  9.90 Gbits/sec                  
[  5]   6.00-7.00   sec  1.15 GBytes  9.89 Gbits/sec                  
[  5]   7.00-8.00   sec  1.14 GBytes  9.83 Gbits/sec                  
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec                  
[  5]   9.00-10.00  sec  1.13 GBytes  9.73 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  11.5 GBytes  9.87 Gbits/sec                  sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.87 Gbits/sec                  receiver

iperf Done.

Re: Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 3:15 am
by Trexx
Make sure you have packet signing turned off.
https://support.apple.com/en-us/HT205926

Disk FW is under storage/snapshots > disks/vjbod > disk info button


Sent from my iPad using Tapatalk

Re: Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 3:23 am
by robske_110
Unfortunately turning packet signing off didn't have any impact.

Firmware matches (all have FN02)

Re: Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 4:49 am
by MrVideo
What surprises me is that none of the respondents said anything about using RAID5 with that many drives and drives of that size. You should be using RAID6, otherwise, IMHO, it is a disaster waiting to happen.

Re: Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 5:02 am
by Trexx
MrVideo wrote: Fri Sep 06, 2019 4:49 am What surprises me is that none of the respondents said anything about using RAID5 with that many drives and drives of that size. You should be using RAID6, otherwise, IMHO, it is a disaster waiting to happen.
He had stated previously he had tried raid-5/6 and was using raid-0 to test parity overhead impact on his low perf #'s. Didn't sound like was going to be a permanent configuration.

Re: Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 5:05 am
by Trexx
robske_110 wrote: Fri Sep 06, 2019 3:23 am Unfortunately turning packet signing off didn't have any impact.

Firmware matches (all have FN02)
Here are a few more things to try. https://community.spiceworks.com/how_to ... rom-mac-os

Which OSX version are you running? I do recall a couple people saying they ended up resolving their slow perf issues by basically wiping their laptops and re-installing OSX on their laptops from scratch.

Re: Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 5:12 am
by Toxic17
What 10GbE cards are you using?

I'm sure Bob Zelin will have a fix :)

viewtopic.php?f=11&t=141272&start=15#p671452

Re: Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 5:48 am
by robske_110
Thank you once again for your endless supply of nice and informative replies, I really didn't expect to get so much help :D
Trexx wrote: Fri Sep 06, 2019 5:05 am
robske_110 wrote: Fri Sep 06, 2019 3:23 am Unfortunately turning packet signing off didn't have any impact.

Firmware matches (all have FN02)
Here are a few more things to try. https://community.spiceworks.com/how_to ... rom-mac-os

Which OSX version are you running? I do recall a couple people saying they ended up resolving their slow perf issues by basically wiping their laptops and re-installing OSX on their laptops from scratch.
I've done the first recommendation from that article, but am unsure how to apply the third one on a qnap NAS (if even necessarry/possible)
Also the second doesn't seem to be an option in either the qnap adapter settings or the macOS ones.

I am running macOS 10.14.6. I'm going to see if I can test with another mac (and also Windows, although I'd like to have full speeds with macOS too) tomorrow.
Toxic17 wrote: Fri Sep 06, 2019 5:12 am What 10GbE cards are you using?

I'm sure Bob Zelin will have a fix :)

viewtopic.php?f=11&t=141272&start=15#p671452
In the NAS: QXG-10G1T
On the client: OWC Thunderbolt 3 10GbE Adapter

And since iperf3 is reporting (almost) full duplex 10Gbit/s, I really think this is a problem with either the NAS or the macOS SMB client and not the actual 10GbE connection.

Re: Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 6:54 am
by Bob Zelin
it sounds like you have a Win 10 PC with a thunderbolt 3 port, and you are using the OWC T3 to 10G adaptor. I have no idea about your PC. If I am wrong, and you are using a Mac (you just said about the macOS SMB client) -
make sure to have SMB signing disabled in terminal, and reboot your machine. And enable Jumbo Frames in both System Preferences for your 10G port, and on the QNAP 10G port.
Why are you only using 6 drives in a TVS-873 - put eight drives in there. If you follow my instructions, you will get 800 MB/sec (and probably not more than 500 MB/sec write).
And stop staring at the specs on the QNAP website (2300 MB/sec) - I have no idea of what they are referring to. With a 10G connection, even theoretical bandwidth can never exceed 1250 MB/sec, and in real life, I have never
seen over 1100 MB/sec on ANY 10G client - no matter what. 2300 MB/sec probably means total aggregate bandwidth - and there is no way that a TVS-873 is going to perform like this. There is real world, and there is fantasy.

Bob Zelin

Re: Performance issues using 6-drive RAID on a TS-873

Posted: Fri Sep 06, 2019 10:59 am
by MrVideo
Trexx wrote: Fri Sep 06, 2019 5:02 amHe had stated previously he had tried raid-5/6 and was using raid-0 to test parity overhead impact on his low perf #'s.
Not that I saw. Only saw a RAID5 comment in the first post.