Performance issues using 6-drive RAID on a TS-873

Questions about SNMP, Power, System, Logs, disk, & RAID.
robske_110
New here
Posts: 8
Joined: Fri Sep 06, 2019 12:53 am

Performance issues using 6-drive RAID on a TS-873

Post by robske_110 »

Hi!
I got a TS-873 with 6 Seagate Exos 14TB installed. I was planning to use them in a RAID5 configuration with a 10GbE card and was expecting speeds of at least 900MB/s for seq. reads.
However, over SMB I seem to only be able to reach maximums of 600MB/s seq. read and at most 500MB/s seq. write.
I already tested the 10GbE connection with iperf3, and it reports it being stable around 9.9Gbit/s. (Even with simultaneous bidirectional testing) (I also used 9000 Jumbo MTU)

Is there anything that comes up on your minds that could cause this? qnap reports a RAID5 of 8 SSDs was able to reach up to 2300MB/s (using a dual 10GbE connection). Since the HDDs I am using are well able to do sequential 230MB/s (rated for 260) I expected that a RAID5 of 6 HDDs should at least be able to reach speeds around 10Gbit/s.
I also, for testing, made a RAID0 out of the 6 drives but I am still getting the same speeds, which leads me to think that the bottleneck is defiantly not the drives.

I'm using QTS 4.3.6.1050
dv2.png
dv2.png
You do not have the required permissions to view the files attached to this post.
User avatar
Toxic17
Ask me anything
Posts: 6469
Joined: Tue Jan 25, 2011 11:41 pm
Location: Planet Earth
Contact:

Re: Performance issues using 6-drive RAID on a TS-873

Post by Toxic17 »

you could run the internal tests as well.

SSH to your NAS.

then type in the following commands, cut and paste the results into a post (use the BBcode quotes/code) to make them readable in a post.

Code: Select all

qcli_storage -d

Code: Select all

qcli_storage -T force=1

Code: Select all

qcli_storage -t force=1
is there any raid syncing or scrubbing going on?
Regards Simon

Qnap Downloads
MyQNap.Org Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following


NAS: TS-673A QuTS hero h5.1.2.2534 • TS-121 4.3.3.2420 • APC Back-UPS ES 700G
Network: VM Hub3: 500/50 • UniFi UDM Pro: 3.2.9 • UniFi Network Controller: 8.0.28
USW-Aggregation: 6.6.61 • US-16-150W: 6.6.61 • 2x USW Mini Flex 2.0.0 • UniFi AC Pro 6.6.62 • UniFi U6-LR 6.6.62
UniFi Protect: 2.11.21/8TB Skyhawk AI • 3x G3 Instants: 4.69.55 • UniFi G3 Flex: 4.69.55 • UniFi G5 Flex: 4.69.55
User avatar
Toxic17
Ask me anything
Posts: 6469
Joined: Tue Jan 25, 2011 11:41 pm
Location: Planet Earth
Contact:

Re: Performance issues using 6-drive RAID on a TS-873

Post by Toxic17 »

I take it your using a new CAT6 or higher cable? if the PC directly connected to the 10GbE port or through a switch?
Regards Simon

Qnap Downloads
MyQNap.Org Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following


NAS: TS-673A QuTS hero h5.1.2.2534 • TS-121 4.3.3.2420 • APC Back-UPS ES 700G
Network: VM Hub3: 500/50 • UniFi UDM Pro: 3.2.9 • UniFi Network Controller: 8.0.28
USW-Aggregation: 6.6.61 • US-16-150W: 6.6.61 • 2x USW Mini Flex 2.0.0 • UniFi AC Pro 6.6.62 • UniFi U6-LR 6.6.62
UniFi Protect: 2.11.21/8TB Skyhawk AI • 3x G3 Instants: 4.69.55 • UniFi G3 Flex: 4.69.55 • UniFi G5 Flex: 4.69.55
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: Performance issues using 6-drive RAID on a TS-873

Post by Trexx »

Make sure you have SMB 3.x enabled, which is not the default setting on the QNAP. Which 10GbE card did you put in your QNAP? Also would need more details about the client machine (HW, 10GbE card, OS, etc.)

Also check that your HDD FW is the same on all drives. I remember seeing a post in the forum recently related to slow performance and it was due to a FW mismatch between the drives.
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
robske_110
New here
Posts: 8
Joined: Fri Sep 06, 2019 12:53 am

Re: Performance issues using 6-drive RAID on a TS-873

Post by robske_110 »

Thank you for your great responses!
Toxic17 wrote: Fri Sep 06, 2019 1:45 am you could run the internal tests as well.

SSH to your NAS.

then type in the following commands, cut and paste the results into a post (use the BBcode quotes/code) to make them readable in a post.

Code: Select all

qcli_storage -d

Code: Select all

qcli_storage -T force=1

Code: Select all

qcli_storage -t force=1
is there any raid syncing or scrubbing going on?
very great that one can run internal tests, here are the results:

Code: Select all

[~] # qcli_storage -d
Enclosure  Port  Sys_Name     Type      Size      Alias          Signature   Partitions  Model  
NAS_HOST   1     /dev/sda     SSD:data  931.51 GB M.2 SSD 1      QNAP FLEX   5           ATA CT1000MX500SSD4 
NAS_HOST   3     /dev/sde     HDD:data  12.73 TB  Disk 1         QNAP FLEX   5           Seagate ST14000NM0018-2H4101
NAS_HOST   4     /dev/sdd     HDD:data  12.73 TB  Disk 2         QNAP FLEX   5           Seagate ST14000NM0018-2H4101
NAS_HOST   5     /dev/sdg     HDD:data  12.73 TB  Disk 3         QNAP FLEX   5           Seagate ST14000NM0018-2H4101
NAS_HOST   6     /dev/sdf     HDD:data  12.73 TB  Disk 4         QNAP FLEX   5           Seagate ST14000NM0018-2H4101
NAS_HOST   7     /dev/sdc     HDD:data  12.73 TB  Disk 5         QNAP FLEX   5           Seagate ST14000NM0018-2H4101
NAS_HOST   8     /dev/sdb     HDD:data  12.73 TB  Disk 6         QNAP FLEX   5           Seagate ST14000NM0018-2H4101
[~] # qcli_storage -T force=1                                                                                                       
fio test command for physical disk: /sbin/fio --filename=test_device --direct=1 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
fio test command for RAID: /sbin/fio --filename=test_device --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
Start testing!
Performance test is finished 100.000%...
Enclosure  Port  Sys_Name      Throughput    RAID        RAID_Type    RAID_Throughput   Pool  
NAS_HOST   1     /dev/sda      537.02 MB/s   /dev/md1    Single       538.50 MB/s       1     
NAS_HOST   3     /dev/sde      258.38 MB/s   /dev/md2    RAID 0       1.02 GB/s         288   
NAS_HOST   4     /dev/sdd      245.01 MB/s   /dev/md2    RAID 0       1.02 GB/s         288   
NAS_HOST   5     /dev/sdg      254.98 MB/s   /dev/md2    RAID 0       1.02 GB/s         288   
NAS_HOST   6     /dev/sdf      244.61 MB/s   /dev/md2    RAID 0       1.02 GB/s         288   
NAS_HOST   7     /dev/sdc      240.62 MB/s   /dev/md2    RAID 0       1.02 GB/s         288   
NAS_HOST   8     /dev/sdb      249.80 MB/s   /dev/md2    RAID 0       1.02 GB/s         288   
[~] # qcli_storage -t force=1
fio test command for LV layer: /sbin/fio --filename=test_device --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
fio test command for File system: /sbin/fio --filename=test_device/qcli_storage --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 --size=128m &>/tmp/qcli_storage.log
Start testing!
Performance test is finished 100.000%...
VolID   VolName             Pool     Mapping_Name            Throughput      Mount_Path                    FS_Throughput
1       DataVol1            1        /dev/mapper/cachedev1   538.30 MB/s     /share/CACHEDEV1_DATA         492.31 MB/s
2       DataVol2            288      /dev/mapper/ce_cachedev21.09 GB/s       /share/CE_CACHEDEV2_DATA      977.00 MB/s
[~] # 
Looks the RAID (DataVol2) is able to do 1.02GB/s according to the first test command, and at least 977MB/s according to the second.

probably didn't clarify that enough, but at the moment I have the 6 drives in a RAID0 until I figure out what slows performance. (but when i tested the RAID5 the syncing was completed)
Toxic17 wrote: Fri Sep 06, 2019 1:54 am I take it your using a new CAT6 or higher cable? if the PC directly connected to the 10GbE port or through a switch?
yes I am using a CAT6A cable, and the Computer is directly connected.
Trexx wrote: Fri Sep 06, 2019 2:05 am Make sure you have SMB 3.x enabled, which is not the default setting on the QNAP. Which 10GbE card did you put in your QNAP? Also would need more details about the client machine (HW, 10GbE card, OS, etc.)

Also check that your HDD FW is the same on all drives. I remember seeing a post in the forum recently related to slow performance and it was due to a FW mismatch between the drives.
I don't know where to enable SMB 3.x, but will figure that out and test it later.
How do I check the HDD FW in the NAS, I couldn't find anything on that yet. (Or do I need to pull the drives and stick them into a computer for that?)
10GbE card is the QXG-10G1T.
The client machine is a MacBook Pro (2016) with macOS with a Thunderbolt OWC 10GbE adapter.
robske_110
New here
Posts: 8
Joined: Fri Sep 06, 2019 12:53 am

Re: Performance issues using 6-drive RAID on a TS-873

Post by robske_110 »

SMB3 was enable (Highest was set to SMB3), and I also tested again after setting lowest to SMB3 too, but I still only get 450 write and 560 read.
Still don't know where to check HDD firmware though.
And I don't think it's the network connection itself, since an iperf3 test between the macbook and the nas says this:

Code: Select all

bash-3.2$ iperf3 --bidir -c 192.168.5.1
Connecting to host 192.168.5.1, port 5201
[  5] local 192.168.5.2 port 56433 connected to 192.168.5.1 port 5201
[ ID][Role] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.15 GBytes  9.86 Gbits/sec                  
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec                  
[  5]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec                  
[  5]   3.00-4.00   sec  1.15 GBytes  9.90 Gbits/sec                  
[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec                  
[  5]   5.00-6.00   sec  1.15 GBytes  9.90 Gbits/sec                  
[  5]   6.00-7.00   sec  1.15 GBytes  9.89 Gbits/sec                  
[  5]   7.00-8.00   sec  1.14 GBytes  9.83 Gbits/sec                  
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec                  
[  5]   9.00-10.00  sec  1.13 GBytes  9.73 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  11.5 GBytes  9.87 Gbits/sec                  sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.87 Gbits/sec                  receiver

iperf Done.
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: Performance issues using 6-drive RAID on a TS-873

Post by Trexx »

Make sure you have packet signing turned off.
https://support.apple.com/en-us/HT205926

Disk FW is under storage/snapshots > disks/vjbod > disk info button


Sent from my iPad using Tapatalk
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
robske_110
New here
Posts: 8
Joined: Fri Sep 06, 2019 12:53 am

Re: Performance issues using 6-drive RAID on a TS-873

Post by robske_110 »

Unfortunately turning packet signing off didn't have any impact.

Firmware matches (all have FN02)
User avatar
MrVideo
Experience counts
Posts: 4742
Joined: Fri May 03, 2013 2:26 pm

Re: Performance issues using 6-drive RAID on a TS-873

Post by MrVideo »

What surprises me is that none of the respondents said anything about using RAID5 with that many drives and drives of that size. You should be using RAID6, otherwise, IMHO, it is a disaster waiting to happen.
QTS MANUALS
Submit QNAP Support Ticket - QNAP Tutorials, FAQs, Downloads, Wiki - Product Support Status - Moogle's QNAP FAQ help V2
Asking a question, include the following
(Thanks to Toxic17)
QNAP md_checker nasreport (release 20210309)
===============================
Model: TS-869L -- RAM: 3G -- FW: QTS 4.1.4 Build 20150522 (used for data storage)
WD60EFRX-68L0BN1(x1)/68MYMN1(x7) Red HDDs -- RAID6: 8x6TB -- Cold spare: 1x6TB
Entware
===============================
Model: TS-451A -- RAM: 2G -- FW: QTS 4.5.2 Build 20210202 (used as a video server)
WL3000GSA6472(x3) White label NAS HDDs -- RAID5: 3x3TB
Entware -- MyKodi 17.3 (default is Kodi 16)
===============================
My 2017 Total Solar Eclipse Photos | My 2019 N. Ireland Game of Thrones tour
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: Performance issues using 6-drive RAID on a TS-873

Post by Trexx »

MrVideo wrote: Fri Sep 06, 2019 4:49 am What surprises me is that none of the respondents said anything about using RAID5 with that many drives and drives of that size. You should be using RAID6, otherwise, IMHO, it is a disaster waiting to happen.
He had stated previously he had tried raid-5/6 and was using raid-0 to test parity overhead impact on his low perf #'s. Didn't sound like was going to be a permanent configuration.
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: Performance issues using 6-drive RAID on a TS-873

Post by Trexx »

robske_110 wrote: Fri Sep 06, 2019 3:23 am Unfortunately turning packet signing off didn't have any impact.

Firmware matches (all have FN02)
Here are a few more things to try. https://community.spiceworks.com/how_to ... rom-mac-os

Which OSX version are you running? I do recall a couple people saying they ended up resolving their slow perf issues by basically wiping their laptops and re-installing OSX on their laptops from scratch.
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
User avatar
Toxic17
Ask me anything
Posts: 6469
Joined: Tue Jan 25, 2011 11:41 pm
Location: Planet Earth
Contact:

Re: Performance issues using 6-drive RAID on a TS-873

Post by Toxic17 »

What 10GbE cards are you using?

I'm sure Bob Zelin will have a fix :)

viewtopic.php?f=11&t=141272&start=15#p671452
Regards Simon

Qnap Downloads
MyQNap.Org Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following


NAS: TS-673A QuTS hero h5.1.2.2534 • TS-121 4.3.3.2420 • APC Back-UPS ES 700G
Network: VM Hub3: 500/50 • UniFi UDM Pro: 3.2.9 • UniFi Network Controller: 8.0.28
USW-Aggregation: 6.6.61 • US-16-150W: 6.6.61 • 2x USW Mini Flex 2.0.0 • UniFi AC Pro 6.6.62 • UniFi U6-LR 6.6.62
UniFi Protect: 2.11.21/8TB Skyhawk AI • 3x G3 Instants: 4.69.55 • UniFi G3 Flex: 4.69.55 • UniFi G5 Flex: 4.69.55
robske_110
New here
Posts: 8
Joined: Fri Sep 06, 2019 12:53 am

Re: Performance issues using 6-drive RAID on a TS-873

Post by robske_110 »

Thank you once again for your endless supply of nice and informative replies, I really didn't expect to get so much help :D
Trexx wrote: Fri Sep 06, 2019 5:05 am
robske_110 wrote: Fri Sep 06, 2019 3:23 am Unfortunately turning packet signing off didn't have any impact.

Firmware matches (all have FN02)
Here are a few more things to try. https://community.spiceworks.com/how_to ... rom-mac-os

Which OSX version are you running? I do recall a couple people saying they ended up resolving their slow perf issues by basically wiping their laptops and re-installing OSX on their laptops from scratch.
I've done the first recommendation from that article, but am unsure how to apply the third one on a qnap NAS (if even necessarry/possible)
Also the second doesn't seem to be an option in either the qnap adapter settings or the macOS ones.

I am running macOS 10.14.6. I'm going to see if I can test with another mac (and also Windows, although I'd like to have full speeds with macOS too) tomorrow.
Toxic17 wrote: Fri Sep 06, 2019 5:12 am What 10GbE cards are you using?

I'm sure Bob Zelin will have a fix :)

viewtopic.php?f=11&t=141272&start=15#p671452
In the NAS: QXG-10G1T
On the client: OWC Thunderbolt 3 10GbE Adapter

And since iperf3 is reporting (almost) full duplex 10Gbit/s, I really think this is a problem with either the NAS or the macOS SMB client and not the actual 10GbE connection.
Bob Zelin
Experience counts
Posts: 1374
Joined: Mon Nov 21, 2016 12:55 am
Location: Orlando, FL.
Contact:

Re: Performance issues using 6-drive RAID on a TS-873

Post by Bob Zelin »

it sounds like you have a Win 10 PC with a thunderbolt 3 port, and you are using the OWC T3 to 10G adaptor. I have no idea about your PC. If I am wrong, and you are using a Mac (you just said about the macOS SMB client) -
make sure to have SMB signing disabled in terminal, and reboot your machine. And enable Jumbo Frames in both System Preferences for your 10G port, and on the QNAP 10G port.
Why are you only using 6 drives in a TVS-873 - put eight drives in there. If you follow my instructions, you will get 800 MB/sec (and probably not more than 500 MB/sec write).
And stop staring at the specs on the QNAP website (2300 MB/sec) - I have no idea of what they are referring to. With a 10G connection, even theoretical bandwidth can never exceed 1250 MB/sec, and in real life, I have never
seen over 1100 MB/sec on ANY 10G client - no matter what. 2300 MB/sec probably means total aggregate bandwidth - and there is no way that a TVS-873 is going to perform like this. There is real world, and there is fantasy.

Bob Zelin
Bob Zelin / Rescue 1, Inc.
http://www.bobzelin.com
User avatar
MrVideo
Experience counts
Posts: 4742
Joined: Fri May 03, 2013 2:26 pm

Re: Performance issues using 6-drive RAID on a TS-873

Post by MrVideo »

Trexx wrote: Fri Sep 06, 2019 5:02 amHe had stated previously he had tried raid-5/6 and was using raid-0 to test parity overhead impact on his low perf #'s.
Not that I saw. Only saw a RAID5 comment in the first post.
QTS MANUALS
Submit QNAP Support Ticket - QNAP Tutorials, FAQs, Downloads, Wiki - Product Support Status - Moogle's QNAP FAQ help V2
Asking a question, include the following
(Thanks to Toxic17)
QNAP md_checker nasreport (release 20210309)
===============================
Model: TS-869L -- RAM: 3G -- FW: QTS 4.1.4 Build 20150522 (used for data storage)
WD60EFRX-68L0BN1(x1)/68MYMN1(x7) Red HDDs -- RAID6: 8x6TB -- Cold spare: 1x6TB
Entware
===============================
Model: TS-451A -- RAM: 2G -- FW: QTS 4.5.2 Build 20210202 (used as a video server)
WL3000GSA6472(x3) White label NAS HDDs -- RAID5: 3x3TB
Entware -- MyKodi 17.3 (default is Kodi 16)
===============================
My 2017 Total Solar Eclipse Photos | My 2019 N. Ireland Game of Thrones tour
Post Reply

Return to “System & Disk Volume Management”