Slow internal write/read performance with thick volume in RAID 5 storage pool

Questions about SNMP, Power, System, Logs, disk, & RAID.
Post Reply
somy1982
Easy as a breeze
Posts: 356
Joined: Fri Apr 08, 2016 6:42 pm

Slow internal write/read performance with thick volume in RAID 5 storage pool

Post by somy1982 » Mon Sep 16, 2019 8:16 pm

Hi,
Recently I noticed read/write performance decrease, initially I thought it was SMB or network problem, but now it looks like some file system issues. I have bad performance on all my thick volumes (unencrypted) created in a RAID 5 storage pool with 4X4TB HDDs (I know I should avoid RAID 5).
I tried the following to test the speed, all uncessary services/apps were disabled when I test:
1) qcli_storage -T for disk speed, which looks fine:

Enclosure Port Sys_Name Throughput RAID RAID_Type RAID_Throughput Pool

NAS_HOST 1 /dev/sdd 139.10 MB/s /dev/md1 RAID 5 226.29 MB/s 1

NAS_HOST 2 /dev/sdc 139.51 MB/s /dev/md1 RAID 5 226.29 MB/s 1

NAS_HOST 3 /dev/sdb 121.89 MB/s /dev/md1 RAID 5 226.29 MB/s 1

NAS_HOST 4 /dev/sda 112.86 MB/s /dev/md1 RAID 5 226.29 MB/s 1


2) qcli_storage -t for file system speed:
VolID VolName Pool Mapping_Name Throughput Mount_Path FS_Throughput

1 SYSTEM 1 /dev/mapper/cachedev1 65.11 MB/s /share/CACHEDEV1_DATA 57.04 MB/s

3 BACKUP 1 /dev/mapper/cachedev3 50.26 MB/s /share/CACHEDEV3_DATA 57.74 MB/s

I have tried the following without help:
1) Scanned HDDs, no problem
2) Run through a RAID scrubbing
3) Checked file systems of the volumes


What could go wrong? Any suggestion is appreciated!

dolbyman
Guru
Posts: 15267
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: Slow internal write/read performance with thick volume in RAID 5 storage pool

Post by dolbyman » Mon Sep 16, 2019 9:03 pm

what nas model?
what firmware and build?
why would you want to avoid raid5?
what is the cpu usage during benchmarks?

somy1982
Easy as a breeze
Posts: 356
Joined: Fri Apr 08, 2016 6:42 pm

Re: Slow internal write/read performance with thick volume in RAID 5 storage pool

Post by somy1982 » Mon Sep 16, 2019 9:12 pm

dolbyman wrote:
Mon Sep 16, 2019 9:03 pm
what nas model?
what firmware and build?
why would you want to avoid raid5?
what is the cpu usage during benchmarks?
Sorry didn't provide all info.
The NAS is TS-453mini based on X86 J1900 CPU, and running with 16G memory.
The firmware is 4.3.6 the latest official one.
I read the problem with RAID 5/URE, so if I could do it again I would avoid RAID 5 (nothing related to the problem).
CPU usage is between 10% -30% during test.

dolbyman
Guru
Posts: 15267
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: Slow internal write/read performance with thick volume in RAID 5 storage pool

Post by dolbyman » Mon Sep 16, 2019 9:23 pm

please state the build as there was two recent recalled builds

so you would go with raid6 then ? (curiosity)

somy1982
Easy as a breeze
Posts: 356
Joined: Fri Apr 08, 2016 6:42 pm

Re: Slow internal write/read performance with thick volume in RAID 5 storage pool

Post by somy1982 » Mon Sep 16, 2019 9:29 pm

dolbyman wrote:
Mon Sep 16, 2019 9:23 pm
please state the build as there was two recent recalled builds

so you would go with raid6 then ? (curiosity)
It is 4.3.6.1050
I would simply go for RAID 1 + single SSD (not cache)

dolbyman
Guru
Posts: 15267
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: Slow internal write/read performance with thick volume in RAID 5 storage pool

Post by dolbyman » Mon Sep 16, 2019 10:39 pm

that one was recalled .. best to downgrade

viewtopic.php?f=142&t=150355

not sure where all the fear for raid levels 5 and 6 come from

I run RAID6 for dual parity with comprehensive backups...all good

User avatar
Toxic17
Ask me anything
Posts: 5231
Joined: Tue Jan 25, 2011 11:41 pm
Location: Planet Earth
Contact:

Re: Slow internal write/read performance with thick volume in RAID 5 storage pool

Post by Toxic17 » Mon Sep 16, 2019 11:15 pm

what Model of Hard disk are you running?

Code: Select all

qcli_storage -d
Regards Simon

QTS 4.x User Guidex

QNAP Club Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following


NAS: TS-473-32GB QM2-2P QXG-10G1T 4.4.1.1086 • TVS-463-16GB 4.4.1.1086 QM2-2S10G1TB • TS-459 Pro 2GB 4.2.6 • TS-121 4.3.3.1051 • APC Back-UPS ES 700G •
QPKG's: TwonkyServer 8.51 • Apache73 v2441.7310 • QSonarr 3.0.3.645 • QNBZGet 21.0 • phpMyAdmin 4.9.0.1 • Qmono 5.20.1.19 • McAfee 3.0.1 • Lychee 3.2.16 • HBS 3.0.191016 • LEgo v3.1.0
Network: VM Hub 3.0 <500/35> • UniFi USG Pro 4 • UniFi USW-16-150W • UniFi USW-8-60W • UniFi CloudKey Gen2+• UniFi G3-Flex • UAP AC Pro • UAP AC Lite • SLM2008 • Dell 7050 MFF •

somy1982
Easy as a breeze
Posts: 356
Joined: Fri Apr 08, 2016 6:42 pm

Re: Slow internal write/read performance with thick volume in RAID 5 storage pool

Post by somy1982 » Mon Sep 16, 2019 11:54 pm

Toxic17 wrote:
Mon Sep 16, 2019 11:15 pm
what Model of Hard disk are you running?

Code: Select all

qcli_storage -d
I have a mix of HDDs from different manufactures:

Enclosure Port Sys_Name Type Size Alias Signature Partitions Model
NAS_HOST 1 /dev/sdd HDD:data 3.64 TB -- QNAP FLEX 5 WDC WD4001FFSX-68JNUN0
NAS_HOST 2 /dev/sdc HDD:data 3.64 TB -- QNAP FLEX 5 Seagate ST4000VN008-2DR166
NAS_HOST 3 /dev/sdb HDD:data 3.64 TB -- QNAP FLEX 5 HGST HDN724040ALE640
NAS_HOST 4 /dev/sda HDD:data 3.64 TB -- QNAP FLEX 5 HGST HDN724040ALE640

Recall the performance was Ok initially.

somy1982
Easy as a breeze
Posts: 356
Joined: Fri Apr 08, 2016 6:42 pm

Re: Slow internal write/read performance with thick volume in RAID 5 storage pool

Post by somy1982 » Mon Sep 16, 2019 11:55 pm

dolbyman wrote:
Mon Sep 16, 2019 10:39 pm
that one was recalled .. best to downgrade

viewtopic.php?f=142&t=150355

not sure where all the fear for raid levels 5 and 6 come from

I run RAID6 for dual parity with comprehensive backups...all good
Looks like it's pulled due to SMB issues and the performance problem I have now was there for a few months.
I will downgrade anyways to be sure.

somy1982
Easy as a breeze
Posts: 356
Joined: Fri Apr 08, 2016 6:42 pm

Re: Slow internal write/read performance with thick volume in RAID 5 storage pool

Post by somy1982 » Tue Sep 17, 2019 12:53 am

Downgraded to 0933 same problem :(

User avatar
Toxic17
Ask me anything
Posts: 5231
Joined: Tue Jan 25, 2011 11:41 pm
Location: Planet Earth
Contact:

Re: Slow internal write/read performance with thick volume in RAID 5 storage pool

Post by Toxic17 » Tue Sep 17, 2019 1:41 am

the 3 drive manufactured models have different spindle speeds.

WD 5400 RPM
Seagate: 5900 RPM
HGST: 7200 RPM.

this is not an ideal setup. whether this has a knock on affect in disk access I dont know, but I would guess its not helping matters when disk data is being accessed across multiple speed drives in one RAID5 array.

As to you claims of the performance being "Ok" initially, was this months or years ago? and what speed if any was "ok"? was their much data on the drives then?

you might get better performance putting the two HGST drives in RAID1 since they are the same spec drive. the other two are less of a difference in speed and you might get away with another RAID1, but putting them all in the same Raid Array will in my eyes cause throughput issues.
Regards Simon

QTS 4.x User Guidex

QNAP Club Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following


NAS: TS-473-32GB QM2-2P QXG-10G1T 4.4.1.1086 • TVS-463-16GB 4.4.1.1086 QM2-2S10G1TB • TS-459 Pro 2GB 4.2.6 • TS-121 4.3.3.1051 • APC Back-UPS ES 700G •
QPKG's: TwonkyServer 8.51 • Apache73 v2441.7310 • QSonarr 3.0.3.645 • QNBZGet 21.0 • phpMyAdmin 4.9.0.1 • Qmono 5.20.1.19 • McAfee 3.0.1 • Lychee 3.2.16 • HBS 3.0.191016 • LEgo v3.1.0
Network: VM Hub 3.0 <500/35> • UniFi USG Pro 4 • UniFi USW-16-150W • UniFi USW-8-60W • UniFi CloudKey Gen2+• UniFi G3-Flex • UAP AC Pro • UAP AC Lite • SLM2008 • Dell 7050 MFF •

somy1982
Easy as a breeze
Posts: 356
Joined: Fri Apr 08, 2016 6:42 pm

Re: Slow internal write/read performance with thick volume in RAID 5 storage pool

Post by somy1982 » Tue Sep 17, 2019 2:53 am

Toxic17 wrote:
Tue Sep 17, 2019 1:41 am
the 3 drive manufactured models have different spindle speeds.

WD 5400 RPM
Seagate: 5900 RPM
HGST: 7200 RPM.

this is not an ideal setup. whether this has a knock on affect in disk access I dont know, but I would guess its not helping matters when disk data is being accessed across multiple speed drives in one RAID5 array.

As to you claims of the performance being "Ok" initially, was this months or years ago? and what speed if any was "ok"? was their much data on the drives then?

you might get better performance putting the two HGST drives in RAID1 since they are the same spec drive. the other two are less of a difference in speed and you might get away with another RAID1, but putting them all in the same Raid Array will in my eyes cause throughput issues.
True, two of them are 5400RPM and the other two are 7200RPM. Maybe it's time to upgrade......

Post Reply

Return to “System & Disk Volume Management”