[HOWTO] How to increase raid rebuild speed

Discussion on setting up QNAP NAS products.
Locked
solomonshv
Starting out
Posts: 24
Joined: Thu Feb 02, 2017 4:20 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by solomonshv »

this fix worked on older version of the QNAP OS, but on the newest version, changing these values has literally ZERO effect. Changing resync priority in the volume manager also does nothing. WTF is QNAP thinking?
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R »

solomonshv wrote:this fix worked on older version of the QNAP OS, but on the newest version, changing these values has literally ZERO effect.
In my experience it have made less and less difference and I think that's because the faster CPUs of today are fast enough already on the standard setting to not be a bottleneck. With a fast CPU, the disks instead become the bottleneck and you can't tune your way out of that.

Rebuilds does take time, especially with huge disks. Disk sizes increase dramatically but they don't become more than marginally faster.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
solomonshv
Starting out
Posts: 24
Joined: Thu Feb 02, 2017 4:20 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by solomonshv »

P3R wrote:In my experience it have made less and less difference and I think that's because the faster CPUs of today are fast enough already on the standard setting to not be a bottleneck. With a fast CPU, the disks instead become the bottleneck and you can't tune your way out of that.

Rebuilds does take time, especially with huge disks. Disk sizes increase dramatically but they don't become more than marginally faster.
before QNAP released this garbage OS (4.3), i was getting speeds of over 100MB/s when rebuilding or adding a drive to expand the volume. these are WD Red Pro drives after all, they are very, very fast. now i'm getting 5MB/s. QNAP screwed something up and their support is useless.
skypx wrote:The RAID rebuild speed setting was moved to the storage and snapshots app.
raid speed.jpg
that was the first thing i tried. that setting changes absolutely nothing. rebuild speed stays at 5MB/s no matter what options you select or how many times you reboot.
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R »

solomonshv wrote:before QNAP released this garbage OS (4.3), i was getting speeds of over 100MB/s when rebuilding or adding a drive to expand the volume.
...//...
now i'm getting 5MB/s.
If a 20-times reduction in rebuild speed had been introduced in QTS 4.3, I think we would have heard much more about that from many other participants here in the forum.

Maybe there's some issue specific to your installation causing the very low rate of progress?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
jsteng
Getting the hang of things
Posts: 98
Joined: Thu Mar 29, 2018 7:48 am

Re: [HOWTO] How to increase raid rebuild speed

Post by jsteng »

Hi gandalf,

I just bought the same TS431P, populated it with 3 x 3TB of HDDs in RAID5 and an SSD.
Now, I am experiencing your pain as well.
I am interested in how you managed to speed things up.

Right now with all this frustration, I am not a happy QNAP user. :(
User avatar
dolbyman
Guru
Posts: 35022
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: [HOWTO] How to increase raid rebuild speed

Post by dolbyman »

what are you doing with an ssd in a 431?

what are you trying to speed up?
321liftoff
Know my way around
Posts: 214
Joined: Sat Oct 22, 2011 6:54 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by 321liftoff »

Using a TS-453Be with 4.3.4.0597 build 20180518. I'm currently migrating from 3x2TB RAID5 to 4x2TB RAID6, WD Red. Priority is "Resync First". CPU hovering around 7-10%. Running about 6MB/s, which is way below HDD speeds and CPU isn't pegged. Below is an output:

Code: Select all

[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] 
md1 : active raid6 sdb3[3] sdd3[0] sda3[2] sdc3[1]
      3887119232 blocks super 1.0 level 6, 64k chunk, algorithm 18 [4/3] [UUU_]
      [=>...................]  reshape =  6.2% (120815616/1943559616) finish=4843.2min speed=6272K/sec
      
md322 : active raid1 sdb5[3](S) sda5[2](S) sdd5[1] sdc5[0]
      7235136 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md256 : active raid1 sdb2[3](S) sda2[2](S) sdd2[1] sdc2[0]
      530112 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md13 : active raid1 sdb4[34] sdd4[0] sda4[33] sdc4[32]
      458880 blocks super 1.0 [32/4] [UUUU____________________________]
      bitmap: 1/1 pages [4KB], 65536KB chunk

md9 : active raid1 sdb1[34] sdd1[0] sda1[33] sdc1[32]
      530048 blocks super 1.0 [32/4] [UUUU____________________________]
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>
[~] #
Seems slower than it should be...
User avatar
lithiumus
Starting out
Posts: 45
Joined: Tue Sep 27, 2016 11:22 am

Re: [HOWTO] How to increase raid rebuild speed

Post by lithiumus »

I am also rebuilding RAID5 to RAID6 and the reshaping is hovering at 3.5-3.7MB/s. I know for a fact that the last time I added a hard disk and expanded the RAID5 with reshaping, I was able to set the Min and was getting between 50-80MB/s but the new firmware has made changes and reshaping is stuck at 3.5-3.7MB/s.

I've set the priority to Fastest / High but even when I changed the setting to Lowest, the speed dropped to about 3.1-3.2MB/s essentially making no difference at all. I've opened up a ticket with Support. Really do not want to wait 24-30 days in a degraded mode, not to mention if I expand the RAID I will be faced with another 24-30 days in degraded mode. That's just not right...
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: [HOWTO] How to increase raid rebuild speed

Post by Trexx »

RAID5 expansion vs. RAID-6 migration are significantly different processes. RAID-6 involves both restriping and the calculation of 2 sets of parity blocks. Speed for these calculation is dependent on the QNAP model & CPU's capabilities to leverage CPU HW capabilities to help with the parity calculations.

Also the time to complete listed in SSH is not accurate as throughput rates will change depending where on the drives you are.
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
User avatar
lithiumus
Starting out
Posts: 45
Joined: Tue Sep 27, 2016 11:22 am

Re: [HOWTO] How to increase raid rebuild speed

Post by lithiumus »

Trexx wrote:RAID5 expansion vs. RAID-6 migration are significantly different processes. RAID-6 involves both restriping and the calculation of 2 sets of parity blocks. Speed for these calculation is dependent on the QNAP model & CPU's capabilities to leverage CPU HW capabilities to help with the parity calculations.

Also the time to complete listed in SSH is not accurate as throughput rates will change depending where on the drives you are.
Well, that's actually good news. I don't mind the longer wait to migrate to RAID-6 but definitely not when I expand my RAID-6 at a later date. I've got the ARM TS-1231XU with 8GB of memory. At the moment, CPU fluctuating between 2-12% most functionality disabled. I tried to increase the strip_cache and the box hung so that was a bad idea... it's set to 256 and I tried 16384... I'll just wait it out.
321liftoff
Know my way around
Posts: 214
Joined: Sat Oct 22, 2011 6:54 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by 321liftoff »

I definitely noticed that the QNAP was not being fully utilized during the reshaping. I still think something is getting throttled unnecessarily.
User avatar
lithiumus
Starting out
Posts: 45
Joined: Tue Sep 27, 2016 11:22 am

Re: [HOWTO] How to increase raid rebuild speed

Post by lithiumus »

lithiumus wrote:
Trexx wrote:RAID5 expansion vs. RAID-6 migration are significantly different processes. RAID-6 involves both restriping and the calculation of 2 sets of parity blocks. Speed for these calculation is dependent on the QNAP model & CPU's capabilities to leverage CPU HW capabilities to help with the parity calculations.

Also the time to complete listed in SSH is not accurate as throughput rates will change depending where on the drives you are.
Well, that's actually good news. I don't mind the longer wait to migrate to RAID-6 but definitely not when I expand my RAID-6 at a later date. I've got the ARM TS-1231XU with 8GB of memory. At the moment, CPU fluctuating between 2-12% most functionality disabled. I tried to increase the strip_cache and the box hung so that was a bad idea... it's set to 256 and I tried 16384... I'll just wait it out.
Well, 11 days since my last post and I'm at 43.3% still at about 3.5Mb/s. I reached out to Qnap support and we checked all the same things and they said it's going as fast as it can. The initial numbers were saying it was going to take 22-26 days and based on the SSH mdstat, it says I've got about 14 more days left... let's see if there is any change in speed after the 50% mark, but I'm not hopeful...

I do have 8 x 8TB drives going from a 7 x 8TB RAID5 to a 8 x 8TB RAID6 so yes, I know it's a lot of data and the size is large... just that 3.5mb/s sounds crazy...
arnaldo
New here
Posts: 8
Joined: Thu Sep 06, 2018 9:54 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by arnaldo »

Running firmware 4.3.4 I was getting a 36hrs rebuild time for RAID5 array (4x4TB on a TS-469Pro). After I changed it to to "Rsync First" (high priority) the build time was down to 13hrs. It's the second drive that goes bad in a month (WD SE running for almost 5 years).

I would love to have a finer grain control and not just "low, med, high" (I don't mind the old style tweak), but for most GUI oriented folks it's nice to be able to have at lease some control.

Also, my guess is that, at least on my box, I could go a little faster as there is no resource contention (CPU is at 15-25%, I/O is not bad and there is still plenty of free memory).

Maybe QNAP could add an "advanced" mode, where one could really fine tune it.
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R »

Can't a moderator (maybe even Don?) please close this great thread before it celebrate 10 years? Everything important had been said long ago and it's just getting worse with every post. It's already far too long so nobody bothers to read it and additional posts are only adding confusion. People don't understand the differences of different tasks like Resync (rebuild) and Reshaping. It's also a mix of cat1 and cat2 (that now have the Resync priority in the gui). It's better that those that want to report their progress or ask questions that haven't already been answered here, do it in new threads.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
User avatar
Toxic17
Ask me anything
Posts: 6469
Joined: Tue Jan 25, 2011 11:41 pm
Location: Planet Earth
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by Toxic17 »

P3R wrote:Can't a moderator (maybe even Don?) please close this great thread before it celebrate 10 years?
Thread Locked
Regards Simon

Qnap Downloads
MyQNap.Org Repository
Submit a ticket • QNAP Helpdesk
QNAP Tutorials, User Manuals, FAQs, Downloads, Wiki
When you ask a question, please include the following


NAS: TS-673A QuTS hero h5.1.2.2534 • TS-121 4.3.3.2420 • APC Back-UPS ES 700G
Network: VM Hub3: 500/50 • UniFi UDM Pro: 3.2.9 • UniFi Network Controller: 8.0.28
USW-Aggregation: 6.6.61 • US-16-150W: 6.6.61 • 2x USW Mini Flex 2.0.0 • UniFi AC Pro 6.6.62 • UniFi U6-LR 6.6.62
UniFi Protect: 2.11.21/8TB Skyhawk AI • 3x G3 Instants: 4.69.55 • UniFi G3 Flex: 4.69.55 • UniFi G5 Flex: 4.69.55
Locked

Return to “Turbo Station Installation & Setup”