[HOWTO] How to increase raid rebuild speed

Discussion on setting up QNAP NAS products.
Locked
solomonshv
Starting out
Posts: 21
Joined: Thu Feb 02, 2017 4:20 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by solomonshv » Mon Mar 12, 2018 3:47 pm

this fix worked on older version of the QNAP OS, but on the newest version, changing these values has literally ZERO effect. Changing resync priority in the volume manager also does nothing. WTF is QNAP thinking?

P3R
Guru
Posts: 10899
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R » Mon Mar 12, 2018 4:55 pm

solomonshv wrote:this fix worked on older version of the QNAP OS, but on the newest version, changing these values has literally ZERO effect.
In my experience it have made less and less difference and I think that's because the faster CPUs of today are fast enough already on the standard setting to not be a bottleneck. With a fast CPU, the disks instead become the bottleneck and you can't tune your way out of that.

Rebuilds does take time, especially with huge disks. Disk sizes increase dramatically but they don't become more than marginally faster.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

User avatar
skypx
Easy as a breeze
Posts: 250
Joined: Sat Aug 15, 2009 4:44 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by skypx » Tue Mar 13, 2018 6:27 am

solomonshv wrote:this fix worked on older version of the QNAP OS, but on the newest version, changing these values has literally ZERO effect. Changing resync priority in the volume manager also does nothing. WTF is QNAP thinking?


The RAID rebuild speed setting was moved to the storage and snapshots app.

raid speed.jpg
You do not have the required permissions to view the files attached to this post.
TVS-871 i7 16GB Ram
Firmware 4.3.6.0805 Build 20181228
8x HGST HUH721212ALN600 LEGNT3D0 RAID 5
QNAP Network LAN-10G2T-X550T2
Netgear XS708Ev2

solomonshv
Starting out
Posts: 21
Joined: Thu Feb 02, 2017 4:20 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by solomonshv » Sun Mar 18, 2018 5:42 am

P3R wrote:In my experience it have made less and less difference and I think that's because the faster CPUs of today are fast enough already on the standard setting to not be a bottleneck. With a fast CPU, the disks instead become the bottleneck and you can't tune your way out of that.

Rebuilds does take time, especially with huge disks. Disk sizes increase dramatically but they don't become more than marginally faster.


before QNAP released this garbage OS (4.3), i was getting speeds of over 100MB/s when rebuilding or adding a drive to expand the volume. these are WD Red Pro drives after all, they are very, very fast. now i'm getting 5MB/s. QNAP screwed something up and their support is useless.

skypx wrote:The RAID rebuild speed setting was moved to the storage and snapshots app.
raid speed.jpg


that was the first thing i tried. that setting changes absolutely nothing. rebuild speed stays at 5MB/s no matter what options you select or how many times you reboot.

P3R
Guru
Posts: 10899
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R » Sun Mar 18, 2018 7:08 am

solomonshv wrote:before QNAP released this garbage OS (4.3), i was getting speeds of over 100MB/s when rebuilding or adding a drive to expand the volume.
...//...
now i'm getting 5MB/s.
If a 20-times reduction in rebuild speed had been introduced in QTS 4.3, I think we would have heard much more about that from many other participants here in the forum.

Maybe there's some issue specific to your installation causing the very low rate of progress?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

User avatar
skypx
Easy as a breeze
Posts: 250
Joined: Sat Aug 15, 2009 4:44 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by skypx » Sun Mar 18, 2018 7:29 am

solomonshv wrote:that was the first thing i tried. that setting changes absolutely nothing. rebuild speed stays at 5MB/s no matter what options you select or how many times you reboot.


Maybe you should open a ticket with QNAP since that's not the expected behaviour.
TVS-871 i7 16GB Ram
Firmware 4.3.6.0805 Build 20181228
8x HGST HUH721212ALN600 LEGNT3D0 RAID 5
QNAP Network LAN-10G2T-X550T2
Netgear XS708Ev2

jsteng
Getting the hang of things
Posts: 52
Joined: Thu Mar 29, 2018 7:48 am

Re: [HOWTO] How to increase raid rebuild speed

Post by jsteng » Thu Mar 29, 2018 8:20 am

Hi gandalf,

I just bought the same TS431P, populated it with 3 x 3TB of HDDs in RAID5 and an SSD.
Now, I am experiencing your pain as well.
I am interested in how you managed to speed things up.

Right now with all this frustration, I am not a happy QNAP user. :(

dolbyman
Guru
Posts: 14230
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: [HOWTO] How to increase raid rebuild speed

Post by dolbyman » Thu Mar 29, 2018 9:37 am

what are you doing with an ssd in a 431?

what are you trying to speed up?

321liftoff
Know my way around
Posts: 214
Joined: Sat Oct 22, 2011 6:54 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by 321liftoff » Sat Jun 23, 2018 5:28 pm

Using a TS-453Be with 4.3.4.0597 build 20180518. I'm currently migrating from 3x2TB RAID5 to 4x2TB RAID6, WD Red. Priority is "Resync First". CPU hovering around 7-10%. Running about 6MB/s, which is way below HDD speeds and CPU isn't pegged. Below is an output:

Code: Select all

[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md1 : active raid6 sdb3[3] sdd3[0] sda3[2] sdc3[1]
      3887119232 blocks super 1.0 level 6, 64k chunk, algorithm 18 [4/3] [UUU_]
      [=>...................]  reshape =  6.2% (120815616/1943559616) finish=4843.2min speed=6272K/sec
     
md322 : active raid1 sdb5[3](S) sda5[2](S) sdd5[1] sdc5[0]
      7235136 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md256 : active raid1 sdb2[3](S) sda2[2](S) sdd2[1] sdc2[0]
      530112 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md13 : active raid1 sdb4[34] sdd4[0] sda4[33] sdc4[32]
      458880 blocks super 1.0 [32/4] [UUUU____________________________]
      bitmap: 1/1 pages [4KB], 65536KB chunk

md9 : active raid1 sdb1[34] sdd1[0] sda1[33] sdc1[32]
      530048 blocks super 1.0 [32/4] [UUUU____________________________]
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>
[~] #

Seems slower than it should be...

User avatar
lithiumus
Starting out
Posts: 36
Joined: Tue Sep 27, 2016 11:22 am

Re: [HOWTO] How to increase raid rebuild speed

Post by lithiumus » Wed Aug 22, 2018 2:24 am

I am also rebuilding RAID5 to RAID6 and the reshaping is hovering at 3.5-3.7MB/s. I know for a fact that the last time I added a hard disk and expanded the RAID5 with reshaping, I was able to set the Min and was getting between 50-80MB/s but the new firmware has made changes and reshaping is stuck at 3.5-3.7MB/s.

I've set the priority to Fastest / High but even when I changed the setting to Lowest, the speed dropped to about 3.1-3.2MB/s essentially making no difference at all. I've opened up a ticket with Support. Really do not want to wait 24-30 days in a degraded mode, not to mention if I expand the RAID I will be faced with another 24-30 days in degraded mode. That's just not right...

User avatar
Trexx
Experience counts
Posts: 4903
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by Trexx » Wed Aug 22, 2018 3:26 am

RAID5 expansion vs. RAID-6 migration are significantly different processes. RAID-6 involves both restriping and the calculation of 2 sets of parity blocks. Speed for these calculation is dependent on the QNAP model & CPU's capabilities to leverage CPU HW capabilities to help with the parity calculations.

Also the time to complete listed in SSH is not accurate as throughput rates will change depending where on the drives you are.
Paul

Model: TS-877-1600 FW: 4.4.1.x
QTier (HDD): [RAID-5] 6 x 3TB HGST DeskStar NAS QTier (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
(SSD): [RAID-1] 2 x 500GB Evo 860
RAM: Kingston HyperX Fury 64GB DDR4-2666
GPU: EVGA GTX 1060, ACX 2.0(1 Fan), 6GB
UPS: CyberPower AVR1350 Ext. Backup: USB 3.0 Seagate 5TB
Media Boxes: Nvidia ShieldTV Pro, AppleTV 4k, Various Roku's

Model: TVS-673 32GB FW: 4.4.1.x Test/Backup Box
-----------------------------------------------------------------------------------------------------------------------------------------
NAS RAID Rebuild Times | Live QTS Videos | | QNAP NAS Guide | Information needed when you ask for HELP | QNAP Links, Tutorials, etc.
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq

User avatar
lithiumus
Starting out
Posts: 36
Joined: Tue Sep 27, 2016 11:22 am

Re: [HOWTO] How to increase raid rebuild speed

Post by lithiumus » Wed Aug 22, 2018 3:59 am

Trexx wrote:RAID5 expansion vs. RAID-6 migration are significantly different processes. RAID-6 involves both restriping and the calculation of 2 sets of parity blocks. Speed for these calculation is dependent on the QNAP model & CPU's capabilities to leverage CPU HW capabilities to help with the parity calculations.

Also the time to complete listed in SSH is not accurate as throughput rates will change depending where on the drives you are.


Well, that's actually good news. I don't mind the longer wait to migrate to RAID-6 but definitely not when I expand my RAID-6 at a later date. I've got the ARM TS-1231XU with 8GB of memory. At the moment, CPU fluctuating between 2-12% most functionality disabled. I tried to increase the strip_cache and the box hung so that was a bad idea... it's set to 256 and I tried 16384... I'll just wait it out.

321liftoff
Know my way around
Posts: 214
Joined: Sat Oct 22, 2011 6:54 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by 321liftoff » Wed Aug 22, 2018 4:20 am

I definitely noticed that the QNAP was not being fully utilized during the reshaping. I still think something is getting throttled unnecessarily.

User avatar
lithiumus
Starting out
Posts: 36
Joined: Tue Sep 27, 2016 11:22 am

Re: [HOWTO] How to increase raid rebuild speed

Post by lithiumus » Sun Sep 02, 2018 10:01 am

lithiumus wrote:
Trexx wrote:RAID5 expansion vs. RAID-6 migration are significantly different processes. RAID-6 involves both restriping and the calculation of 2 sets of parity blocks. Speed for these calculation is dependent on the QNAP model & CPU's capabilities to leverage CPU HW capabilities to help with the parity calculations.

Also the time to complete listed in SSH is not accurate as throughput rates will change depending where on the drives you are.


Well, that's actually good news. I don't mind the longer wait to migrate to RAID-6 but definitely not when I expand my RAID-6 at a later date. I've got the ARM TS-1231XU with 8GB of memory. At the moment, CPU fluctuating between 2-12% most functionality disabled. I tried to increase the strip_cache and the box hung so that was a bad idea... it's set to 256 and I tried 16384... I'll just wait it out.


Well, 11 days since my last post and I'm at 43.3% still at about 3.5Mb/s. I reached out to Qnap support and we checked all the same things and they said it's going as fast as it can. The initial numbers were saying it was going to take 22-26 days and based on the SSH mdstat, it says I've got about 14 more days left... let's see if there is any change in speed after the 50% mark, but I'm not hopeful...

I do have 8 x 8TB drives going from a 7 x 8TB RAID5 to a 8 x 8TB RAID6 so yes, I know it's a lot of data and the size is large... just that 3.5mb/s sounds crazy...

arnaldo
New here
Posts: 6
Joined: Thu Sep 06, 2018 9:54 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by arnaldo » Sat Sep 08, 2018 1:56 am

Running firmware 4.3.4 I was getting a 36hrs rebuild time for RAID5 array (4x4TB on a TS-469Pro). After I changed it to to "Rsync First" (high priority) the build time was down to 13hrs. It's the second drive that goes bad in a month (WD SE running for almost 5 years).

I would love to have a finer grain control and not just "low, med, high" (I don't mind the old style tweak), but for most GUI oriented folks it's nice to be able to have at lease some control.

Also, my guess is that, at least on my box, I could go a little faster as there is no resource contention (CPU is at 15-25%, I/O is not bad and there is still plenty of free memory).

Maybe QNAP could add an "advanced" mode, where one could really fine tune it.

Locked

Return to “Turbo Station Installation & Setup”