Page 13 of 14

Re: [HOWTO] How to increase raid rebuild speed

Posted: Mon Mar 12, 2018 3:47 pm
by solomonshv
this fix worked on older version of the QNAP OS, but on the newest version, changing these values has literally ZERO effect. Changing resync priority in the volume manager also does nothing. WTF is QNAP thinking?

Re: [HOWTO] How to increase raid rebuild speed

Posted: Mon Mar 12, 2018 4:55 pm
by P3R
solomonshv wrote:this fix worked on older version of the QNAP OS, but on the newest version, changing these values has literally ZERO effect.
In my experience it have made less and less difference and I think that's because the faster CPUs of today are fast enough already on the standard setting to not be a bottleneck. With a fast CPU, the disks instead become the bottleneck and you can't tune your way out of that.

Rebuilds does take time, especially with huge disks. Disk sizes increase dramatically but they don't become more than marginally faster.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Tue Mar 13, 2018 6:27 am
by skypx
solomonshv wrote:this fix worked on older version of the QNAP OS, but on the newest version, changing these values has literally ZERO effect. Changing resync priority in the volume manager also does nothing. WTF is QNAP thinking?


The RAID rebuild speed setting was moved to the storage and snapshots app.

raid speed.jpg

Re: [HOWTO] How to increase raid rebuild speed

Posted: Sun Mar 18, 2018 5:42 am
by solomonshv
P3R wrote:In my experience it have made less and less difference and I think that's because the faster CPUs of today are fast enough already on the standard setting to not be a bottleneck. With a fast CPU, the disks instead become the bottleneck and you can't tune your way out of that.

Rebuilds does take time, especially with huge disks. Disk sizes increase dramatically but they don't become more than marginally faster.


before QNAP released this garbage OS (4.3), i was getting speeds of over 100MB/s when rebuilding or adding a drive to expand the volume. these are WD Red Pro drives after all, they are very, very fast. now i'm getting 5MB/s. QNAP screwed something up and their support is useless.

skypx wrote:The RAID rebuild speed setting was moved to the storage and snapshots app.
raid speed.jpg


that was the first thing i tried. that setting changes absolutely nothing. rebuild speed stays at 5MB/s no matter what options you select or how many times you reboot.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Sun Mar 18, 2018 7:08 am
by P3R
solomonshv wrote:before QNAP released this garbage OS (4.3), i was getting speeds of over 100MB/s when rebuilding or adding a drive to expand the volume.
...//...
now i'm getting 5MB/s.
If a 20-times reduction in rebuild speed had been introduced in QTS 4.3, I think we would have heard much more about that from many other participants here in the forum.

Maybe there's some issue specific to your installation causing the very low rate of progress?

Re: [HOWTO] How to increase raid rebuild speed

Posted: Sun Mar 18, 2018 7:29 am
by skypx
solomonshv wrote:that was the first thing i tried. that setting changes absolutely nothing. rebuild speed stays at 5MB/s no matter what options you select or how many times you reboot.


Maybe you should open a ticket with QNAP since that's not the expected behaviour.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Thu Mar 29, 2018 8:20 am
by jsteng
Hi gandalf,

I just bought the same TS431P, populated it with 3 x 3TB of HDDs in RAID5 and an SSD.
Now, I am experiencing your pain as well.
I am interested in how you managed to speed things up.

Right now with all this frustration, I am not a happy QNAP user. :(

Re: [HOWTO] How to increase raid rebuild speed

Posted: Thu Mar 29, 2018 9:37 am
by dolbyman
what are you doing with an ssd in a 431?

what are you trying to speed up?

Re: [HOWTO] How to increase raid rebuild speed

Posted: Sat Jun 23, 2018 5:28 pm
by 321liftoff
Using a TS-453Be with 4.3.4.0597 build 20180518. I'm currently migrating from 3x2TB RAID5 to 4x2TB RAID6, WD Red. Priority is "Resync First". CPU hovering around 7-10%. Running about 6MB/s, which is way below HDD speeds and CPU isn't pegged. Below is an output:

Code: Select all

[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md1 : active raid6 sdb3[3] sdd3[0] sda3[2] sdc3[1]
      3887119232 blocks super 1.0 level 6, 64k chunk, algorithm 18 [4/3] [UUU_]
      [=>...................]  reshape =  6.2% (120815616/1943559616) finish=4843.2min speed=6272K/sec
     
md322 : active raid1 sdb5[3](S) sda5[2](S) sdd5[1] sdc5[0]
      7235136 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md256 : active raid1 sdb2[3](S) sda2[2](S) sdd2[1] sdc2[0]
      530112 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md13 : active raid1 sdb4[34] sdd4[0] sda4[33] sdc4[32]
      458880 blocks super 1.0 [32/4] [UUUU____________________________]
      bitmap: 1/1 pages [4KB], 65536KB chunk

md9 : active raid1 sdb1[34] sdd1[0] sda1[33] sdc1[32]
      530048 blocks super 1.0 [32/4] [UUUU____________________________]
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>
[~] #

Seems slower than it should be...

Re: [HOWTO] How to increase raid rebuild speed

Posted: Wed Aug 22, 2018 2:24 am
by lithiumus
I am also rebuilding RAID5 to RAID6 and the reshaping is hovering at 3.5-3.7MB/s. I know for a fact that the last time I added a hard disk and expanded the RAID5 with reshaping, I was able to set the Min and was getting between 50-80MB/s but the new firmware has made changes and reshaping is stuck at 3.5-3.7MB/s.

I've set the priority to Fastest / High but even when I changed the setting to Lowest, the speed dropped to about 3.1-3.2MB/s essentially making no difference at all. I've opened up a ticket with Support. Really do not want to wait 24-30 days in a degraded mode, not to mention if I expand the RAID I will be faced with another 24-30 days in degraded mode. That's just not right...

Re: [HOWTO] How to increase raid rebuild speed

Posted: Wed Aug 22, 2018 3:26 am
by Trexx
RAID5 expansion vs. RAID-6 migration are significantly different processes. RAID-6 involves both restriping and the calculation of 2 sets of parity blocks. Speed for these calculation is dependent on the QNAP model & CPU's capabilities to leverage CPU HW capabilities to help with the parity calculations.

Also the time to complete listed in SSH is not accurate as throughput rates will change depending where on the drives you are.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Wed Aug 22, 2018 3:59 am
by lithiumus
Trexx wrote:RAID5 expansion vs. RAID-6 migration are significantly different processes. RAID-6 involves both restriping and the calculation of 2 sets of parity blocks. Speed for these calculation is dependent on the QNAP model & CPU's capabilities to leverage CPU HW capabilities to help with the parity calculations.

Also the time to complete listed in SSH is not accurate as throughput rates will change depending where on the drives you are.


Well, that's actually good news. I don't mind the longer wait to migrate to RAID-6 but definitely not when I expand my RAID-6 at a later date. I've got the ARM TS-1231XU with 8GB of memory. At the moment, CPU fluctuating between 2-12% most functionality disabled. I tried to increase the strip_cache and the box hung so that was a bad idea... it's set to 256 and I tried 16384... I'll just wait it out.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Wed Aug 22, 2018 4:20 am
by 321liftoff
I definitely noticed that the QNAP was not being fully utilized during the reshaping. I still think something is getting throttled unnecessarily.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Sun Sep 02, 2018 10:01 am
by lithiumus
lithiumus wrote:
Trexx wrote:RAID5 expansion vs. RAID-6 migration are significantly different processes. RAID-6 involves both restriping and the calculation of 2 sets of parity blocks. Speed for these calculation is dependent on the QNAP model & CPU's capabilities to leverage CPU HW capabilities to help with the parity calculations.

Also the time to complete listed in SSH is not accurate as throughput rates will change depending where on the drives you are.


Well, that's actually good news. I don't mind the longer wait to migrate to RAID-6 but definitely not when I expand my RAID-6 at a later date. I've got the ARM TS-1231XU with 8GB of memory. At the moment, CPU fluctuating between 2-12% most functionality disabled. I tried to increase the strip_cache and the box hung so that was a bad idea... it's set to 256 and I tried 16384... I'll just wait it out.


Well, 11 days since my last post and I'm at 43.3% still at about 3.5Mb/s. I reached out to Qnap support and we checked all the same things and they said it's going as fast as it can. The initial numbers were saying it was going to take 22-26 days and based on the SSH mdstat, it says I've got about 14 more days left... let's see if there is any change in speed after the 50% mark, but I'm not hopeful...

I do have 8 x 8TB drives going from a 7 x 8TB RAID5 to a 8 x 8TB RAID6 so yes, I know it's a lot of data and the size is large... just that 3.5mb/s sounds crazy...

Re: [HOWTO] How to increase raid rebuild speed

Posted: Sat Sep 08, 2018 1:56 am
by arnaldo
Running firmware 4.3.4 I was getting a 36hrs rebuild time for RAID5 array (4x4TB on a TS-469Pro). After I changed it to to "Rsync First" (high priority) the build time was down to 13hrs. It's the second drive that goes bad in a month (WD SE running for almost 5 years).

I would love to have a finer grain control and not just "low, med, high" (I don't mind the old style tweak), but for most GUI oriented folks it's nice to be able to have at lease some control.

Also, my guess is that, at least on my box, I could go a little faster as there is no resource contention (CPU is at 15-25%, I/O is not bad and there is still plenty of free memory).

Maybe QNAP could add an "advanced" mode, where one could really fine tune it.