[HOWTO] How to increase raid rebuild speed

[trueimage]

cat /proc/sys/dev/raid/speed_limit_max
200000
cat /proc/sys/dev/raid/speed_limit_min
5000

cat /proc/mdstat shows:

md0 : active raid5 sda3[4] sdd3[3] sdc3[2] sdb3[1]
5855836800 blocks level 5, 64k chunk, algorithm 2 [4/3] [_UUU]
[>....................] recovery = 2.4% (48365644/1951945600) finish=68209.6min speed=465K/sec


why so slow? 68,210 minutes is ~48 days????

[Don]

Applies to all models.

[P3R]

why so slow? 68,210 minutes is ~48 days????

Because you have an error on the remaining disks. I doubt that you can solve that by tweaking.

[jello_moto]

The rebuilds have always taken so ridiculously long that I've found its just faster to create a new array and copy all the data back over the network. And now that the newer boxes have 10 gigabit, its really an order of magnitude slower to rebuild than copy over the network.

[schumaku]

The rebuilds have always taken so ridiculously long that I've found its just faster to create a new array and copy all the data back over the network. And now that the newer boxes have 10 gigabit, its really an order of magnitude slower to rebuild than copy over the network.

The key advantage of a RAID rebuild ie. following a disk replacement is the ongoing availability of the storage without interrupting the service.

[Briain]

I've noticed that not only will this increase the RAID rebuild speed but has also increased my network thoroughput. I set my min at 100000 and everything is hauling ** now!!

Thanks OP for the information.

Hi

The above is rather interesting. Has anyone else verified that by testing to see whether it changes performance under normal usage conditions? I often back one NAS (TS-659 with RAID 6) to another NAS (TS-219P+ with RAID 1). It usually takes only about 15 hours to transfer 1 TB of >30 MB FLAC files (and about 200 GB of smaller files), so I'll maybe try that trick on the TS-219P+ just to see if I can detect any difference.

Bri

[Metzlmane]

Thanks for the information. Now it's running a little faster.

2x Hitachi Ultrastar 7K4000 3000GB, SATA 6Gb/s (HUS724030ALE640/0F14684)
Raid 1

Code: Select all

Min 100000
Max 200000

md0 : active raid1 sdb3[1] sda3[0]
      2928697556 blocks super 1.0 [2/2] [UU]
      [=====>...............]  resync = 29.0% (851113152/2928697556) finish=273.3min speed=126652K/sec


on TS1079
The CPU is bored, but the harddisks can't go faster.

[DarkLogix]

I've noticed that not only will this increase the RAID rebuild speed but has also increased my network thoroughput. I set my min at 100000 and everything is hauling ** now!!

Thanks OP for the information.

Have you done some benchmarks on this?
and do you think it'd apply to a TS-469-Pro?

[raffe]

Thanks for a good tip Don!

...
...

Code: Select all

watch cat /proc/mdstat

There is no watch program on the NAS so that part won't work...
...

To get something similar to "watch", it's possible to use a command line like this (just write (or copy/paste) the whole line and press [Enter]):

Code: Select all

while true; do cat /proc/mdstat; sleep 1; done

End it with Ctrl + C. The number "1" in "sleep 1" makes it wait one second before again running the command "cat /proc/mdstat" in a loop. Change 1 to higher value if you want longer time between updates. Not the same as watch, but anyhow...

[chrispage]

Hi guys.

I'm going through the painful task of upgrading 6 x 2TB disks with 3TB WD Red disks in my 659 Pro II .. The first disk took approximately 100 hrs and the second one is going almost as slow!

finish=4673.1min speed=5154K/sec

Any ideas?


Thanks

UPDATE

Okay, i used the echo command to set the minimum and its now running at 48000K/sec! AWESOME

[mannebk]

thx, pushed my 410 to 50 as well, nice tip,

my systemload was already constant over 50% but jumping around and now its a solid 100% for a raid 5 expansion from 3 to 4 500gb hdds.

Manne

[fantomas]

There's one more option for rebuilding RAID5/RAID6 devices:

# cat /sys/block/md0/md/stripe_cache_size
4096

increasing this could help, last time I have tried 16384 (4x more).

[gPaq]

Although this tip is very much appreciated, and all the commands work on my newly purchased TS-469 Pro, making the changes on "speed_limit_min" makes not a lick of difference, unfortunately. I increased this speed gradually from the default 5000 to 50000, up to 150000, and basically nothing happens. The processor utilization does not go up (hovers around 16% before and after changes), and after 11 hours of re-striping I'm at 17%, which puts my ETA at about 4 more days to completion. That's not reasonable anymore... Is there anything that puts a damper on this configuration in newer models? As some other users indicated, the changes seem to have no effect whatsoever. Below are my settings for reference (apparently indicating current throughput at 22868K/sec or almost 23 MB/sec, right?). This is a TS-469 Pro with latest firmware 3.8.1 and 3 GB memory (default 1 GB with 2 GB added), expanding RAID5 from 3 disks to 4 disks, each being a Western Digital 3 GB Red WD30EFRX.

[~] # cat /proc/sys/dev/raid/speed_limit_max
200000
[~] # cat /proc/sys/dev/raid/speed_limit_min
150000
[~] #

[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sdd3[3] sda3[0] sdc3[2] sdb3[1]
5857395072 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[======>..............] reshape = 34.2% (1003124924/2928697536) finish=1403.3min speed=22868K/sec

md4 : active raid1 sdd2[2](S) sdc2[3](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]

md13 : active raid1 sda4[0] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [4/4] [UUUU]
bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [4/4] [UUUU]
bitmap: 0/65 pages [0KB], 4KB chunk

unused devices: <none>
[~] #

[schumaku]

A reshape (change RAID layout) is much more intensive than resync.

[gPaq]

A reshape (change RAID layout) is much more intensive than resync.

Dear schumaku, I believe I understand what you are saying and that may indeed be so, however, from the web GUI I simply selected to Add Hard Drive, going from RAID 5 with 3 disks to RAID 5 with 4 disks, and this is the process that the QNAP is executing. I wish I could change it, or at least make it go faster, especially because there is no data on this volume (other than the default shares): this is a brand new installation and I will copy my data to the NAS at a later time when the RAID configuration is stable. Any further input is appreciated.