Page 5 of 14

Re: [HOWTO] How to increase raid rebuild speed

Posted: Thu Feb 09, 2012 9:12 am
by trueimage
cat /proc/sys/dev/raid/speed_limit_max
200000
cat /proc/sys/dev/raid/speed_limit_min
5000

cat /proc/mdstat shows:

md0 : active raid5 sda3[4] sdd3[3] sdc3[2] sdb3[1]
5855836800 blocks level 5, 64k chunk, algorithm 2 [4/3] [_UUU]
[>....................] recovery = 2.4% (48365644/1951945600) finish=68209.6min speed=465K/sec


why so slow? 68,210 minutes is ~48 days????

Re: [HOWTO] How to increase raid rebuild speed

Posted: Thu Feb 09, 2012 11:23 am
by Don
Applies to all models.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Thu Feb 09, 2012 3:23 pm
by P3R
trueimage wrote:why so slow? 68,210 minutes is ~48 days????
Because you have an error on the remaining disks. I doubt that you can solve that by tweaking.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Thu Feb 16, 2012 7:17 am
by jello_moto
The rebuilds have always taken so ridiculously long that I've found its just faster to create a new array and copy all the data back over the network. And now that the newer boxes have 10 gigabit, its really an order of magnitude slower to rebuild than copy over the network.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Mon Apr 02, 2012 3:36 pm
by schumaku
jello_moto wrote:The rebuilds have always taken so ridiculously long that I've found its just faster to create a new array and copy all the data back over the network. And now that the newer boxes have 10 gigabit, its really an order of magnitude slower to rebuild than copy over the network.
The key advantage of a RAID rebuild ie. following a disk replacement is the ongoing availability of the storage without interrupting the service.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Tue Apr 03, 2012 12:51 am
by Briain
taylortap wrote:I've noticed that not only will this increase the RAID rebuild speed but has also increased my network thoroughput. I set my min at 100000 and everything is hauling ** now!!

Thanks OP for the information.


Hi

The above is rather interesting. Has anyone else verified that by testing to see whether it changes performance under normal usage conditions? I often back one NAS (TS-659 with RAID 6) to another NAS (TS-219P+ with RAID 1). It usually takes only about 15 hours to transfer 1 TB of >30 MB FLAC files (and about 200 GB of smaller files), so I'll maybe try that trick on the TS-219P+ just to see if I can detect any difference.

Bri :)

Re: [HOWTO] How to increase raid rebuild speed

Posted: Sun Jul 01, 2012 6:01 pm
by Metzlmane
Thanks for the information. Now it's running a little faster.

2x Hitachi Ultrastar 7K4000 3000GB, SATA 6Gb/s (HUS724030ALE640/0F14684)
Raid 1

Code: Select all

Min 100000
Max 200000

md0 : active raid1 sdb3[1] sda3[0]
      2928697556 blocks super 1.0 [2/2] [UU]
      [=====>...............]  resync = 29.0% (851113152/2928697556) finish=273.3min speed=126652K/sec

on TS1079
The CPU is bored, but the harddisks can't go faster.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Fri Oct 19, 2012 2:50 am
by DarkLogix
taylortap wrote:I've noticed that not only will this increase the RAID rebuild speed but has also increased my network thoroughput. I set my min at 100000 and everything is hauling ** now!!

Thanks OP for the information.

Have you done some benchmarks on this?
and do you think it'd apply to a TS-469-Pro?

Re: [HOWTO] How to increase raid rebuild speed

Posted: Fri Oct 26, 2012 2:38 am
by raffe
Thanks for a good tip Don!
Don wrote:...
...

Code: Select all

watch cat /proc/mdstat

There is no watch program on the NAS so that part won't work...
...

To get something similar to "watch", it's possible to use a command line like this (just write (or copy/paste) the whole line and press [Enter]):

Code: Select all

while true; do cat /proc/mdstat; sleep 1; done

End it with Ctrl + C. The number "1" in "sleep 1" makes it wait one second before again running the command "cat /proc/mdstat" in a loop. Change 1 to higher value if you want longer time between updates. Not the same as watch, but anyhow...

Re: [HOWTO] How to increase raid rebuild speed

Posted: Sat Dec 01, 2012 2:04 pm
by chrispage
Hi guys.

I'm going through the painful task of upgrading 6 x 2TB disks with 3TB WD Red disks in my 659 Pro II .. The first disk took approximately 100 hrs and the second one is going almost as slow!

finish=4673.1min speed=5154K/sec

Any ideas?


Thanks

UPDATE

Okay, i used the echo command to set the minimum and its now running at 48000K/sec! AWESOME :)

Re: [HOWTO] How to increase raid rebuild speed

Posted: Tue Dec 25, 2012 12:18 am
by mannebk
thx, pushed my 410 to 50 as well, nice tip,

my systemload was already constant over 50% but jumping around and now its a solid 100% for a raid 5 expansion from 3 to 4 500gb hdds.

Manne

Re: [HOWTO] How to increase raid rebuild speed

Posted: Wed Feb 20, 2013 12:56 am
by fantomas
There's one more option for rebuilding RAID5/RAID6 devices:

# cat /sys/block/md0/md/stripe_cache_size
4096

increasing this could help, last time I have tried 16384 (4x more).

Re: [HOWTO] How to increase raid rebuild speed

Posted: Sun Mar 03, 2013 11:33 pm
by gPaq
Although this tip is very much appreciated, and all the commands work on my newly purchased TS-469 Pro, making the changes on "speed_limit_min" makes not a lick of difference, unfortunately. I increased this speed gradually from the default 5000 to 50000, up to 150000, and basically nothing happens. The processor utilization does not go up (hovers around 16% before and after changes), and after 11 hours of re-striping I'm at 17%, which puts my ETA at about 4 more days to completion. That's not reasonable anymore... Is there anything that puts a damper on this configuration in newer models? As some other users indicated, the changes seem to have no effect whatsoever. Below are my settings for reference (apparently indicating current throughput at 22868K/sec or almost 23 MB/sec, right?). This is a TS-469 Pro with latest firmware 3.8.1 and 3 GB memory (default 1 GB with 2 GB added), expanding RAID5 from 3 disks to 4 disks, each being a Western Digital 3 GB Red WD30EFRX.

[~] # cat /proc/sys/dev/raid/speed_limit_max
200000
[~] # cat /proc/sys/dev/raid/speed_limit_min
150000
[~] #

[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sdd3[3] sda3[0] sdc3[2] sdb3[1]
5857395072 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[======>..............] reshape = 34.2% (1003124924/2928697536) finish=1403.3min speed=22868K/sec

md4 : active raid1 sdd2[2](S) sdc2[3](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]

md13 : active raid1 sda4[0] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [4/4] [UUUU]
bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [4/4] [UUUU]
bitmap: 0/65 pages [0KB], 4KB chunk

unused devices: <none>
[~] #

Re: [HOWTO] How to increase raid rebuild speed

Posted: Mon Mar 04, 2013 12:01 am
by schumaku
A reshape (change RAID layout) is much more intensive than resync.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Mon Mar 04, 2013 1:08 am
by gPaq
schumaku wrote:A reshape (change RAID layout) is much more intensive than resync.


Dear schumaku, I believe I understand what you are saying and that may indeed be so, however, from the web GUI I simply selected to Add Hard Drive, going from RAID 5 with 3 disks to RAID 5 with 4 disks, and this is the process that the QNAP is executing. I wish I could change it, or at least make it go faster, especially because there is no data on this volume (other than the default shares): this is a brand new installation and I will copy my data to the NAS at a later time when the RAID configuration is stable. Any further input is appreciated.