Page 10 of 14

Re: [HOWTO] How to increase raid rebuild speed

Posted: Wed Mar 04, 2015 7:18 am
by tom4moore
So I am pleased to report back that my storage configuration (in the above post) completed syncing in just over 10 hours. Amazing stuff... I see that for some people who posted, they report that changing the maximum and minimum speed did not produce any results or change in sync speed. - I played around with the maximum speed and can confidently report back that the system responded according the my input. Speeding up the process as I adjusted the max speed threshold. Probably saved me having to wait around for another day or two judging from what Ive seen in this thread. Good stuff!

TS-451 with FW/ 4.1.3 0217 and 3x HGST HDN724040ALE640 4TB 7200RPM NAS drives in a RAID5 configuration.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Fri Nov 13, 2015 3:25 am
by tletourneau
I'm glad I found this, thanks for posting! It decreased my rebuild from 1161 minutes remaining to 228 minutes, I only wish I had found it last night when I started, it would probably be done by now. Set the max to 500000 and the min to 100000 with negligible CPU impact. I can afford to have the disk IO eaten up by the rebuild to get it done sooner. I just need to remember to set it back when I am done. My units defaults were max=200000 and min=5000.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Sun Nov 22, 2015 5:52 am
by firej
Can I ask something? If it is safe why dont you put this configuration on update firmware so make us all happy without changing it manually?

Re: [HOWTO] How to increase raid rebuild speed

Posted: Sun Nov 22, 2015 8:35 am
by Don
You are asking the wrong audience. This is just a community forum and not QNAP support. If I had to guess I would say that it has to do with NAS availability. The higher the rebuild rate the less responsive the NAS will be. The best solution would be to offer an option to change the rebuild rate either through a slider or a high, medium, and low option. Not everyone wants the same rebuild. Some favor getting rebuilt as quickly as possible at the expense of NAS responsiveness while others favor keeping the NAS responsive and letting the raid rebuild take longer. There should not be a one value fits all approach. Until QNAP offers the option to adjust the rebuild speed, if they ever do, you have the option of doing it via the command line. If you always want the same value then you can add the commands to the autorun file. If you would like to see this added as an option then open a ticket with QNAP support.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Sun Nov 22, 2015 4:52 pm
by firej
I would totally agree with you, you have right. What does bytes per inode? It was set default to 16384. Maybe this decrease the speed? The Max size of volume? How can be 63,99TB when I have only 2x3TB HGS disks?

Re: [HOWTO] How to increase raid rebuild speed

Posted: Wed Dec 16, 2015 4:41 am
by knowles13088
For a TS-469 L, does anybody have suggestions for a top end number for speed_limit_min? This system has no real user interactive I/O at the moment, so I can pretty much monopolize the system in the rebuild. I am in the midst of replacing drives one by one with larger drives to increase capacity. On drive # 3 of a RAID 10 array . Found this thread last night while working on Drive 1 and upped to 150000. Drive 2 went pretty quickly (about 90 minutes), but drive 3 is about 17 % done after 2 hours.

All I have seen in this thread is about speed_limit_min. I have left speed_limit_max alone.

Thanks!

Re: [HOWTO] How to increase raid rebuild speed

Posted: Thu Dec 24, 2015 6:28 pm
by Levo
Totally agree with the comments against the limiting all migrations to multi-day slow speeds to preserve productive capacity.
I currently have my old 639 and am trying to setup the new TS-471 as quickly as possible so I can move my data across and am having to wait.
Specifically at this point I am migrating a WD RED 6TB RAID1 to 2+1 RAID5 and I wanted to share my findings:
- I have played with the speed_limit_min/max parameters and whatever I did, I couldn't get the rebuild speed above ~25MB/s while the progress was below 50%.
- Once I got above 50%, it seems to have settled down to around ~106MB/s, cutting down the finish time by 75%+, i.e. instead of 30+ hours down to 7 or so.
Hope this helps others.
- Update: at 70% progress, rate down to ~94MB/s.

P.S. I am vague on this but I thought in the past I saw the new volume size at half way point but the migration is at 59% and still showing the old size. When could I expect to see the new size?

Re: [HOWTO] How to increase raid rebuild speed

Posted: Fri Jan 08, 2016 11:45 pm
by kwslark
User error. That's all, folks.

Re: [HOWTO] How to increase raid rebuild speed

Posted: Sat Jan 09, 2016 4:40 am
by P3R
kwslark wrote:I logged in to our TVS-1271U-RP-i7-32G via PuTTY using SSH, this command isn't working...
You need to define "isn't working".

Not possible to change, error messages, no effect or something else?

Re: [HOWTO] How to increase raid rebuild speed

Posted: Sun Jan 17, 2016 5:34 pm
by seraax
Hello everyone,

I am reshaping 453mini (RAM 2Gb - 4x HGST IDK 3TB 7200 RPM) from RAID5 (3 disks) to RAID6 (4 disks) - and the speed is extremely slow (around 5000K/sec).
I've tried everything from the post and it doesn’t change the situation at all, CPU load doesn't go over 6-10% and the speed 5300K/sec at max!

Here is what I’ve tried:

Code: Select all

echo 800000 >/proc/sys/dev/raid/speed_limit_max
echo 400000 >/proc/sys/dev/raid/speed_limit_min

I gradually started setting up a speed_limit_min to 50000 as suggested and then tried even 400000 with no impact on CPU or the speed at all.

Also tried to change cache, but no luck:

Code: Select all

cat: /sys/block/md0/md/stripe_cache_size

returns: No such file or directory, why is that?


Code: Select all

cat /proc/mdstat

returns:
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md1 : active raid6 sde3[5] sdc3[2] sda3[3] sdb3[4]
5840623240 blocks super 1.0 level 6, 4k chunk, algorithm 18 [4/3] [UUU_]
[>....................] reshape = 2.9% (86089728/2920311620) finish=9334.9min speed=5059K/sec

md256 : active raid1 sde2[3](S) sda2[2](S) sdb2[1] sdc2[0]
530112 blocks super 1.0 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

md13 : active raid1 sde4[24] sdc4[25] sdb4[27] sda4[26]
458880 blocks super 1.0 [24/4] [UUUU____________________]
bitmap: 1/1 pages [4KB], 65536KB chunk

md9 : active raid1 sde1[24] sdc1[25] sdb1[27] sda1[26]
530048 blocks super 1.0 [24/4] [UUUU____________________]
bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>


Code: Select all

egrep speed /proc/mdstat

returns: (progress after 17 hours)
[==>..................] reshape = 11.1% (324481024/2920311620) finish=8500.6min speed=5088K/sec


What do I do wrong? Can anyone help please?

Thank you

Re: [HOWTO] How to increase raid rebuild speed

Posted: Tue Jan 19, 2016 10:27 pm
by seraax
I've been reshaping RAID5 to RAID6 for over 3 days now and it's only on 44%... I've played with the cache size as well, changed it to 32768, 16384 and 8192, nothing does help.
Does anyone have any suggestions what I can do to make it faster than 5300K/sec please?

Re: [HOWTO] How to increase raid rebuild speed

Posted: Wed Apr 20, 2016 12:57 am
by Ifti
Same here - adjusted mine to 200000 and my CPU doesn't go over 2%, so I can only assume its not working!!
I have a TVS-871T (i7 model)

I get this:

[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec

Re: [HOWTO] How to increase raid rebuild speed

Posted: Wed Apr 20, 2016 6:55 am
by P3R
Ifti wrote:Same here - adjusted mine to 200000 and my CPU doesn't go over 2%, so I can only assume its not working!!
My assumption would instead be that your very fast CPU can easily manage this task already without changing this setting and that your disks are the performance bottlenecks.
I have a TVS-871T (i7 model)
This thread was started in a time when the fastest NAS CPUs had someting like 1/15th or 1/20th of the processing capacity your CPU have.
[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec
Looks decent to me. What's wrong with it?

Re: [HOWTO] How to increase raid rebuild speed

Posted: Thu Apr 21, 2016 12:11 am
by Ifti
P3R wrote:
Ifti wrote:Same here - adjusted mine to 200000 and my CPU doesn't go over 2%, so I can only assume its not working!!
My assumption would instead be that your very fast CPU can easily manage this task already without changing this setting and that your disks are the performance bottlenecks.
I have a TVS-871T (i7 model)
This thread was started in a time when the fastest NAS CPUs had someting like 1/15th or 1/20th of the processing capacity your CPU have.
[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec
Looks decent to me. What's wrong with it?



Thanks
3x6TB disks took around 14 hours in the end!
All set - until I add another disk! lol

Re: [HOWTO] How to increase raid rebuild speed

Posted: Wed Apr 27, 2016 4:06 pm
by Rexxy
Works nicely, I used this last year on a TS431 and cut down a Raid5 from 4 days to 2.5days.

I am currently doing another Raid5 4 x 4TB on a TVS-663 and after changing the stats it immediately increased..


md1 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[0]
11691190848 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[>....................] resync = 2.2% (85832136/3897063616) finish=329.8min speed=192562K/sec

CPU stats look ok too

PID USER STATUS RSS PPID %CPU %MEM COMMAND
26707 admin SW 0 2 13.8 0.0 md1_raid5
27258 admin DWN 0 2 9.1 0.0 md1_resync