[HOWTO] How to increase raid rebuild speed

Discussion on setting up QNAP NAS products.

[HOWTO] How to increase raid rebuild speed

Postby Don » Fri Dec 12, 2008 6:06 am

I found this article on the web. Use it at your own risk as I have not tested it and make no guarantees. If yo do try it please post your results here so others may benefit.

Increase the speed of Linux Software RAID reconstruction

If you are in a situation where you sit in front of the console (or on a remote ssh connection) waiting for a Linux software RAID to finish rebuilding (either you added a new drive, or you replaced a failed one, etc.) then you might be frustrated by how slow this process is running. You are running cat on /proc/mdstat repeatedly (you should really use watch in this case ;)), and this seems to never finish… Obviously that there is a logical reason for this ‘slowness‘ and on a production system you should leave it running with the defaults. But in case you want to speed up this process here is how you can do it. This will place a much higher load on the system so you should use it with care.

To see your Linux kernel speed limits imposed on the RAID reconstruction use:

Code: Select all
cat /proc/sys/dev/raid/speed_limit_max
200000
cat /proc/sys/dev/raid/speed_limit_min
1000

In the system logs you can see something similar to:
Code: Select all
md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.

This means that the minimum guaranteed speed of the rebuild of the array is approx 1MB/s. The actual speed will be higher and will depend on the system load and what other processes are running at that time.
In case you want to increase this minimum speed you need to enter a higher value in speed_limit_min. For example to set this to approx 50 megabytes per second as minimum use:
Code: Select all
echo 50000 >/proc/sys/dev/raid/speed_limit_min

The results are instant… you can return to the watch window to see it running, and hope that this will finish a little faster (this will really depend on the system you are running, the HDDs, controllers, etc.):
Code: Select all
watch cat /proc/mdstat

There is no watch program on the NAS so that part won't work. You will have to play with the numbers until you get one that works for you.

Article is located here --> http://www.ducea.com/2006/06/25/increas ... struction/
Search (upper right hand corner) before posting.

Unofficial QNAP How-To Guide: http://forum.qnap.com/viewtopic.php?f=11&t=10622
How-To increase raid rebuild speed: http://forum.qnap.com/viewtopic.php?f=11&t=10268
How-To install dnsmasq: http://forum.qnap.com/viewtopic.php?f=90&t=26395

NAS: TS-509 F/W 3.6.1.0302 | TS-419P+ F/W 4.1.0.0612| TS-639 F/W 3.6.1.0302 | TS-669 F/W 4.1.0.0612
QPKG: ipkg, LMS 7.7.1, PhpGedView, phpMyAdmin, WordPress, Xdove 1.3
IPKG: atop, busybox, dnsmasq, flac, htop, lame, less, libid3tag, lsof, man, man-pages, unzip, wakelan
OTHER: crontab, phpXmail, SpeedTest
N/W: Asus RT-N16 router | SMC gigagit switch
User avatar
Don
Moderator
 
Posts: 6570
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York
NAS Model: TS-509 Pro

Re: [HOWTO] How to increase raid rebuild speed

Postby KL643 » Tue Feb 17, 2009 3:22 pm

Hi Don,

Good tip! It worked like a charm. I just replaced my 4WD10EADS 1T for the new Samsungs 1 1/2 T in the TS-439 and after my Raid 5 with 3 disks inside was completed (and put all the data from the 4th drive on the raid array) I expand the Raid 5 with the 4th disk. After 2 1/2 days it told me that 39% was finished so I changed a bit in the settings you mentioned before i went to sleep and guess what...when i woke up it was finished :D thanks a lot for sharing this info!

Best regards,
KL643
The solution to the most difficult "problem" is usually very simple.

Qnap support

Before asking questions
User avatar
KL643
Experience counts
 
Posts: 1993
Joined: Sun Sep 23, 2007 9:21 pm
Location: @40000ft
NAS Model: TS-x70

Re: [HOWTO] How to increase raid rebuild speed

Postby alvine » Thu Feb 26, 2009 2:21 am

That's a perfect tip!!

A sincere thank you.
User avatar
alvine
Starting out
 
Posts: 16
Joined: Mon Apr 21, 2008 3:41 am
Location: Nice, France
NAS Model: TS-509 Pro

Re: [HOWTO] How to increase raid rebuild speed

Postby culrich » Sat May 02, 2009 2:20 am

Great tip... for those newb's out there like me, connect using putty on SSH ... regular telnet didn't get a response.
Google for PUTTY.EXE

One question: do you have to restart the synchronization after making the change, or will it pick up speed on-the-fly?

Thanks
culrich
New here
 
Posts: 8
Joined: Wed Apr 29, 2009 4:30 am
NAS Model: NMP-1000

Re: [HOWTO] How to increase raid rebuild speed

Postby Don » Sat May 02, 2009 5:52 am

It picks it up on the fly.
Search (upper right hand corner) before posting.

Unofficial QNAP How-To Guide: http://forum.qnap.com/viewtopic.php?f=11&t=10622
How-To increase raid rebuild speed: http://forum.qnap.com/viewtopic.php?f=11&t=10268
How-To install dnsmasq: http://forum.qnap.com/viewtopic.php?f=90&t=26395

NAS: TS-509 F/W 3.6.1.0302 | TS-419P+ F/W 4.1.0.0612| TS-639 F/W 3.6.1.0302 | TS-669 F/W 4.1.0.0612
QPKG: ipkg, LMS 7.7.1, PhpGedView, phpMyAdmin, WordPress, Xdove 1.3
IPKG: atop, busybox, dnsmasq, flac, htop, lame, less, libid3tag, lsof, man, man-pages, unzip, wakelan
OTHER: crontab, phpXmail, SpeedTest
N/W: Asus RT-N16 router | SMC gigagit switch
User avatar
Don
Moderator
 
Posts: 6570
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York
NAS Model: TS-509 Pro

Re: [HOWTO] How to increase raid rebuild speed

Postby tt55du » Fri May 22, 2009 1:34 am

Don,
Just tried this tip and again, thanks to to one of your posts I've found something useful.

However, what do you think the fastest I can push the Qnap on RAID expansion, I'm at the 50MB limit as suggested in your post. My CPU jumped from 14% to about 33% and my expansion is moving much quicker than before but still slow overall.

I'm adding 2 1TB 7200RPM WD Cavier Blacks to the 3 I have in there now. I figure the bottleneck would be the HD rather than the CPU here, I wonder how much I can safely increase the min.
tt55du
Starting out
 
Posts: 20
Joined: Wed Jan 14, 2009 1:02 pm
NAS Model: TS-419P

Re: [HOWTO] How to increase raid rebuild speed

Postby Don » Fri May 22, 2009 11:27 am

You can kep bunping it up as much as you want. At some point you will either reach the max the bus/harddrives can handle or max out the cpu.
Search (upper right hand corner) before posting.

Unofficial QNAP How-To Guide: http://forum.qnap.com/viewtopic.php?f=11&t=10622
How-To increase raid rebuild speed: http://forum.qnap.com/viewtopic.php?f=11&t=10268
How-To install dnsmasq: http://forum.qnap.com/viewtopic.php?f=90&t=26395

NAS: TS-509 F/W 3.6.1.0302 | TS-419P+ F/W 4.1.0.0612| TS-639 F/W 3.6.1.0302 | TS-669 F/W 4.1.0.0612
QPKG: ipkg, LMS 7.7.1, PhpGedView, phpMyAdmin, WordPress, Xdove 1.3
IPKG: atop, busybox, dnsmasq, flac, htop, lame, less, libid3tag, lsof, man, man-pages, unzip, wakelan
OTHER: crontab, phpXmail, SpeedTest
N/W: Asus RT-N16 router | SMC gigagit switch
User avatar
Don
Moderator
 
Posts: 6570
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York
NAS Model: TS-509 Pro

Re: [HOWTO] How to increase raid rebuild speed

Postby tt55du » Fri May 22, 2009 10:48 pm

Don,
Didn't have a Need to Bump it up it finished by the time I got home, but I had a strange issue.

I went from 3 1TB drives to 5
After Rebuilding was Completed and according to the the Log, about 15 min later I get the error

[RAID5 Disk Volume: Drive 1 2 3 4 5] Disk space is full.

I wasn't home and it was beeping every 5 minutes so I had my GF turn it off

When I got home I was relieved to find every booted ok and All I had to do was expand the raid size
That competed Fine but this morning I got the same error

[RAID5 Disk Volume: Drive 1 2 3 4 5] Disk space is full.

Only this time, according to the log, the expansion competed about 45 min after the Disk Space is full error.
Again I rebooted and everything came up fine.

Did a quick and dirty search, nothing came up quite like this. I wonder if its:
A. The min expansion size change
B. Actually some other application or service reporting the disk is full because it can't access the raid during the expansion
C. Something I do not understand.

Everything seems fine, I hope my systems performance wasn't degraded as a result of any of this.
tt55du
Starting out
 
Posts: 20
Joined: Wed Jan 14, 2009 1:02 pm
NAS Model: TS-419P

Re: [HOWTO] How to increase raid rebuild speed

Postby Don » Sat May 23, 2009 8:17 am

The disk space full message is a bug and supposed to be fixed in the latest firmware. 2.1.6 build 0428. See --> viewtopic.php?f=142&t=14070
Search (upper right hand corner) before posting.

Unofficial QNAP How-To Guide: http://forum.qnap.com/viewtopic.php?f=11&t=10622
How-To increase raid rebuild speed: http://forum.qnap.com/viewtopic.php?f=11&t=10268
How-To install dnsmasq: http://forum.qnap.com/viewtopic.php?f=90&t=26395

NAS: TS-509 F/W 3.6.1.0302 | TS-419P+ F/W 4.1.0.0612| TS-639 F/W 3.6.1.0302 | TS-669 F/W 4.1.0.0612
QPKG: ipkg, LMS 7.7.1, PhpGedView, phpMyAdmin, WordPress, Xdove 1.3
IPKG: atop, busybox, dnsmasq, flac, htop, lame, less, libid3tag, lsof, man, man-pages, unzip, wakelan
OTHER: crontab, phpXmail, SpeedTest
N/W: Asus RT-N16 router | SMC gigagit switch
User avatar
Don
Moderator
 
Posts: 6570
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York
NAS Model: TS-509 Pro

Re: [HOWTO] How to increase raid rebuild speed

Postby Cybix » Sat May 23, 2009 10:52 am

be warned that adjusting the min raid speed can have a huge impact on CPU usage.

I tried 10000 and 20000, I found that 7000 gave approx 75%~ of cpu usage.. 10000 and over was 100% cpu the whole time. This on a TS-639

If your comfortable running it this hard, go right ahead, there's probably no issue with loading it up..
Cybix
Getting the hang of things
 
Posts: 57
Joined: Sun May 03, 2009 7:53 am
NAS Model: TS-239 Pro

Re: [HOWTO] How to increase raid rebuild speed

Postby Don » Sat May 23, 2009 11:00 am

Cybix wrote:be warned that adjusting the min raid speed can have a huge impact on CPU usage.

I tried 10000 and 20000, I found that 7000 gave approx 75%~ of cpu usage.. 10000 and over was 100% cpu the whole time. This on a TS-639

If your comfortable running it this hard, go right ahead, there's probably no issue with loading it up..

Agreed. You have to decide what is more important to you - maintaining access to the system or getting the rebuild done as quickly as possible - and selecting the proper balance.
Search (upper right hand corner) before posting.

Unofficial QNAP How-To Guide: http://forum.qnap.com/viewtopic.php?f=11&t=10622
How-To increase raid rebuild speed: http://forum.qnap.com/viewtopic.php?f=11&t=10268
How-To install dnsmasq: http://forum.qnap.com/viewtopic.php?f=90&t=26395

NAS: TS-509 F/W 3.6.1.0302 | TS-419P+ F/W 4.1.0.0612| TS-639 F/W 3.6.1.0302 | TS-669 F/W 4.1.0.0612
QPKG: ipkg, LMS 7.7.1, PhpGedView, phpMyAdmin, WordPress, Xdove 1.3
IPKG: atop, busybox, dnsmasq, flac, htop, lame, less, libid3tag, lsof, man, man-pages, unzip, wakelan
OTHER: crontab, phpXmail, SpeedTest
N/W: Asus RT-N16 router | SMC gigagit switch
User avatar
Don
Moderator
 
Posts: 6570
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York
NAS Model: TS-509 Pro

Re: [HOWTO] How to increase raid rebuild speed

Postby The-Shadow-TS » Sat Jun 27, 2009 11:20 pm

I added a new 1tb drive to my 3 drive RAID5 on my TS-639 Pro and after 12h it was only at 11%. Needless to say this was going to take a few days at that rate. I did a bid of searching on this great forum and found this post. I checked the speed_limit_min and was set to 5000. I changed it to 10000, then 20000, 50000 and I could immediately hear the hard drives working faster. The transfer at the time seemed to peak at 18333k/sec using the cat /proc/mdstat command with 33% cpu usuage so I left it at 50000 just in case it could speed up during the night. I left it run overnight and in the morning, 12h after, it was done :).

It would be nice to be able adjust this setting in the web management page, with the usual disclaimers on it affecting performance, since 5 days to rebuild a raid array is a little excessive with the default settings.

Thanks again Don,
The-Shadow-TS
The-Shadow-TS
New here
 
Posts: 3
Joined: Mon Jun 22, 2009 9:59 pm
NAS Model: TS-239 Pro

Re: [HOWTO] How to increase raid rebuild speed

Postby jeffster » Mon Jun 29, 2009 3:24 am

I'm a new owner of a TS-809 pro and have just added 5 new 2tb drives...I've tried using the great info from the thread but it doesn't seem to speed up the process on my system - does anyone know a way to speed up the rebuild on this system?

thanks

-J
jeffster
New here
 
Posts: 5
Joined: Mon Jun 29, 2009 3:20 am
NAS Model: TS-639U

Re: [HOWTO] How to increase raid rebuild speed

Postby enderlabs » Wed Jul 01, 2009 12:37 am

tt55du wrote:Don,
Just tried this tip and again, thanks to to one of your posts I've found something useful.

However, what do you think the fastest I can push the Qnap on RAID expansion, I'm at the 50MB limit as suggested in your post. My CPU jumped from 14% to about 33% and my expansion is moving much quicker than before but still slow overall.

I'm adding 2 1TB 7200RPM WD Cavier Blacks to the 3 I have in there now. I figure the bottleneck would be the HD rather than the CPU here, I wonder how much I can safely increase the min.


Can you add step by step instructions for QNAP ?
I can not manage to find the files in order to change the values.
Qnap TS-670 Pro
enderlabs
Getting the hang of things
 
Posts: 53
Joined: Sun Apr 12, 2009 10:36 pm
Location: Bucharest
NAS Model: TS-x70

Re: [HOWTO] How to increase raid rebuild speed

Postby tt55du » Wed Jul 01, 2009 1:11 am

You should not have to change any files. I have a 639, not sure what you have
After Inserting the HD's you wish to expand, Initiate the RAID expansion through the Web GUI.
Once that is complete, ssh into the qnap and run this command at the prompt

echo 50000 >/proc/sys/dev/raid/speed_limit_min

I found 50MB to be still slow, but ok, it took like a whole day.
You can probably increase it to 70000 or higher without too much issue (depending on CPU and how much you intend to access the NAS during expansion.

Also, I would update the Firmware before you expand. the 639 had a false error during rebuild taht was fixed in a later firmware.

Good Luck
tt55du
Starting out
 
Posts: 20
Joined: Wed Jan 14, 2009 1:02 pm
NAS Model: TS-419P

Next

Return to Turbo Station Installation & Setup

Who is online

Users browsing this forum: No registered users and 9 guests