[HOWTO] How to increase raid rebuild speed

Discussion on setting up QNAP NAS products.
Locked
User avatar
Don
Guru
Posts: 12289
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

[HOWTO] How to increase raid rebuild speed

Post by Don »

I found this article on the web. Use it at your own risk as I have not tested it and make no guarantees. If yo do try it please post your results here so others may benefit.
Increase the speed of Linux Software RAID reconstruction

If you are in a situation where you sit in front of the console (or on a remote ssh connection) waiting for a Linux software RAID to finish rebuilding (either you added a new drive, or you replaced a failed one, etc.) then you might be frustrated by how slow this process is running. You are running cat on /proc/mdstat repeatedly (you should really use watch in this case ;)), and this seems to never finish… Obviously that there is a logical reason for this ‘slowness‘ and on a production system you should leave it running with the defaults. But in case you want to speed up this process here is how you can do it. This will place a much higher load on the system so you should use it with care.

To see your Linux kernel speed limits imposed on the RAID reconstruction use:

Code: Select all

cat /proc/sys/dev/raid/speed_limit_max
200000
cat /proc/sys/dev/raid/speed_limit_min
1000
In the system logs you can see something similar to:

Code: Select all

md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.
This means that the minimum guaranteed speed of the rebuild of the array is approx 1MB/s. The actual speed will be higher and will depend on the system load and what other processes are running at that time.
In case you want to increase this minimum speed you need to enter a higher value in speed_limit_min. For example to set this to approx 50 megabytes per second as minimum use:

Code: Select all

echo 50000 >/proc/sys/dev/raid/speed_limit_min
The results are instant… you can return to the watch window to see it running, and hope that this will finish a little faster (this will really depend on the system you are running, the HDDs, controllers, etc.):

Code: Select all

watch cat /proc/mdstat
There is no watch program on the NAS so that part won't work. You will have to play with the numbers until you get one that works for you.

Article is located here --> http://www.ducea.com/2006/06/25/increas ... struction/
Use the forum search feature before posting.

Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
User avatar
KL643
Experience counts
Posts: 1996
Joined: Sun Sep 23, 2007 9:21 pm
Location: @40000ft
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by KL643 »

Hi Don,

Good tip! It worked like a charm. I just replaced my 4WD10EADS 1T for the new Samsungs 1 1/2 T in the TS-439 and after my Raid 5 with 3 disks inside was completed (and put all the data from the 4th drive on the raid array) I expand the Raid 5 with the 4th disk. After 2 1/2 days it told me that 39% was finished so I changed a bit in the settings you mentioned before i went to sleep and guess what...when i woke up it was finished :D thanks a lot for sharing this info!

Best regards,
KL643
User avatar
alvine
Starting out
Posts: 16
Joined: Mon Apr 21, 2008 3:41 am
Location: Nice, France

Re: [HOWTO] How to increase raid rebuild speed

Post by alvine »

That's a perfect tip!!

A sincere thank you.
culrich
New here
Posts: 8
Joined: Wed Apr 29, 2009 4:30 am

Re: [HOWTO] How to increase raid rebuild speed

Post by culrich »

Great tip... for those newb's out there like me, connect using putty on SSH ... regular telnet didn't get a response.
Google for PUTTY.EXE

One question: do you have to restart the synchronization after making the change, or will it pick up speed on-the-fly?

Thanks
User avatar
Don
Guru
Posts: 12289
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: [HOWTO] How to increase raid rebuild speed

Post by Don »

It picks it up on the fly.
Use the forum search feature before posting.

Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
tt55du
Starting out
Posts: 24
Joined: Wed Jan 14, 2009 1:02 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by tt55du »

Don,
Just tried this tip and again, thanks to to one of your posts I've found something useful.

However, what do you think the fastest I can push the Qnap on RAID expansion, I'm at the 50MB limit as suggested in your post. My CPU jumped from 14% to about 33% and my expansion is moving much quicker than before but still slow overall.

I'm adding 2 1TB 7200RPM WD Cavier Blacks to the 3 I have in there now. I figure the bottleneck would be the HD rather than the CPU here, I wonder how much I can safely increase the min.
User avatar
Don
Guru
Posts: 12289
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: [HOWTO] How to increase raid rebuild speed

Post by Don »

You can kep bunping it up as much as you want. At some point you will either reach the max the bus/harddrives can handle or max out the cpu.
Use the forum search feature before posting.

Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
tt55du
Starting out
Posts: 24
Joined: Wed Jan 14, 2009 1:02 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by tt55du »

Don,
Didn't have a Need to Bump it up it finished by the time I got home, but I had a strange issue.

I went from 3 1TB drives to 5
After Rebuilding was Completed and according to the the Log, about 15 min later I get the error

[RAID5 Disk Volume: Drive 1 2 3 4 5] Disk space is full.

I wasn't home and it was beeping every 5 minutes so I had my GF turn it off

When I got home I was relieved to find every booted ok and All I had to do was expand the raid size
That competed Fine but this morning I got the same error

[RAID5 Disk Volume: Drive 1 2 3 4 5] Disk space is full.

Only this time, according to the log, the expansion competed about 45 min after the Disk Space is full error.
Again I rebooted and everything came up fine.

Did a quick and dirty search, nothing came up quite like this. I wonder if its:
A. The min expansion size change
B. Actually some other application or service reporting the disk is full because it can't access the raid during the expansion
C. Something I do not understand.

Everything seems fine, I hope my systems performance wasn't degraded as a result of any of this.
User avatar
Don
Guru
Posts: 12289
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: [HOWTO] How to increase raid rebuild speed

Post by Don »

The disk space full message is a bug and supposed to be fixed in the latest firmware. 2.1.6 build 0428. See --> http://forum.qnap.com/viewtopic.php?f=142&t=14070
Use the forum search feature before posting.

Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
Cybix
Getting the hang of things
Posts: 57
Joined: Sun May 03, 2009 7:53 am

Re: [HOWTO] How to increase raid rebuild speed

Post by Cybix »

be warned that adjusting the min raid speed can have a huge impact on CPU usage.

I tried 10000 and 20000, I found that 7000 gave approx 75%~ of cpu usage.. 10000 and over was 100% cpu the whole time. This on a TS-639

If your comfortable running it this hard, go right ahead, there's probably no issue with loading it up..
User avatar
Don
Guru
Posts: 12289
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: [HOWTO] How to increase raid rebuild speed

Post by Don »

Cybix wrote:be warned that adjusting the min raid speed can have a huge impact on CPU usage.

I tried 10000 and 20000, I found that 7000 gave approx 75%~ of cpu usage.. 10000 and over was 100% cpu the whole time. This on a TS-639

If your comfortable running it this hard, go right ahead, there's probably no issue with loading it up..
Agreed. You have to decide what is more important to you - maintaining access to the system or getting the rebuild done as quickly as possible - and selecting the proper balance.
Use the forum search feature before posting.

Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
The-Shadow-TS
New here
Posts: 3
Joined: Mon Jun 22, 2009 9:59 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by The-Shadow-TS »

I added a new 1tb drive to my 3 drive RAID5 on my TS-639 Pro and after 12h it was only at 11%. Needless to say this was going to take a few days at that rate. I did a bid of searching on this great forum and found this post. I checked the speed_limit_min and was set to 5000. I changed it to 10000, then 20000, 50000 and I could immediately hear the hard drives working faster. The transfer at the time seemed to peak at 18333k/sec using the cat /proc/mdstat command with 33% cpu usuage so I left it at 50000 just in case it could speed up during the night. I left it run overnight and in the morning, 12h after, it was done :).

It would be nice to be able adjust this setting in the web management page, with the usual disclaimers on it affecting performance, since 5 days to rebuild a raid array is a little excessive with the default settings.

Thanks again Don,
The-Shadow-TS
jeffster
New here
Posts: 5
Joined: Mon Jun 29, 2009 3:20 am

Re: [HOWTO] How to increase raid rebuild speed

Post by jeffster »

I'm a new owner of a TS-809 pro and have just added 5 new 2tb drives...I've tried using the great info from the thread but it doesn't seem to speed up the process on my system - does anyone know a way to speed up the rebuild on this system?

thanks

-J
enderlabs
Getting the hang of things
Posts: 55
Joined: Sun Apr 12, 2009 10:36 pm
Location: Bucharest
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by enderlabs »

tt55du wrote:Don,
Just tried this tip and again, thanks to to one of your posts I've found something useful.

However, what do you think the fastest I can push the Qnap on RAID expansion, I'm at the 50MB limit as suggested in your post. My CPU jumped from 14% to about 33% and my expansion is moving much quicker than before but still slow overall.

I'm adding 2 1TB 7200RPM WD Cavier Blacks to the 3 I have in there now. I figure the bottleneck would be the HD rather than the CPU here, I wonder how much I can safely increase the min.
Can you add step by step instructions for QNAP ?
I can not manage to find the files in order to change the values.
Qnap TS-670 Pro
tt55du
Starting out
Posts: 24
Joined: Wed Jan 14, 2009 1:02 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by tt55du »

You should not have to change any files. I have a 639, not sure what you have
After Inserting the HD's you wish to expand, Initiate the RAID expansion through the Web GUI.
Once that is complete, ssh into the qnap and run this command at the prompt

echo 50000 >/proc/sys/dev/raid/speed_limit_min

I found 50MB to be still slow, but ok, it took like a whole day.
You can probably increase it to 70000 or higher without too much issue (depending on CPU and how much you intend to access the NAS during expansion.

Also, I would update the Firmware before you expand. the 639 had a false error during rebuild taht was fixed in a later firmware.

Good Luck
Locked

Return to “Turbo Station Installation & Setup”