[HOWTO] How to increase raid rebuild speed

Discussion on setting up QNAP NAS products.
Locked
trueimage
Getting the hang of things
Posts: 71
Joined: Sun May 16, 2010 5:11 am

Re: [HOWTO] How to increase raid rebuild speed

Post by trueimage » Thu Feb 09, 2012 9:12 am

cat /proc/sys/dev/raid/speed_limit_max
200000
cat /proc/sys/dev/raid/speed_limit_min
5000

cat /proc/mdstat shows:

md0 : active raid5 sda3[4] sdd3[3] sdc3[2] sdb3[1]
5855836800 blocks level 5, 64k chunk, algorithm 2 [4/3] [_UUU]
[>....................] recovery = 2.4% (48365644/1951945600) finish=68209.6min speed=465K/sec


why so slow? 68,210 minutes is ~48 days????
TS-651 x86 8GB RAM
FW 4.3.3.0188

User avatar
Don
Guru
Posts: 11987
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: [HOWTO] How to increase raid rebuild speed

Post by Don » Thu Feb 09, 2012 11:23 am

Applies to all models.
Read the Online Manuals and use the forum search feature before posting.

It is a recommended to use RAID and have external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

Submit bugs and feature requests to QNAP via their Helpdesk app.

NAS: TVS-882BR | F/W: 4.3.6.0895 | 40GB | 2 x M.2 SATA RAID 1 (System/VMs) | 4 x M.2 NMVe QM2-4P-384A RAID 5 (Cache) | 5 x 4TB HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-663 | F/W: 4.4.1.1086 | 16GB | 2 x M.2 NMVe QM2-2P RAID 1 (Cache) | 4 x 4TB RAID 5
Apps: Boinc, Squid, DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS, Entware, DLstation, +others

P3R
Guru
Posts: 11756
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R » Thu Feb 09, 2012 3:23 pm

trueimage wrote:why so slow? 68,210 minutes is ~48 days????
Because you have an error on the remaining disks. I doubt that you can solve that by tweaking.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

jello_moto
Know my way around
Posts: 105
Joined: Mon Mar 09, 2009 11:14 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by jello_moto » Thu Feb 16, 2012 7:17 am

The rebuilds have always taken so ridiculously long that I've found its just faster to create a new array and copy all the data back over the network. And now that the newer boxes have 10 gigabit, its really an order of magnitude slower to rebuild than copy over the network.

User avatar
schumaku
Guru
Posts: 43663
Joined: Mon Jan 21, 2008 4:41 pm
Location: Kloten (Zurich), Switzerland -- Skype: schumaku
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by schumaku » Mon Apr 02, 2012 3:36 pm

jello_moto wrote:The rebuilds have always taken so ridiculously long that I've found its just faster to create a new array and copy all the data back over the network. And now that the newer boxes have 10 gigabit, its really an order of magnitude slower to rebuild than copy over the network.
The key advantage of a RAID rebuild ie. following a disk replacement is the ongoing availability of the storage without interrupting the service.

User avatar
Briain
Experience counts
Posts: 1733
Joined: Tue Apr 20, 2010 11:56 pm
Location: Edinburgh (Scotland)

Re: [HOWTO] How to increase raid rebuild speed

Post by Briain » Tue Apr 03, 2012 12:51 am

taylortap wrote:I've noticed that not only will this increase the RAID rebuild speed but has also increased my network thoroughput. I set my min at 100000 and everything is hauling ** now!!

Thanks OP for the information.


Hi

The above is rather interesting. Has anyone else verified that by testing to see whether it changes performance under normal usage conditions? I often back one NAS (TS-659 with RAID 6) to another NAS (TS-219P+ with RAID 1). It usually takes only about 15 hours to transfer 1 TB of >30 MB FLAC files (and about 200 GB of smaller files), so I'll maybe try that trick on the TS-219P+ just to see if I can detect any difference.

Bri :)
TS-119, 1 X Seagate ~~ TS-219, 2 X Seagate (R1) ~~ TS-453A, 2 X 3 TB WD Red (R1) ~~ TS-659, 5 X 1 TB Hitachi Enterprise (R6)
APC Smart-UPS 750

Metzlmane
Starting out
Posts: 16
Joined: Tue Apr 24, 2012 7:32 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by Metzlmane » Sun Jul 01, 2012 6:01 pm

Thanks for the information. Now it's running a little faster.

2x Hitachi Ultrastar 7K4000 3000GB, SATA 6Gb/s (HUS724030ALE640/0F14684)
Raid 1

Code: Select all

Min 100000
Max 200000

md0 : active raid1 sdb3[1] sda3[0]
      2928697556 blocks super 1.0 [2/2] [UU]
      [=====>...............]  resync = 29.0% (851113152/2928697556) finish=273.3min speed=126652K/sec

on TS1079
The CPU is bored, but the harddisks can't go faster.

DarkLogix
Getting the hang of things
Posts: 66
Joined: Thu Oct 11, 2012 5:02 am

Re: [HOWTO] How to increase raid rebuild speed

Post by DarkLogix » Fri Oct 19, 2012 2:50 am

taylortap wrote:I've noticed that not only will this increase the RAID rebuild speed but has also increased my network thoroughput. I set my min at 100000 and everything is hauling ** now!!

Thanks OP for the information.

Have you done some benchmarks on this?
and do you think it'd apply to a TS-469-Pro?

User avatar
raffe
Know my way around
Posts: 165
Joined: Thu Nov 08, 2007 8:03 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by raffe » Fri Oct 26, 2012 2:38 am

Thanks for a good tip Don!
Don wrote:...
...

Code: Select all

watch cat /proc/mdstat

There is no watch program on the NAS so that part won't work...
...

To get something similar to "watch", it's possible to use a command line like this (just write (or copy/paste) the whole line and press [Enter]):

Code: Select all

while true; do cat /proc/mdstat; sleep 1; done

End it with Ctrl + C. The number "1" in "sleep 1" makes it wait one second before again running the command "cat /proc/mdstat" in a loop. Change 1 to higher value if you want longer time between updates. Not the same as watch, but anyhow...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ If you are too open-minded, your brain may fall out ]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

User avatar
chrispage
Getting the hang of things
Posts: 62
Joined: Sun May 01, 2011 1:53 am
Location: Melbourne Australia
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by chrispage » Sat Dec 01, 2012 2:04 pm

Hi guys.

I'm going through the painful task of upgrading 6 x 2TB disks with 3TB WD Red disks in my 659 Pro II .. The first disk took approximately 100 hrs and the second one is going almost as slow!

finish=4673.1min speed=5154K/sec

Any ideas?


Thanks

UPDATE

Okay, i used the echo command to set the minimum and its now running at 48000K/sec! AWESOME :)
QNAP TS-653 Pro 8G
QTS: 4.2 Beta
Raid 5 - 6x3TB WD Red)

mannebk
Getting the hang of things
Posts: 64
Joined: Fri Apr 02, 2010 5:54 pm
Location: Stuttgart

Re: [HOWTO] How to increase raid rebuild speed

Post by mannebk » Tue Dec 25, 2012 12:18 am

thx, pushed my 410 to 50 as well, nice tip,

my systemload was already constant over 50% but jumping around and now its a solid 100% for a raid 5 expansion from 3 to 4 500gb hdds.

Manne
Running a TS410 and a TS659pro. Right now (2012/12/15) I would gladly exchange them to any other similar system not produced by QNAP. Most of the QNAP frontends are just so CRAPPY as if a trainee did the development.

fantomas
Experience counts
Posts: 1549
Joined: Mon Feb 07, 2011 5:40 am
Location: Bratislava, Slovakia
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by fantomas » Wed Feb 20, 2013 12:56 am

There's one more option for rebuilding RAID5/RAID6 devices:

# cat /sys/block/md0/md/stripe_cache_size
4096

increasing this could help, last time I have tried 16384 (4x more).
experience with administration of UN*X (mostly linux) and applications on internet servers since 1994...

gPaq
New here
Posts: 3
Joined: Mon Feb 11, 2013 8:28 am

Re: [HOWTO] How to increase raid rebuild speed

Post by gPaq » Sun Mar 03, 2013 11:33 pm

Although this tip is very much appreciated, and all the commands work on my newly purchased TS-469 Pro, making the changes on "speed_limit_min" makes not a lick of difference, unfortunately. I increased this speed gradually from the default 5000 to 50000, up to 150000, and basically nothing happens. The processor utilization does not go up (hovers around 16% before and after changes), and after 11 hours of re-striping I'm at 17%, which puts my ETA at about 4 more days to completion. That's not reasonable anymore... Is there anything that puts a damper on this configuration in newer models? As some other users indicated, the changes seem to have no effect whatsoever. Below are my settings for reference (apparently indicating current throughput at 22868K/sec or almost 23 MB/sec, right?). This is a TS-469 Pro with latest firmware 3.8.1 and 3 GB memory (default 1 GB with 2 GB added), expanding RAID5 from 3 disks to 4 disks, each being a Western Digital 3 GB Red WD30EFRX.

[~] # cat /proc/sys/dev/raid/speed_limit_max
200000
[~] # cat /proc/sys/dev/raid/speed_limit_min
150000
[~] #

[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sdd3[3] sda3[0] sdc3[2] sdb3[1]
5857395072 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[======>..............] reshape = 34.2% (1003124924/2928697536) finish=1403.3min speed=22868K/sec

md4 : active raid1 sdd2[2](S) sdc2[3](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]

md13 : active raid1 sda4[0] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [4/4] [UUUU]
bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [4/4] [UUUU]
bitmap: 0/65 pages [0KB], 4KB chunk

unused devices: <none>
[~] #

User avatar
schumaku
Guru
Posts: 43663
Joined: Mon Jan 21, 2008 4:41 pm
Location: Kloten (Zurich), Switzerland -- Skype: schumaku
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by schumaku » Mon Mar 04, 2013 12:01 am

A reshape (change RAID layout) is much more intensive than resync.

gPaq
New here
Posts: 3
Joined: Mon Feb 11, 2013 8:28 am

Re: [HOWTO] How to increase raid rebuild speed

Post by gPaq » Mon Mar 04, 2013 1:08 am

schumaku wrote:A reshape (change RAID layout) is much more intensive than resync.


Dear schumaku, I believe I understand what you are saying and that may indeed be so, however, from the web GUI I simply selected to Add Hard Drive, going from RAID 5 with 3 disks to RAID 5 with 4 disks, and this is the process that the QNAP is executing. I wish I could change it, or at least make it go faster, especially because there is no data on this volume (other than the default shares): this is a brand new installation and I will copy my data to the NAS at a later time when the RAID configuration is stable. Any further input is appreciated.

Locked

Return to “Turbo Station Installation & Setup”