[HOWTO] How to increase raid rebuild speed

Discussion on setting up QNAP NAS products.
Locked
trueimage
Getting the hang of things
Posts: 73
Joined: Sun May 16, 2010 5:11 am

Re: [HOWTO] How to increase raid rebuild speed

Post by trueimage »

cat /proc/sys/dev/raid/speed_limit_max
200000
cat /proc/sys/dev/raid/speed_limit_min
5000

cat /proc/mdstat shows:

md0 : active raid5 sda3[4] sdd3[3] sdc3[2] sdb3[1]
5855836800 blocks level 5, 64k chunk, algorithm 2 [4/3] [_UUU]
[>....................] recovery = 2.4% (48365644/1951945600) finish=68209.6min speed=465K/sec


why so slow? 68,210 minutes is ~48 days????
TS-651 x86 8GB RAM
FW 4.3.3.0188
User avatar
Don
Guru
Posts: 12289
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: [HOWTO] How to increase raid rebuild speed

Post by Don »

Applies to all models.
Use the forum search feature before posting.

Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R »

trueimage wrote:why so slow? 68,210 minutes is ~48 days????
Because you have an error on the remaining disks. I doubt that you can solve that by tweaking.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
jello_moto
Know my way around
Posts: 105
Joined: Mon Mar 09, 2009 11:14 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by jello_moto »

The rebuilds have always taken so ridiculously long that I've found its just faster to create a new array and copy all the data back over the network. And now that the newer boxes have 10 gigabit, its really an order of magnitude slower to rebuild than copy over the network.
User avatar
schumaku
Guru
Posts: 43579
Joined: Mon Jan 21, 2008 4:41 pm
Location: Kloten (Zurich), Switzerland -- Skype: schumaku
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by schumaku »

jello_moto wrote:The rebuilds have always taken so ridiculously long that I've found its just faster to create a new array and copy all the data back over the network. And now that the newer boxes have 10 gigabit, its really an order of magnitude slower to rebuild than copy over the network.
The key advantage of a RAID rebuild ie. following a disk replacement is the ongoing availability of the storage without interrupting the service.
User avatar
Briain
Experience counts
Posts: 1749
Joined: Tue Apr 20, 2010 11:56 pm
Location: Edinburgh (Scotland)

Re: [HOWTO] How to increase raid rebuild speed

Post by Briain »

taylortap wrote:I've noticed that not only will this increase the RAID rebuild speed but has also increased my network thoroughput. I set my min at 100000 and everything is hauling ** now!!

Thanks OP for the information.
Hi

The above is rather interesting. Has anyone else verified that by testing to see whether it changes performance under normal usage conditions? I often back one NAS (TS-659 with RAID 6) to another NAS (TS-219P+ with RAID 1). It usually takes only about 15 hours to transfer 1 TB of >30 MB FLAC files (and about 200 GB of smaller files), so I'll maybe try that trick on the TS-219P+ just to see if I can detect any difference.

Bri :)
TS-119, 1 X Seagate ~~ TS-219, 2 X Seagate (R1) ~~ TS-453A, 2 X 3 TB WD Red (R1) ~~ TS-659, 5 X 1 TB Hitachi Enterprise (R6)
APC Smart-UPS 750
Metzlmane
Starting out
Posts: 16
Joined: Tue Apr 24, 2012 7:32 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by Metzlmane »

Thanks for the information. Now it's running a little faster.

2x Hitachi Ultrastar 7K4000 3000GB, SATA 6Gb/s (HUS724030ALE640/0F14684)
Raid 1

Code: Select all

Min 100000
Max 200000

md0 : active raid1 sdb3[1] sda3[0]
      2928697556 blocks super 1.0 [2/2] [UU]
      [=====>...............]  resync = 29.0% (851113152/2928697556) finish=273.3min speed=126652K/sec
on TS1079
The CPU is bored, but the harddisks can't go faster.
DarkLogix
Getting the hang of things
Posts: 66
Joined: Thu Oct 11, 2012 5:02 am

Re: [HOWTO] How to increase raid rebuild speed

Post by DarkLogix »

taylortap wrote:I've noticed that not only will this increase the RAID rebuild speed but has also increased my network thoroughput. I set my min at 100000 and everything is hauling ** now!!

Thanks OP for the information.
Have you done some benchmarks on this?
and do you think it'd apply to a TS-469-Pro?
User avatar
raffe
Know my way around
Posts: 166
Joined: Thu Nov 08, 2007 8:03 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by raffe »

Thanks for a good tip Don!
Don wrote:...
...

Code: Select all

watch cat /proc/mdstat
There is no watch program on the NAS so that part won't work...
...
To get something similar to "watch", it's possible to use a command line like this (just write (or copy/paste) the whole line and press [Enter]):

Code: Select all

while true; do cat /proc/mdstat; sleep 1; done
End it with Ctrl + C. The number "1" in "sleep 1" makes it wait one second before again running the command "cat /proc/mdstat" in a loop. Change 1 to higher value if you want longer time between updates. Not the same as watch, but anyhow...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ If you are too open-minded, your brain may fall out ]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
User avatar
chrispage
Getting the hang of things
Posts: 62
Joined: Sun May 01, 2011 1:53 am
Location: Melbourne Australia
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by chrispage »

Hi guys.

I'm going through the painful task of upgrading 6 x 2TB disks with 3TB WD Red disks in my 659 Pro II .. The first disk took approximately 100 hrs and the second one is going almost as slow!

finish=4673.1min speed=5154K/sec

Any ideas?


Thanks

UPDATE

Okay, i used the echo command to set the minimum and its now running at 48000K/sec! AWESOME :)
QNAP TS-653 Pro 8G
QTS: 4.2 Beta
Raid 5 - 6x3TB WD Red)
mannebk
Getting the hang of things
Posts: 64
Joined: Fri Apr 02, 2010 5:54 pm
Location: Stuttgart

Re: [HOWTO] How to increase raid rebuild speed

Post by mannebk »

thx, pushed my 410 to 50 as well, nice tip,

my systemload was already constant over 50% but jumping around and now its a solid 100% for a raid 5 expansion from 3 to 4 500gb hdds.

Manne
Running a TS410 and a TS659pro. Right now (2012/12/15) I would gladly exchange them to any other similar system not produced by QNAP. Most of the QNAP frontends are just so CRAPPY as if a trainee did the development.
fantomas
Experience counts
Posts: 1560
Joined: Mon Feb 07, 2011 5:40 am
Location: Bratislava, Slovakia
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by fantomas »

There's one more option for rebuilding RAID5/RAID6 devices:

# cat /sys/block/md0/md/stripe_cache_size
4096

increasing this could help, last time I have tried 16384 (4x more).
experience with administration of UN*X (mostly linux) and applications on internet servers since 1994...
gPaq
New here
Posts: 7
Joined: Mon Feb 11, 2013 8:28 am

Re: [HOWTO] How to increase raid rebuild speed

Post by gPaq »

Although this tip is very much appreciated, and all the commands work on my newly purchased TS-469 Pro, making the changes on "speed_limit_min" makes not a lick of difference, unfortunately. I increased this speed gradually from the default 5000 to 50000, up to 150000, and basically nothing happens. The processor utilization does not go up (hovers around 16% before and after changes), and after 11 hours of re-striping I'm at 17%, which puts my ETA at about 4 more days to completion. That's not reasonable anymore... Is there anything that puts a damper on this configuration in newer models? As some other users indicated, the changes seem to have no effect whatsoever. Below are my settings for reference (apparently indicating current throughput at 22868K/sec or almost 23 MB/sec, right?). This is a TS-469 Pro with latest firmware 3.8.1 and 3 GB memory (default 1 GB with 2 GB added), expanding RAID5 from 3 disks to 4 disks, each being a Western Digital 3 GB Red WD30EFRX.

[~] # cat /proc/sys/dev/raid/speed_limit_max
200000
[~] # cat /proc/sys/dev/raid/speed_limit_min
150000
[~] #

[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sdd3[3] sda3[0] sdc3[2] sdb3[1]
5857395072 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[======>..............] reshape = 34.2% (1003124924/2928697536) finish=1403.3min speed=22868K/sec

md4 : active raid1 sdd2[2](S) sdc2[3](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]

md13 : active raid1 sda4[0] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [4/4] [UUUU]
bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [4/4] [UUUU]
bitmap: 0/65 pages [0KB], 4KB chunk

unused devices: <none>
[~] #
User avatar
schumaku
Guru
Posts: 43579
Joined: Mon Jan 21, 2008 4:41 pm
Location: Kloten (Zurich), Switzerland -- Skype: schumaku
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by schumaku »

A reshape (change RAID layout) is much more intensive than resync.
gPaq
New here
Posts: 7
Joined: Mon Feb 11, 2013 8:28 am

Re: [HOWTO] How to increase raid rebuild speed

Post by gPaq »

schumaku wrote:A reshape (change RAID layout) is much more intensive than resync.
Dear schumaku, I believe I understand what you are saying and that may indeed be so, however, from the web GUI I simply selected to Add Hard Drive, going from RAID 5 with 3 disks to RAID 5 with 4 disks, and this is the process that the QNAP is executing. I wish I could change it, or at least make it go faster, especially because there is no data on this volume (other than the default shares): this is a brand new installation and I will copy my data to the NAS at a later time when the RAID configuration is stable. Any further input is appreciated.
Locked

Return to “Turbo Station Installation & Setup”