[HOWTO] How to increase raid rebuild speed
-
- Getting the hang of things
- Posts: 73
- Joined: Sun May 16, 2010 5:11 am
Re: [HOWTO] How to increase raid rebuild speed
cat /proc/sys/dev/raid/speed_limit_max
200000
cat /proc/sys/dev/raid/speed_limit_min
5000
cat /proc/mdstat shows:
md0 : active raid5 sda3[4] sdd3[3] sdc3[2] sdb3[1]
5855836800 blocks level 5, 64k chunk, algorithm 2 [4/3] [_UUU]
[>....................] recovery = 2.4% (48365644/1951945600) finish=68209.6min speed=465K/sec
why so slow? 68,210 minutes is ~48 days????
200000
cat /proc/sys/dev/raid/speed_limit_min
5000
cat /proc/mdstat shows:
md0 : active raid5 sda3[4] sdd3[3] sdc3[2] sdb3[1]
5855836800 blocks level 5, 64k chunk, algorithm 2 [4/3] [_UUU]
[>....................] recovery = 2.4% (48365644/1951945600) finish=68209.6min speed=465K/sec
why so slow? 68,210 minutes is ~48 days????
TS-651 x86 8GB RAM
FW 4.3.3.0188
FW 4.3.3.0188
- Don
- Guru
- Posts: 12289
- Joined: Thu Jan 03, 2008 4:56 am
- Location: Long Island, New York
Re: [HOWTO] How to increase raid rebuild speed
Applies to all models.
Use the forum search feature before posting.
Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.
NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.
NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
-
- Guru
- Posts: 13192
- Joined: Sat Dec 29, 2007 1:39 am
- Location: Stockholm, Sweden (UTC+01:00)
Re: [HOWTO] How to increase raid rebuild speed
Because you have an error on the remaining disks. I doubt that you can solve that by tweaking.trueimage wrote:why so slow? 68,210 minutes is ~48 days????
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
-
- Know my way around
- Posts: 105
- Joined: Mon Mar 09, 2009 11:14 pm
Re: [HOWTO] How to increase raid rebuild speed
The rebuilds have always taken so ridiculously long that I've found its just faster to create a new array and copy all the data back over the network. And now that the newer boxes have 10 gigabit, its really an order of magnitude slower to rebuild than copy over the network.
- schumaku
- Guru
- Posts: 43578
- Joined: Mon Jan 21, 2008 4:41 pm
- Location: Kloten (Zurich), Switzerland -- Skype: schumaku
- Contact:
Re: [HOWTO] How to increase raid rebuild speed
The key advantage of a RAID rebuild ie. following a disk replacement is the ongoing availability of the storage without interrupting the service.jello_moto wrote:The rebuilds have always taken so ridiculously long that I've found its just faster to create a new array and copy all the data back over the network. And now that the newer boxes have 10 gigabit, its really an order of magnitude slower to rebuild than copy over the network.
- Briain
- Experience counts
- Posts: 1749
- Joined: Tue Apr 20, 2010 11:56 pm
- Location: Edinburgh (Scotland)
Re: [HOWTO] How to increase raid rebuild speed
Hitaylortap wrote:I've noticed that not only will this increase the RAID rebuild speed but has also increased my network thoroughput. I set my min at 100000 and everything is hauling ** now!!
Thanks OP for the information.
The above is rather interesting. Has anyone else verified that by testing to see whether it changes performance under normal usage conditions? I often back one NAS (TS-659 with RAID 6) to another NAS (TS-219P+ with RAID 1). It usually takes only about 15 hours to transfer 1 TB of >30 MB FLAC files (and about 200 GB of smaller files), so I'll maybe try that trick on the TS-219P+ just to see if I can detect any difference.
Bri
TS-119, 1 X Seagate ~~ TS-219, 2 X Seagate (R1) ~~ TS-453A, 2 X 3 TB WD Red (R1) ~~ TS-659, 5 X 1 TB Hitachi Enterprise (R6)
APC Smart-UPS 750
APC Smart-UPS 750
-
- Starting out
- Posts: 16
- Joined: Tue Apr 24, 2012 7:32 pm
Re: [HOWTO] How to increase raid rebuild speed
Thanks for the information. Now it's running a little faster.
2x Hitachi Ultrastar 7K4000 3000GB, SATA 6Gb/s (HUS724030ALE640/0F14684)
Raid 1
on TS1079
The CPU is bored, but the harddisks can't go faster.
2x Hitachi Ultrastar 7K4000 3000GB, SATA 6Gb/s (HUS724030ALE640/0F14684)
Raid 1
Code: Select all
Min 100000
Max 200000
md0 : active raid1 sdb3[1] sda3[0]
2928697556 blocks super 1.0 [2/2] [UU]
[=====>...............] resync = 29.0% (851113152/2928697556) finish=273.3min speed=126652K/sec
The CPU is bored, but the harddisks can't go faster.
-
- Getting the hang of things
- Posts: 66
- Joined: Thu Oct 11, 2012 5:02 am
Re: [HOWTO] How to increase raid rebuild speed
Have you done some benchmarks on this?taylortap wrote:I've noticed that not only will this increase the RAID rebuild speed but has also increased my network thoroughput. I set my min at 100000 and everything is hauling ** now!!
Thanks OP for the information.
and do you think it'd apply to a TS-469-Pro?
- raffe
- Know my way around
- Posts: 166
- Joined: Thu Nov 08, 2007 8:03 pm
Re: [HOWTO] How to increase raid rebuild speed
Thanks for a good tip Don!
End it with Ctrl + C. The number "1" in "sleep 1" makes it wait one second before again running the command "cat /proc/mdstat" in a loop. Change 1 to higher value if you want longer time between updates. Not the same as watch, but anyhow...
To get something similar to "watch", it's possible to use a command line like this (just write (or copy/paste) the whole line and press [Enter]):Don wrote:...
...There is no watch program on the NAS so that part won't work...Code: Select all
watch cat /proc/mdstat
...
Code: Select all
while true; do cat /proc/mdstat; sleep 1; done
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ If you are too open-minded, your brain may fall out ]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ If you are too open-minded, your brain may fall out ]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- chrispage
- Getting the hang of things
- Posts: 62
- Joined: Sun May 01, 2011 1:53 am
- Location: Melbourne Australia
- Contact:
Re: [HOWTO] How to increase raid rebuild speed
Hi guys.
I'm going through the painful task of upgrading 6 x 2TB disks with 3TB WD Red disks in my 659 Pro II .. The first disk took approximately 100 hrs and the second one is going almost as slow!
finish=4673.1min speed=5154K/sec
Any ideas?
Thanks
UPDATE
Okay, i used the echo command to set the minimum and its now running at 48000K/sec! AWESOME
I'm going through the painful task of upgrading 6 x 2TB disks with 3TB WD Red disks in my 659 Pro II .. The first disk took approximately 100 hrs and the second one is going almost as slow!
finish=4673.1min speed=5154K/sec
Any ideas?
Thanks
UPDATE
Okay, i used the echo command to set the minimum and its now running at 48000K/sec! AWESOME
QNAP TS-653 Pro 8G
QTS: 4.2 Beta
Raid 5 - 6x3TB WD Red)
QTS: 4.2 Beta
Raid 5 - 6x3TB WD Red)
-
- Getting the hang of things
- Posts: 64
- Joined: Fri Apr 02, 2010 5:54 pm
- Location: Stuttgart
Re: [HOWTO] How to increase raid rebuild speed
thx, pushed my 410 to 50 as well, nice tip,
my systemload was already constant over 50% but jumping around and now its a solid 100% for a raid 5 expansion from 3 to 4 500gb hdds.
Manne
my systemload was already constant over 50% but jumping around and now its a solid 100% for a raid 5 expansion from 3 to 4 500gb hdds.
Manne
Running a TS410 and a TS659pro. Right now (2012/12/15) I would gladly exchange them to any other similar system not produced by QNAP. Most of the QNAP frontends are just so CRAPPY as if a trainee did the development.
-
- Experience counts
- Posts: 1560
- Joined: Mon Feb 07, 2011 5:40 am
- Location: Bratislava, Slovakia
- Contact:
Re: [HOWTO] How to increase raid rebuild speed
There's one more option for rebuilding RAID5/RAID6 devices:
# cat /sys/block/md0/md/stripe_cache_size
4096
increasing this could help, last time I have tried 16384 (4x more).
# cat /sys/block/md0/md/stripe_cache_size
4096
increasing this could help, last time I have tried 16384 (4x more).
experience with administration of UN*X (mostly linux) and applications on internet servers since 1994...
-
- New here
- Posts: 7
- Joined: Mon Feb 11, 2013 8:28 am
Re: [HOWTO] How to increase raid rebuild speed
Although this tip is very much appreciated, and all the commands work on my newly purchased TS-469 Pro, making the changes on "speed_limit_min" makes not a lick of difference, unfortunately. I increased this speed gradually from the default 5000 to 50000, up to 150000, and basically nothing happens. The processor utilization does not go up (hovers around 16% before and after changes), and after 11 hours of re-striping I'm at 17%, which puts my ETA at about 4 more days to completion. That's not reasonable anymore... Is there anything that puts a damper on this configuration in newer models? As some other users indicated, the changes seem to have no effect whatsoever. Below are my settings for reference (apparently indicating current throughput at 22868K/sec or almost 23 MB/sec, right?). This is a TS-469 Pro with latest firmware 3.8.1 and 3 GB memory (default 1 GB with 2 GB added), expanding RAID5 from 3 disks to 4 disks, each being a Western Digital 3 GB Red WD30EFRX.
[~] # cat /proc/sys/dev/raid/speed_limit_max
200000
[~] # cat /proc/sys/dev/raid/speed_limit_min
150000
[~] #
[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sdd3[3] sda3[0] sdc3[2] sdb3[1]
5857395072 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[======>..............] reshape = 34.2% (1003124924/2928697536) finish=1403.3min speed=22868K/sec
md4 : active raid1 sdd2[2](S) sdc2[3](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]
md13 : active raid1 sda4[0] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [4/4] [UUUU]
bitmap: 0/57 pages [0KB], 4KB chunk
md9 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [4/4] [UUUU]
bitmap: 0/65 pages [0KB], 4KB chunk
unused devices: <none>
[~] #
[~] # cat /proc/sys/dev/raid/speed_limit_max
200000
[~] # cat /proc/sys/dev/raid/speed_limit_min
150000
[~] #
[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sdd3[3] sda3[0] sdc3[2] sdb3[1]
5857395072 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[======>..............] reshape = 34.2% (1003124924/2928697536) finish=1403.3min speed=22868K/sec
md4 : active raid1 sdd2[2](S) sdc2[3](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]
md13 : active raid1 sda4[0] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [4/4] [UUUU]
bitmap: 0/57 pages [0KB], 4KB chunk
md9 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [4/4] [UUUU]
bitmap: 0/65 pages [0KB], 4KB chunk
unused devices: <none>
[~] #
- schumaku
- Guru
- Posts: 43578
- Joined: Mon Jan 21, 2008 4:41 pm
- Location: Kloten (Zurich), Switzerland -- Skype: schumaku
- Contact:
Re: [HOWTO] How to increase raid rebuild speed
A reshape (change RAID layout) is much more intensive than resync.
-
- New here
- Posts: 7
- Joined: Mon Feb 11, 2013 8:28 am
Re: [HOWTO] How to increase raid rebuild speed
Dear schumaku, I believe I understand what you are saying and that may indeed be so, however, from the web GUI I simply selected to Add Hard Drive, going from RAID 5 with 3 disks to RAID 5 with 4 disks, and this is the process that the QNAP is executing. I wish I could change it, or at least make it go faster, especially because there is no data on this volume (other than the default shares): this is a brand new installation and I will copy my data to the NAS at a later time when the RAID configuration is stable. Any further input is appreciated.schumaku wrote:A reshape (change RAID layout) is much more intensive than resync.