[HOWTO] How to increase raid rebuild speed

Discussion on setting up QNAP NAS products.
Locked
tom4moore
Starting out
Posts: 22
Joined: Mon Dec 22, 2014 4:46 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by tom4moore »

So I am pleased to report back that my storage configuration (in the above post) completed syncing in just over 10 hours. Amazing stuff... I see that for some people who posted, they report that changing the maximum and minimum speed did not produce any results or change in sync speed. - I played around with the maximum speed and can confidently report back that the system responded according the my input. Speeding up the process as I adjusted the max speed threshold. Probably saved me having to wait around for another day or two judging from what Ive seen in this thread. Good stuff!

TS-451 with FW/ 4.1.3 0217 and 3x HGST HDN724040ALE640 4TB 7200RPM NAS drives in a RAID5 configuration.
tletourneau
Starting out
Posts: 40
Joined: Sun May 24, 2015 12:05 pm
Location: SE MN, USA
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by tletourneau »

I'm glad I found this, thanks for posting! It decreased my rebuild from 1161 minutes remaining to 228 minutes, I only wish I had found it last night when I started, it would probably be done by now. Set the max to 500000 and the min to 100000 with negligible CPU impact. I can afford to have the disk IO eaten up by the rebuild to get it done sooner. I just need to remember to set it back when I am done. My units defaults were max=200000 and min=5000.
Thanks,
Tom
---------------------
NAS : TVS-871-i7-4790S (upgraded from PT) - 16GB RAM
Firmware: 4.4.3.1439
8x Toshiba 4TB HDWQ140XZSTA - 1 Storage Pool - RAID-5; QM2-2P with 2x Samsung 250GB EVO 960 NVMe automatic SSD caching
NAS : TS-963X - 16GB RAM
Firmware: 4.4.3.1439
5x Seagate 3TB ST3000DM001 - 1 Storage Pool - RAID-5, RAID-5; 4x Samsung 120GB SSD automatic SSD caching
firej
Starting out
Posts: 40
Joined: Sat Nov 14, 2015 5:56 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by firej »

Can I ask something? If it is safe why dont you put this configuration on update firmware so make us all happy without changing it manually?
QNAP TS-453 Pro with 2GB Ram
2x HGST HDN724030ALE640 2.73TB
(RAID1) - Static Single Volume 2.65TB
FW: QTS 4.3.3.0404
User avatar
Don
Guru
Posts: 12289
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: [HOWTO] How to increase raid rebuild speed

Post by Don »

You are asking the wrong audience. This is just a community forum and not QNAP support. If I had to guess I would say that it has to do with NAS availability. The higher the rebuild rate the less responsive the NAS will be. The best solution would be to offer an option to change the rebuild rate either through a slider or a high, medium, and low option. Not everyone wants the same rebuild. Some favor getting rebuilt as quickly as possible at the expense of NAS responsiveness while others favor keeping the NAS responsive and letting the raid rebuild take longer. There should not be a one value fits all approach. Until QNAP offers the option to adjust the rebuild speed, if they ever do, you have the option of doing it via the command line. If you always want the same value then you can add the commands to the autorun file. If you would like to see this added as an option then open a ticket with QNAP support.
Use the forum search feature before posting.

Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
firej
Starting out
Posts: 40
Joined: Sat Nov 14, 2015 5:56 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by firej »

I would totally agree with you, you have right. What does bytes per inode? It was set default to 16384. Maybe this decrease the speed? The Max size of volume? How can be 63,99TB when I have only 2x3TB HGS disks?
QNAP TS-453 Pro with 2GB Ram
2x HGST HDN724030ALE640 2.73TB
(RAID1) - Static Single Volume 2.65TB
FW: QTS 4.3.3.0404
knowles13088
Starting out
Posts: 15
Joined: Mon May 06, 2013 5:11 am

Re: [HOWTO] How to increase raid rebuild speed

Post by knowles13088 »

For a TS-469 L, does anybody have suggestions for a top end number for speed_limit_min? This system has no real user interactive I/O at the moment, so I can pretty much monopolize the system in the rebuild. I am in the midst of replacing drives one by one with larger drives to increase capacity. On drive # 3 of a RAID 10 array . Found this thread last night while working on Drive 1 and upped to 150000. Drive 2 went pretty quickly (about 90 minutes), but drive 3 is about 17 % done after 2 hours.

All I have seen in this thread is about speed_limit_min. I have left speed_limit_max alone.

Thanks!
Levo
Starting out
Posts: 27
Joined: Mon Aug 24, 2009 10:13 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by Levo »

Totally agree with the comments against the limiting all migrations to multi-day slow speeds to preserve productive capacity.
I currently have my old 639 and am trying to setup the new TS-471 as quickly as possible so I can move my data across and am having to wait.
Specifically at this point I am migrating a WD RED 6TB RAID1 to 2+1 RAID5 and I wanted to share my findings:
- I have played with the speed_limit_min/max parameters and whatever I did, I couldn't get the rebuild speed above ~25MB/s while the progress was below 50%.
- Once I got above 50%, it seems to have settled down to around ~106MB/s, cutting down the finish time by 75%+, i.e. instead of 30+ hours down to 7 or so.
Hope this helps others.
- Update: at 70% progress, rate down to ~94MB/s.

P.S. I am vague on this but I thought in the past I saw the new volume size at half way point but the migration is at 59% and still showing the old size. When could I expect to see the new size?
TVS-471 16GB - 5.0.0 - 3x WD Red 6TB + WD 10TB
TS-639 Pro - 4.2.6 - 3x WD Red 4TB + WD 10TB
kwslark
Starting out
Posts: 12
Joined: Tue Nov 17, 2015 11:05 pm
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by kwslark »

User error. That's all, folks.
Last edited by kwslark on Fri Jan 15, 2016 6:35 pm, edited 1 time in total.
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R »

kwslark wrote:I logged in to our TVS-1271U-RP-i7-32G via PuTTY using SSH, this command isn't working...
You need to define "isn't working".

Not possible to change, error messages, no effect or something else?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
seraax
New here
Posts: 2
Joined: Sun Jan 17, 2016 4:41 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by seraax »

Hello everyone,

I am reshaping 453mini (RAM 2Gb - 4x HGST IDK 3TB 7200 RPM) from RAID5 (3 disks) to RAID6 (4 disks) - and the speed is extremely slow (around 5000K/sec).
I've tried everything from the post and it doesn’t change the situation at all, CPU load doesn't go over 6-10% and the speed 5300K/sec at max!

Here is what I’ve tried:

Code: Select all

echo 800000 >/proc/sys/dev/raid/speed_limit_max
echo 400000 >/proc/sys/dev/raid/speed_limit_min
I gradually started setting up a speed_limit_min to 50000 as suggested and then tried even 400000 with no impact on CPU or the speed at all.

Also tried to change cache, but no luck:

Code: Select all

cat: /sys/block/md0/md/stripe_cache_size
returns: No such file or directory, why is that?

Code: Select all

cat /proc/mdstat
returns:
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md1 : active raid6 sde3[5] sdc3[2] sda3[3] sdb3[4]
5840623240 blocks super 1.0 level 6, 4k chunk, algorithm 18 [4/3] [UUU_]
[>....................] reshape = 2.9% (86089728/2920311620) finish=9334.9min speed=5059K/sec

md256 : active raid1 sde2[3](S) sda2[2](S) sdb2[1] sdc2[0]
530112 blocks super 1.0 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

md13 : active raid1 sde4[24] sdc4[25] sdb4[27] sda4[26]
458880 blocks super 1.0 [24/4] [UUUU____________________]
bitmap: 1/1 pages [4KB], 65536KB chunk

md9 : active raid1 sde1[24] sdc1[25] sdb1[27] sda1[26]
530048 blocks super 1.0 [24/4] [UUUU____________________]
bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>

Code: Select all

egrep speed /proc/mdstat
returns: (progress after 17 hours)
[==>..................] reshape = 11.1% (324481024/2920311620) finish=8500.6min speed=5088K/sec


What do I do wrong? Can anyone help please?

Thank you
seraax
New here
Posts: 2
Joined: Sun Jan 17, 2016 4:41 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by seraax »

I've been reshaping RAID5 to RAID6 for over 3 days now and it's only on 44%... I've played with the cache size as well, changed it to 32768, 16384 and 8192, nothing does help.
Does anyone have any suggestions what I can do to make it faster than 5300K/sec please?
Ifti
Getting the hang of things
Posts: 82
Joined: Tue Mar 22, 2011 4:38 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by Ifti »

Same here - adjusted mine to 200000 and my CPU doesn't go over 2%, so I can only assume its not working!!
I have a TVS-871T (i7 model)

I get this:

[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R »

Ifti wrote:Same here - adjusted mine to 200000 and my CPU doesn't go over 2%, so I can only assume its not working!!
My assumption would instead be that your very fast CPU can easily manage this task already without changing this setting and that your disks are the performance bottlenecks.
I have a TVS-871T (i7 model)
This thread was started in a time when the fastest NAS CPUs had someting like 1/15th or 1/20th of the processing capacity your CPU have.
[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec
Looks decent to me. What's wrong with it?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
Ifti
Getting the hang of things
Posts: 82
Joined: Tue Mar 22, 2011 4:38 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by Ifti »

P3R wrote:
Ifti wrote:Same here - adjusted mine to 200000 and my CPU doesn't go over 2%, so I can only assume its not working!!
My assumption would instead be that your very fast CPU can easily manage this task already without changing this setting and that your disks are the performance bottlenecks.
I have a TVS-871T (i7 model)
This thread was started in a time when the fastest NAS CPUs had someting like 1/15th or 1/20th of the processing capacity your CPU have.
[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec
Looks decent to me. What's wrong with it?

Thanks
3x6TB disks took around 14 hours in the end!
All set - until I add another disk! lol
Rexxy
First post
Posts: 1
Joined: Wed Apr 27, 2016 3:59 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by Rexxy »

Works nicely, I used this last year on a TS431 and cut down a Raid5 from 4 days to 2.5days.

I am currently doing another Raid5 4 x 4TB on a TVS-663 and after changing the stats it immediately increased..


md1 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[0]
11691190848 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[>....................] resync = 2.2% (85832136/3897063616) finish=329.8min speed=192562K/sec

CPU stats look ok too

PID USER STATUS RSS PPID %CPU %MEM COMMAND
26707 admin SW 0 2 13.8 0.0 md1_raid5
27258 admin DWN 0 2 9.1 0.0 md1_resync
Locked

Return to “Turbo Station Installation & Setup”