[HOWTO] How to increase raid rebuild speed

Discussion on setting up QNAP NAS products.
Locked
tom4moore
Starting out
Posts: 22
Joined: Mon Dec 22, 2014 4:46 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by tom4moore » Wed Mar 04, 2015 7:18 am

So I am pleased to report back that my storage configuration (in the above post) completed syncing in just over 10 hours. Amazing stuff... I see that for some people who posted, they report that changing the maximum and minimum speed did not produce any results or change in sync speed. - I played around with the maximum speed and can confidently report back that the system responded according the my input. Speeding up the process as I adjusted the max speed threshold. Probably saved me having to wait around for another day or two judging from what Ive seen in this thread. Good stuff!

TS-451 with FW/ 4.1.3 0217 and 3x HGST HDN724040ALE640 4TB 7200RPM NAS drives in a RAID5 configuration.

tletourneau
Starting out
Posts: 39
Joined: Sun May 24, 2015 12:05 pm
Location: SE MN, USA
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by tletourneau » Fri Nov 13, 2015 3:25 am

I'm glad I found this, thanks for posting! It decreased my rebuild from 1161 minutes remaining to 228 minutes, I only wish I had found it last night when I started, it would probably be done by now. Set the max to 500000 and the min to 100000 with negligible CPU impact. I can afford to have the disk IO eaten up by the rebuild to get it done sooner. I just need to remember to set it back when I am done. My units defaults were max=200000 and min=5000.
Thanks,
Tom
---------------------
NAS : TVS-871-i7-4790S (upgraded from PT) - 16GB RAM
Firmware: 4.3.4.0411 Build 20171208
8x Toshiba 4TB HDWQ140XZSTA - 1 Storage Pool, RAID-5; QM2-2P with 2x Samsung 250GB EVO 960 NVMe, 1 Storage Pool (Automatic QTier 2.0) with RAID-1

firej
Starting out
Posts: 40
Joined: Sat Nov 14, 2015 5:56 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by firej » Sun Nov 22, 2015 5:52 am

Can I ask something? If it is safe why dont you put this configuration on update firmware so make us all happy without changing it manually?
QNAP TS-453 Pro with 2GB Ram
2x HGST HDN724030ALE640 2.73TB
(RAID1) - Static Single Volume 2.65TB
FW: QTS 4.3.3.0404

User avatar
Don
Guru
Posts: 11970
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by Don » Sun Nov 22, 2015 8:35 am

You are asking the wrong audience. This is just a community forum and not QNAP support. If I had to guess I would say that it has to do with NAS availability. The higher the rebuild rate the less responsive the NAS will be. The best solution would be to offer an option to change the rebuild rate either through a slider or a high, medium, and low option. Not everyone wants the same rebuild. Some favor getting rebuilt as quickly as possible at the expense of NAS responsiveness while others favor keeping the NAS responsive and letting the raid rebuild take longer. There should not be a one value fits all approach. Until QNAP offers the option to adjust the rebuild speed, if they ever do, you have the option of doing it via the command line. If you always want the same value then you can add the commands to the autorun file. If you would like to see this added as an option then open a ticket with QNAP support.
Read the Online Manuals and use the forum search feature before posting.

It is a recommended to use RAID and have external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

Submit bugs and feature requests to QNAP via their Helpdesk app.

NAS: TVS-882BR | F/W: 4.3.6.0895 | 40GB | 2 x M.2 SATA RAID 1 (System/VMs) | 4 x M.2 NMVe QM2-4P-384A RAID 5 (Cache) | 5 x 4TB HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-663 | F/W: 4.4.1.1086 | 16GB | 2 x M.2 NMVe QM2-2P RAID 1 (Cache) | 4 x 4TB RAID 5
Apps: Boinc, Squid, DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS, Entware, DLstation, +others

firej
Starting out
Posts: 40
Joined: Sat Nov 14, 2015 5:56 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by firej » Sun Nov 22, 2015 4:52 pm

I would totally agree with you, you have right. What does bytes per inode? It was set default to 16384. Maybe this decrease the speed? The Max size of volume? How can be 63,99TB when I have only 2x3TB HGS disks?
QNAP TS-453 Pro with 2GB Ram
2x HGST HDN724030ALE640 2.73TB
(RAID1) - Static Single Volume 2.65TB
FW: QTS 4.3.3.0404

knowles13088
Starting out
Posts: 15
Joined: Mon May 06, 2013 5:11 am

Re: [HOWTO] How to increase raid rebuild speed

Post by knowles13088 » Wed Dec 16, 2015 4:41 am

For a TS-469 L, does anybody have suggestions for a top end number for speed_limit_min? This system has no real user interactive I/O at the moment, so I can pretty much monopolize the system in the rebuild. I am in the midst of replacing drives one by one with larger drives to increase capacity. On drive # 3 of a RAID 10 array . Found this thread last night while working on Drive 1 and upped to 150000. Drive 2 went pretty quickly (about 90 minutes), but drive 3 is about 17 % done after 2 hours.

All I have seen in this thread is about speed_limit_min. I have left speed_limit_max alone.

Thanks!

Levo
Starting out
Posts: 23
Joined: Mon Aug 24, 2009 10:13 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by Levo » Thu Dec 24, 2015 6:28 pm

Totally agree with the comments against the limiting all migrations to multi-day slow speeds to preserve productive capacity.
I currently have my old 639 and am trying to setup the new TS-471 as quickly as possible so I can move my data across and am having to wait.
Specifically at this point I am migrating a WD RED 6TB RAID1 to 2+1 RAID5 and I wanted to share my findings:
- I have played with the speed_limit_min/max parameters and whatever I did, I couldn't get the rebuild speed above ~25MB/s while the progress was below 50%.
- Once I got above 50%, it seems to have settled down to around ~106MB/s, cutting down the finish time by 75%+, i.e. instead of 30+ hours down to 7 or so.
Hope this helps others.
- Update: at 70% progress, rate down to ~94MB/s.

P.S. I am vague on this but I thought in the past I saw the new volume size at half way point but the migration is at 59% and still showing the old size. When could I expect to see the new size?
TVS-471 16GB - 4.3.4 - 3x WD Red 6TB + WD 10TB
TS-639 Pro - 4.2 - 3x WD Red 4TB

kwslark
Starting out
Posts: 12
Joined: Tue Nov 17, 2015 11:05 pm
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by kwslark » Fri Jan 08, 2016 11:45 pm

User error. That's all, folks.
Last edited by kwslark on Fri Jan 15, 2016 6:35 pm, edited 1 time in total.

P3R
Guru
Posts: 10981
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R » Sat Jan 09, 2016 4:40 am

kwslark wrote:I logged in to our TVS-1271U-RP-i7-32G via PuTTY using SSH, this command isn't working...
You need to define "isn't working".

Not possible to change, error messages, no effect or something else?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

seraax
New here
Posts: 2
Joined: Sun Jan 17, 2016 4:41 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by seraax » Sun Jan 17, 2016 5:34 pm

Hello everyone,

I am reshaping 453mini (RAM 2Gb - 4x HGST IDK 3TB 7200 RPM) from RAID5 (3 disks) to RAID6 (4 disks) - and the speed is extremely slow (around 5000K/sec).
I've tried everything from the post and it doesn’t change the situation at all, CPU load doesn't go over 6-10% and the speed 5300K/sec at max!

Here is what I’ve tried:

Code: Select all

echo 800000 >/proc/sys/dev/raid/speed_limit_max
echo 400000 >/proc/sys/dev/raid/speed_limit_min

I gradually started setting up a speed_limit_min to 50000 as suggested and then tried even 400000 with no impact on CPU or the speed at all.

Also tried to change cache, but no luck:

Code: Select all

cat: /sys/block/md0/md/stripe_cache_size

returns: No such file or directory, why is that?


Code: Select all

cat /proc/mdstat

returns:
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md1 : active raid6 sde3[5] sdc3[2] sda3[3] sdb3[4]
5840623240 blocks super 1.0 level 6, 4k chunk, algorithm 18 [4/3] [UUU_]
[>....................] reshape = 2.9% (86089728/2920311620) finish=9334.9min speed=5059K/sec

md256 : active raid1 sde2[3](S) sda2[2](S) sdb2[1] sdc2[0]
530112 blocks super 1.0 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

md13 : active raid1 sde4[24] sdc4[25] sdb4[27] sda4[26]
458880 blocks super 1.0 [24/4] [UUUU____________________]
bitmap: 1/1 pages [4KB], 65536KB chunk

md9 : active raid1 sde1[24] sdc1[25] sdb1[27] sda1[26]
530048 blocks super 1.0 [24/4] [UUUU____________________]
bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>


Code: Select all

egrep speed /proc/mdstat

returns: (progress after 17 hours)
[==>..................] reshape = 11.1% (324481024/2920311620) finish=8500.6min speed=5088K/sec


What do I do wrong? Can anyone help please?

Thank you

seraax
New here
Posts: 2
Joined: Sun Jan 17, 2016 4:41 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by seraax » Tue Jan 19, 2016 10:27 pm

I've been reshaping RAID5 to RAID6 for over 3 days now and it's only on 44%... I've played with the cache size as well, changed it to 32768, 16384 and 8192, nothing does help.
Does anyone have any suggestions what I can do to make it faster than 5300K/sec please?

Ifti
Getting the hang of things
Posts: 53
Joined: Tue Mar 22, 2011 4:38 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by Ifti » Wed Apr 20, 2016 12:57 am

Same here - adjusted mine to 200000 and my CPU doesn't go over 2%, so I can only assume its not working!!
I have a TVS-871T (i7 model)

I get this:

[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec

P3R
Guru
Posts: 10981
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R » Wed Apr 20, 2016 6:55 am

Ifti wrote:Same here - adjusted mine to 200000 and my CPU doesn't go over 2%, so I can only assume its not working!!
My assumption would instead be that your very fast CPU can easily manage this task already without changing this setting and that your disks are the performance bottlenecks.
I have a TVS-871T (i7 model)
This thread was started in a time when the fastest NAS CPUs had someting like 1/15th or 1/20th of the processing capacity your CPU have.
[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec
Looks decent to me. What's wrong with it?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

Ifti
Getting the hang of things
Posts: 53
Joined: Tue Mar 22, 2011 4:38 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by Ifti » Thu Apr 21, 2016 12:11 am

P3R wrote:
Ifti wrote:Same here - adjusted mine to 200000 and my CPU doesn't go over 2%, so I can only assume its not working!!
My assumption would instead be that your very fast CPU can easily manage this task already without changing this setting and that your disks are the performance bottlenecks.
I have a TVS-871T (i7 model)
This thread was started in a time when the fastest NAS CPUs had someting like 1/15th or 1/20th of the processing capacity your CPU have.
[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec
Looks decent to me. What's wrong with it?



Thanks
3x6TB disks took around 14 hours in the end!
All set - until I add another disk! lol

Rexxy
First post
Posts: 1
Joined: Wed Apr 27, 2016 3:59 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by Rexxy » Wed Apr 27, 2016 4:06 pm

Works nicely, I used this last year on a TS431 and cut down a Raid5 from 4 days to 2.5days.

I am currently doing another Raid5 4 x 4TB on a TVS-663 and after changing the stats it immediately increased..


md1 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[0]
11691190848 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[>....................] resync = 2.2% (85832136/3897063616) finish=329.8min speed=192562K/sec

CPU stats look ok too

PID USER STATUS RSS PPID %CPU %MEM COMMAND
26707 admin SW 0 2 13.8 0.0 md1_raid5
27258 admin DWN 0 2 9.1 0.0 md1_resync

Locked

Return to “Turbo Station Installation & Setup”