[HOWTO] How to increase raid rebuild speed
-
- Starting out
- Posts: 22
- Joined: Mon Dec 22, 2014 4:46 pm
Re: [HOWTO] How to increase raid rebuild speed
So I am pleased to report back that my storage configuration (in the above post) completed syncing in just over 10 hours. Amazing stuff... I see that for some people who posted, they report that changing the maximum and minimum speed did not produce any results or change in sync speed. - I played around with the maximum speed and can confidently report back that the system responded according the my input. Speeding up the process as I adjusted the max speed threshold. Probably saved me having to wait around for another day or two judging from what Ive seen in this thread. Good stuff!
TS-451 with FW/ 4.1.3 0217 and 3x HGST HDN724040ALE640 4TB 7200RPM NAS drives in a RAID5 configuration.
TS-451 with FW/ 4.1.3 0217 and 3x HGST HDN724040ALE640 4TB 7200RPM NAS drives in a RAID5 configuration.
-
- Starting out
- Posts: 40
- Joined: Sun May 24, 2015 12:05 pm
- Location: SE MN, USA
- Contact:
Re: [HOWTO] How to increase raid rebuild speed
I'm glad I found this, thanks for posting! It decreased my rebuild from 1161 minutes remaining to 228 minutes, I only wish I had found it last night when I started, it would probably be done by now. Set the max to 500000 and the min to 100000 with negligible CPU impact. I can afford to have the disk IO eaten up by the rebuild to get it done sooner. I just need to remember to set it back when I am done. My units defaults were max=200000 and min=5000.
Thanks,
Tom
---------------------
NAS : TVS-871-i7-4790S (upgraded from PT) - 16GB RAM
Firmware: 4.4.3.1439
8x Toshiba 4TB HDWQ140XZSTA - 1 Storage Pool - RAID-5; QM2-2P with 2x Samsung 250GB EVO 960 NVMe automatic SSD caching
NAS : TS-963X - 16GB RAM
Firmware: 4.4.3.1439
5x Seagate 3TB ST3000DM001 - 1 Storage Pool - RAID-5, RAID-5; 4x Samsung 120GB SSD automatic SSD caching
Tom
---------------------
NAS : TVS-871-i7-4790S (upgraded from PT) - 16GB RAM
Firmware: 4.4.3.1439
8x Toshiba 4TB HDWQ140XZSTA - 1 Storage Pool - RAID-5; QM2-2P with 2x Samsung 250GB EVO 960 NVMe automatic SSD caching
NAS : TS-963X - 16GB RAM
Firmware: 4.4.3.1439
5x Seagate 3TB ST3000DM001 - 1 Storage Pool - RAID-5, RAID-5; 4x Samsung 120GB SSD automatic SSD caching
-
- Starting out
- Posts: 40
- Joined: Sat Nov 14, 2015 5:56 pm
Re: [HOWTO] How to increase raid rebuild speed
Can I ask something? If it is safe why dont you put this configuration on update firmware so make us all happy without changing it manually?
QNAP TS-453 Pro with 2GB Ram
2x HGST HDN724030ALE640 2.73TB
(RAID1) - Static Single Volume 2.65TB
FW: QTS 4.3.3.0404
2x HGST HDN724030ALE640 2.73TB
(RAID1) - Static Single Volume 2.65TB
FW: QTS 4.3.3.0404
- Don
- Guru
- Posts: 12289
- Joined: Thu Jan 03, 2008 4:56 am
- Location: Long Island, New York
Re: [HOWTO] How to increase raid rebuild speed
You are asking the wrong audience. This is just a community forum and not QNAP support. If I had to guess I would say that it has to do with NAS availability. The higher the rebuild rate the less responsive the NAS will be. The best solution would be to offer an option to change the rebuild rate either through a slider or a high, medium, and low option. Not everyone wants the same rebuild. Some favor getting rebuilt as quickly as possible at the expense of NAS responsiveness while others favor keeping the NAS responsive and letting the raid rebuild take longer. There should not be a one value fits all approach. Until QNAP offers the option to adjust the rebuild speed, if they ever do, you have the option of doing it via the command line. If you always want the same value then you can add the commands to the autorun file. If you would like to see this added as an option then open a ticket with QNAP support.
Use the forum search feature before posting.
Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.
NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.
NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
-
- Starting out
- Posts: 40
- Joined: Sat Nov 14, 2015 5:56 pm
Re: [HOWTO] How to increase raid rebuild speed
I would totally agree with you, you have right. What does bytes per inode? It was set default to 16384. Maybe this decrease the speed? The Max size of volume? How can be 63,99TB when I have only 2x3TB HGS disks?
QNAP TS-453 Pro with 2GB Ram
2x HGST HDN724030ALE640 2.73TB
(RAID1) - Static Single Volume 2.65TB
FW: QTS 4.3.3.0404
2x HGST HDN724030ALE640 2.73TB
(RAID1) - Static Single Volume 2.65TB
FW: QTS 4.3.3.0404
-
- Starting out
- Posts: 15
- Joined: Mon May 06, 2013 5:11 am
Re: [HOWTO] How to increase raid rebuild speed
For a TS-469 L, does anybody have suggestions for a top end number for speed_limit_min? This system has no real user interactive I/O at the moment, so I can pretty much monopolize the system in the rebuild. I am in the midst of replacing drives one by one with larger drives to increase capacity. On drive # 3 of a RAID 10 array . Found this thread last night while working on Drive 1 and upped to 150000. Drive 2 went pretty quickly (about 90 minutes), but drive 3 is about 17 % done after 2 hours.
All I have seen in this thread is about speed_limit_min. I have left speed_limit_max alone.
Thanks!
All I have seen in this thread is about speed_limit_min. I have left speed_limit_max alone.
Thanks!
-
- Starting out
- Posts: 27
- Joined: Mon Aug 24, 2009 10:13 pm
Re: [HOWTO] How to increase raid rebuild speed
Totally agree with the comments against the limiting all migrations to multi-day slow speeds to preserve productive capacity.
I currently have my old 639 and am trying to setup the new TS-471 as quickly as possible so I can move my data across and am having to wait.
Specifically at this point I am migrating a WD RED 6TB RAID1 to 2+1 RAID5 and I wanted to share my findings:
- I have played with the speed_limit_min/max parameters and whatever I did, I couldn't get the rebuild speed above ~25MB/s while the progress was below 50%.
- Once I got above 50%, it seems to have settled down to around ~106MB/s, cutting down the finish time by 75%+, i.e. instead of 30+ hours down to 7 or so.
Hope this helps others.
- Update: at 70% progress, rate down to ~94MB/s.
P.S. I am vague on this but I thought in the past I saw the new volume size at half way point but the migration is at 59% and still showing the old size. When could I expect to see the new size?
I currently have my old 639 and am trying to setup the new TS-471 as quickly as possible so I can move my data across and am having to wait.
Specifically at this point I am migrating a WD RED 6TB RAID1 to 2+1 RAID5 and I wanted to share my findings:
- I have played with the speed_limit_min/max parameters and whatever I did, I couldn't get the rebuild speed above ~25MB/s while the progress was below 50%.
- Once I got above 50%, it seems to have settled down to around ~106MB/s, cutting down the finish time by 75%+, i.e. instead of 30+ hours down to 7 or so.
Hope this helps others.
- Update: at 70% progress, rate down to ~94MB/s.
P.S. I am vague on this but I thought in the past I saw the new volume size at half way point but the migration is at 59% and still showing the old size. When could I expect to see the new size?
TVS-471 16GB - 5.0.0 - 3x WD Red 6TB + WD 10TB
TS-639 Pro - 4.2.6 - 3x WD Red 4TB + WD 10TB
TS-639 Pro - 4.2.6 - 3x WD Red 4TB + WD 10TB
-
- Starting out
- Posts: 12
- Joined: Tue Nov 17, 2015 11:05 pm
- Contact:
Re: [HOWTO] How to increase raid rebuild speed
User error. That's all, folks.
Last edited by kwslark on Fri Jan 15, 2016 6:35 pm, edited 1 time in total.
-
- Guru
- Posts: 13192
- Joined: Sat Dec 29, 2007 1:39 am
- Location: Stockholm, Sweden (UTC+01:00)
Re: [HOWTO] How to increase raid rebuild speed
You need to define "isn't working".kwslark wrote:I logged in to our TVS-1271U-RP-i7-32G via PuTTY using SSH, this command isn't working...
Not possible to change, error messages, no effect or something else?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
-
- New here
- Posts: 2
- Joined: Sun Jan 17, 2016 4:41 pm
Re: [HOWTO] How to increase raid rebuild speed
Hello everyone,
I am reshaping 453mini (RAM 2Gb - 4x HGST IDK 3TB 7200 RPM) from RAID5 (3 disks) to RAID6 (4 disks) - and the speed is extremely slow (around 5000K/sec).
I've tried everything from the post and it doesn’t change the situation at all, CPU load doesn't go over 6-10% and the speed 5300K/sec at max!
Here is what I’ve tried:
I gradually started setting up a speed_limit_min to 50000 as suggested and then tried even 400000 with no impact on CPU or the speed at all.
Also tried to change cache, but no luck:
returns: No such file or directory, why is that?
returns:
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md1 : active raid6 sde3[5] sdc3[2] sda3[3] sdb3[4]
5840623240 blocks super 1.0 level 6, 4k chunk, algorithm 18 [4/3] [UUU_]
[>....................] reshape = 2.9% (86089728/2920311620) finish=9334.9min speed=5059K/sec
md256 : active raid1 sde2[3](S) sda2[2](S) sdb2[1] sdc2[0]
530112 blocks super 1.0 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
md13 : active raid1 sde4[24] sdc4[25] sdb4[27] sda4[26]
458880 blocks super 1.0 [24/4] [UUUU____________________]
bitmap: 1/1 pages [4KB], 65536KB chunk
md9 : active raid1 sde1[24] sdc1[25] sdb1[27] sda1[26]
530048 blocks super 1.0 [24/4] [UUUU____________________]
bitmap: 1/1 pages [4KB], 65536KB chunk
unused devices: <none>
returns: (progress after 17 hours)
[==>..................] reshape = 11.1% (324481024/2920311620) finish=8500.6min speed=5088K/sec
What do I do wrong? Can anyone help please?
Thank you
I am reshaping 453mini (RAM 2Gb - 4x HGST IDK 3TB 7200 RPM) from RAID5 (3 disks) to RAID6 (4 disks) - and the speed is extremely slow (around 5000K/sec).
I've tried everything from the post and it doesn’t change the situation at all, CPU load doesn't go over 6-10% and the speed 5300K/sec at max!
Here is what I’ve tried:
Code: Select all
echo 800000 >/proc/sys/dev/raid/speed_limit_max
echo 400000 >/proc/sys/dev/raid/speed_limit_min
Also tried to change cache, but no luck:
Code: Select all
cat: /sys/block/md0/md/stripe_cache_size
Code: Select all
cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md1 : active raid6 sde3[5] sdc3[2] sda3[3] sdb3[4]
5840623240 blocks super 1.0 level 6, 4k chunk, algorithm 18 [4/3] [UUU_]
[>....................] reshape = 2.9% (86089728/2920311620) finish=9334.9min speed=5059K/sec
md256 : active raid1 sde2[3](S) sda2[2](S) sdb2[1] sdc2[0]
530112 blocks super 1.0 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
md13 : active raid1 sde4[24] sdc4[25] sdb4[27] sda4[26]
458880 blocks super 1.0 [24/4] [UUUU____________________]
bitmap: 1/1 pages [4KB], 65536KB chunk
md9 : active raid1 sde1[24] sdc1[25] sdb1[27] sda1[26]
530048 blocks super 1.0 [24/4] [UUUU____________________]
bitmap: 1/1 pages [4KB], 65536KB chunk
unused devices: <none>
Code: Select all
egrep speed /proc/mdstat
[==>..................] reshape = 11.1% (324481024/2920311620) finish=8500.6min speed=5088K/sec
What do I do wrong? Can anyone help please?
Thank you
-
- New here
- Posts: 2
- Joined: Sun Jan 17, 2016 4:41 pm
Re: [HOWTO] How to increase raid rebuild speed
I've been reshaping RAID5 to RAID6 for over 3 days now and it's only on 44%... I've played with the cache size as well, changed it to 32768, 16384 and 8192, nothing does help.
Does anyone have any suggestions what I can do to make it faster than 5300K/sec please?
Does anyone have any suggestions what I can do to make it faster than 5300K/sec please?
-
- Getting the hang of things
- Posts: 82
- Joined: Tue Mar 22, 2011 4:38 pm
Re: [HOWTO] How to increase raid rebuild speed
Same here - adjusted mine to 200000 and my CPU doesn't go over 2%, so I can only assume its not working!!
I have a TVS-871T (i7 model)
I get this:
[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec
I have a TVS-871T (i7 model)
I get this:
[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec
-
- Guru
- Posts: 13192
- Joined: Sat Dec 29, 2007 1:39 am
- Location: Stockholm, Sweden (UTC+01:00)
Re: [HOWTO] How to increase raid rebuild speed
My assumption would instead be that your very fast CPU can easily manage this task already without changing this setting and that your disks are the performance bottlenecks.Ifti wrote:Same here - adjusted mine to 200000 and my CPU doesn't go over 2%, so I can only assume its not working!!
This thread was started in a time when the fastest NAS CPUs had someting like 1/15th or 1/20th of the processing capacity your CPU have.I have a TVS-871T (i7 model)
Looks decent to me. What's wrong with it?[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
-
- Getting the hang of things
- Posts: 82
- Joined: Tue Mar 22, 2011 4:38 pm
Re: [HOWTO] How to increase raid rebuild speed
P3R wrote:My assumption would instead be that your very fast CPU can easily manage this task already without changing this setting and that your disks are the performance bottlenecks.Ifti wrote:Same here - adjusted mine to 200000 and my CPU doesn't go over 2%, so I can only assume its not working!!This thread was started in a time when the fastest NAS CPUs had someting like 1/15th or 1/20th of the processing capacity your CPU have.I have a TVS-871T (i7 model)Looks decent to me. What's wrong with it?[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec
Thanks
3x6TB disks took around 14 hours in the end!
All set - until I add another disk! lol
-
- First post
- Posts: 1
- Joined: Wed Apr 27, 2016 3:59 pm
Re: [HOWTO] How to increase raid rebuild speed
Works nicely, I used this last year on a TS431 and cut down a Raid5 from 4 days to 2.5days.
I am currently doing another Raid5 4 x 4TB on a TVS-663 and after changing the stats it immediately increased..
md1 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[0]
11691190848 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[>....................] resync = 2.2% (85832136/3897063616) finish=329.8min speed=192562K/sec
CPU stats look ok too
PID USER STATUS RSS PPID %CPU %MEM COMMAND
26707 admin SW 0 2 13.8 0.0 md1_raid5
27258 admin DWN 0 2 9.1 0.0 md1_resync
I am currently doing another Raid5 4 x 4TB on a TVS-663 and after changing the stats it immediately increased..
md1 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[0]
11691190848 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[>....................] resync = 2.2% (85832136/3897063616) finish=329.8min speed=192562K/sec
CPU stats look ok too
PID USER STATUS RSS PPID %CPU %MEM COMMAND
26707 admin SW 0 2 13.8 0.0 md1_raid5
27258 admin DWN 0 2 9.1 0.0 md1_resync