[HOWTO] How to increase raid rebuild speed

Discussion on setting up QNAP NAS products.
Locked
musashi77
Starting out
Posts: 17
Joined: Mon Jan 28, 2008 8:02 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by musashi77 » Sat May 11, 2013 8:40 am

Finnally finished, however the extra harddrive space is not recognised. It shows that all 4 drives are part of the RAID but on the space I originally had with 3. Have tried a "check" volume and a re-start of the unit. Any ideas?

User avatar
Don
Guru
Posts: 11998
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: [HOWTO] How to increase raid rebuild speed

Post by Don » Sat May 11, 2013 10:20 am

musashi77 wrote:Finnally finished, however the extra harddrive space is not recognised. It shows that all 4 drives are part of the RAID but on the space I originally had with 3. Have tried a "check" volume and a re-start of the unit. Any ideas?

Off topic. Start a new thread.
Read the Online Manuals and use the forum search feature before posting.

It is a recommended to use RAID and have external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

Submit bugs and feature requests to QNAP via their Helpdesk app.

NAS: TVS-882BR | F/W: 4.3.6.0895 | 40GB | 2 x M.2 SATA RAID 1 (System/VMs) | 4 x M.2 NMVe QM2-4P-384A RAID 5 (Cache) | 5 x 4TB HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-663 | F/W: 4.4.1.1086 | 16GB | 2 x M.2 NMVe QM2-2P RAID 1 (Cache) | 4 x 4TB RAID 5
Apps: Boinc, Squid, DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS, Entware, DLstation, +others

AUSTraveler
New here
Posts: 6
Joined: Sun Jan 17, 2010 9:46 am

Re: [HOWTO] How to increase raid rebuild speed

Post by AUSTraveler » Sun May 12, 2013 10:02 pm

AUSTraveler wrote:TS-809 Pro running FW 3.8.2. Upgraded hard drives this past week and now performing a RAID6 reshape from 4 drives to 5 drives (all Seagate ST4000DM000s). Started the process last night, and it was only 20% when I woke up (according to mdstat). Found this thread and tried a few different values for speed_limit_min, speed_limit_max, and stripe_cache_size.

CPU core 1 was between 19% and 25% for all values tested (running md0_raid6). CPU core 2 was between 4% and 7% (running md0_reshape). CPU has headroom, and I've stopped all non-esssential services.

Best speed I got was 23.8K/sec on the reshape, and that was with the default values (min: 50000, max: 100000, stripe_cache_size: 4096). Increasing the min had very little effect. Increasing the stripe_cache_size used more memory but dropped speed of reshape as follows

stripe_cache_size: 8192 --> speed: 22.2K/sec
stripe_cache_size: 16384 --> speed: 20K/sec

So, I've gone back to the defaults for now and will wait out the remaining ~37 hours.

Also, when I tried to lower stripe_cache_size, I found I had to set 4096 with the echo command several times before it would drop back down to 4096.


At about 70 or 75%, the speed was above 43K/sec with CPUs near 30% and 10%. So, the rebuild was faster at the end with no changes from my end.

Also, it failed to complete on its own, and I had to follow the steps on this thread to resize manually - http://forum.qnap.com/viewtopic.php?t=12109

User avatar
marshalleq
Easy as a breeze
Posts: 302
Joined: Mon Jul 23, 2012 2:44 pm
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by marshalleq » Sat Jun 15, 2013 11:17 am

You will get a bit more speed by:

# echo 8192 >/sys/block/md0/md/stripe_cache_size

You can get a bit more by using blockdev (which appears not to be installed by default on qnap by:

blockdev --report (Shows you what your readahead is set to then:
blockdev --setra 16384 /dev/md0

I have a 669-L and my CPU doesn't get past 50% with all this on. Though there are no users on it but me... :)
TS669-L (3GB RAM) fw 4.3.2.0050 Build 20170104 with 6x4TB Seagate ST4000VN000-1H41SC43 2x2TB Hitachi HDS723020BLA642 2x8TB ST8000VN0022-2EL112
QNAP UX-500P External 5 Bay USB3 Expansion Cabinet
Raspberry Pi 1,2 & 3
Mac Mini
Apple TV 3/4
Windows
Sabayon Linux
Ubuntu Linux
Android
Docker
Ryzen 1800x custom system

Web
https://www.tech-knowhow.com

Twitter
@Marshalleq
@TechKn0wH0w

fantomas
Experience counts
Posts: 1549
Joined: Mon Feb 07, 2011 5:40 am
Location: Bratislava, Slovakia
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by fantomas » Thu Jun 27, 2013 1:55 am

marshalleq wrote:You will get a bit more speed by:

# echo 8192 >/sys/block/md0/md/stripe_cache_size


I have already posted this recommendation. Note that this only applies for RAID5 and RAID6.
experience with administration of UN*X (mostly linux) and applications on internet servers since 1994...

User avatar
marshalleq
Easy as a breeze
Posts: 302
Joined: Mon Jul 23, 2012 2:44 pm
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by marshalleq » Thu Jun 27, 2013 4:51 am

Yes apologies I somehow missed it.
TS669-L (3GB RAM) fw 4.3.2.0050 Build 20170104 with 6x4TB Seagate ST4000VN000-1H41SC43 2x2TB Hitachi HDS723020BLA642 2x8TB ST8000VN0022-2EL112
QNAP UX-500P External 5 Bay USB3 Expansion Cabinet
Raspberry Pi 1,2 & 3
Mac Mini
Apple TV 3/4
Windows
Sabayon Linux
Ubuntu Linux
Android
Docker
Ryzen 1800x custom system

Web
https://www.tech-knowhow.com

Twitter
@Marshalleq
@TechKn0wH0w

Jorgje
New here
Posts: 6
Joined: Mon Nov 08, 2010 12:49 am

Re: [HOWTO] How to increase raid rebuild speed

Post by Jorgje » Wed Jul 10, 2013 12:59 am

Hint: DISABLE your download station

In my case, I was creating a new RAID1 volume with two empty disks (2x Western Digital WD30EFRX) on my TS-639 PRO.
Meanwhile, my download station was still running. Although I'm downloading data to another disk that is outside of the RAID array I was creating, in the past I always noticed that the download station has a lot of activity on all disks. This seems to be activity on the all-disk RAID1 volume that keeps the QNAP system config (/dev/md9, which is mounted /mnt/HDA_ROOT in my case).

To give you an idea:
Sync of the new RAID1 volume was originally running around 22 MB/s.
After raising the /proc/sys/dev/raid/speed_limit_min to 150000, RAID sync speeded up to 44 MB/s.
After disabling the download station (and as a result stop letting the download station bother all disks), RAID sync speeded up to around 100 MB/s.
TS-639 Pro - 2x WD WD30EFRX 3TB, 2x Samsung HD204UI 2TB, 2x Seagate Barracuda 7200.11 320GB
TS-219P+ - 2x Samsung HD204UI 2TB

takeanap
New here
Posts: 5
Joined: Mon Feb 08, 2010 4:43 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by takeanap » Fri Jul 19, 2013 7:20 pm

I think it's my first post - yeah - and just want to say thank you and provide some numbers on how good this HOWTO helped in getting my old 410er upgraded:

Situation: TS410 w/ 4 x 500G Seagate Barracuda (some Enterprise 24/7 Disks); RAID 5, EXT3 (no bitmap)

FW: 3.7.3 Build 20120801

Ordered: 4 x 2 TB Seagate ST2000VX000 (not so enterprise, but 24/7 ... maybe)

//cut
QNAP TS-410: 4 x 500G => 4 x 2T (ext3, no bitmap)

2013/07/02:
12:50: replaced disk 1
12:52: rebuilding/recovery (mdstat=> 20 MB/s) (rest 360min)
13:20: speed_limit_min => 7000 =>10000 => 30000 => _50000_ (default 5000) (MDstat => 24MB/s)
14:20: MDstat => 28MB/s (28%) (rest 200min)
14:50: MDstat => 28MB/s (38%) (rest ???min)
16:50: MDstat => 29MB/s (79%) (rest 58min)
17:52: Total: ~4:58
17:57: replaced disk 2
18:00: MDstat => 29MB/s ( 1%) (rest 290min)

2013/07/03
14:00: : resync started (estimated 1200min, 21 MB/s)
...
2013/07/04
09:20: : resync (estimated 10min, 21 MB/s)
09:43: Total 19:55
09:45: e2fsck ...
11:55: done
//cut

In short:
- overall it took ~2 days including nicht times where I did not swap disks
- bumped rebuild throughput by ~50%; limit was '50000' no firther improvement above that (w/ TS410)
- Overall time saved (my estimate) ~ 6 h
- CPU at ~100%
- no services runnnig/available (manually disabled all of them)
- Admin UI and ssh was possible.

Good night, and good luck

x3mgroup
New here
Posts: 3
Joined: Sat Aug 17, 2013 4:42 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by x3mgroup » Sat Aug 17, 2013 5:18 pm

So i just purchased two new QNAP's
TS-269L (with 2x 3TB/7200rpm HDD - Seagate ST3000DM001-1CH1CC27 )
TS-269 Pro (with 2 4TB/7200rpm HDD - Hitachi HDS724040ALE640 MJAO)

After playing with the PRO, i decided to return to "Factory Settings" and reinitialize the drives.
Meanwhile i was reading the forum, and came along this thread. And indeed i wouldn't mind to have the whole process of re-syncing the drives go a little quicker!
Doing some calc in would take about 10 HOURS!!!

So i followed the tips, and applied the tweaks

Code: Select all

echo 50000 >/proc/sys/dev/raid/speed_limit_min


But no performance increase, and the CPU stayed on an average of 12%

I changed the value (50000) up to 1000000 !!!, but no change in preformance and CPU usage

So it looks that this will not work on the latest devices, with firmware version 4.0.2

It will take about 10 HOURS to finish

Does anyone else have had similar experience?
Does anyone know why?


Regards

fantomas
Experience counts
Posts: 1549
Joined: Mon Feb 07, 2011 5:40 am
Location: Bratislava, Slovakia
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by fantomas » Tue Aug 20, 2013 6:46 pm

x3mgroup wrote:TS-269L (with 2x 3TB/7200rpm HDD - Seagate ST3000DM001-1CH1CC27 )


These were reported to be buggy and fail too often (search forums for more info)

x3mgroup wrote:TS-269 Pro (with 2 4TB/7200rpm HDD - Hitachi HDS724040ALE640 MJAO)

After playing with the PRO, i decided to return to "Factory Settings" and reinitialize the drives.
Meanwhile i was reading the forum, and came along this thread. And indeed i wouldn't mind to have the whole process of re-syncing the drives go a little quicker!
Doing some calc in would take about 10 HOURS!!!


copying 4TB of data will take much time. even on 100MB/s it takes 11 hours. 100MB/s is quite good performance with current disks. You have apparently encountered your NAS limits.

x3mgroup wrote:So i followed the tips, and applied the tweaks

Code: Select all

echo 50000 >/proc/sys/dev/raid/speed_limit_min


But no performance increase, and the CPU stayed on an average of 12%

I changed the value (50000) up to 1000000 !!!, but no change in preformance and CPU usages


the speed_limit_min only changes how the disk performance is split between raid rebuilding and standard usage (clients, applications etc). It would only give bigger rebuild speed if clients were reading or writing the data on your NAS, slowing clients down. It does not make the disks faster.
experience with administration of UN*X (mostly linux) and applications on internet servers since 1994...

User avatar
marshalleq
Easy as a breeze
Posts: 302
Joined: Mon Jul 23, 2012 2:44 pm
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by marshalleq » Wed Aug 21, 2013 6:04 am

I don't agree with the third point, for me I noticed quite a performance increase for a single user. But like anything your mileage will vary. I have no facts to base this on, but given you have a 269 which I assume is similar hardware to my 669 and that I got performance improvements on both the 669 and the lower spec 412, I suspect the issue has more to do with your running raid 1. Quite possibly RAID-1 is less effected by these commands.
TS669-L (3GB RAM) fw 4.3.2.0050 Build 20170104 with 6x4TB Seagate ST4000VN000-1H41SC43 2x2TB Hitachi HDS723020BLA642 2x8TB ST8000VN0022-2EL112
QNAP UX-500P External 5 Bay USB3 Expansion Cabinet
Raspberry Pi 1,2 & 3
Mac Mini
Apple TV 3/4
Windows
Sabayon Linux
Ubuntu Linux
Android
Docker
Ryzen 1800x custom system

Web
https://www.tech-knowhow.com

Twitter
@Marshalleq
@TechKn0wH0w

chodaboy19
Know my way around
Posts: 196
Joined: Thu Nov 26, 2009 4:28 am
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by chodaboy19 » Fri Aug 30, 2013 4:36 am

I wanted to confirm this works nicely with the TS-559 Pro+ and QTS v4.0.2:

[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sde3[5] sda3[0] sdd3[3] sdc3[2] sdb3[1]
7807782400 blocks level 5, 64k chunk, algorithm 2 [5/4] [UUUU_]
[=>...................] recovery = 6.6% (130539324/1951945600) finish=242.7min speed=125057K/sec

md5 : active raid1 sde2[2](S) sdd2[3](S) sdc2[4](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]

md13 : active raid1 sde4[4] sda4[0] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [5/5] [UUUUU]
bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sde1[4] sda1[0] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [5/5] [UUUUU]
bitmap: 0/65 pages [0KB], 4KB chunk

unused devices: <none>
[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sde3[5] sda3[0] sdd3[3] sdc3[2] sdb3[1]
7807782400 blocks level 5, 64k chunk, algorithm 2 [5/4] [UUUU_]
[=>...................] recovery = 6.8% (133306172/1951945600) finish=239.9min speed=126298K/sec

md5 : active raid1 sde2[2](S) sdd2[3](S) sdc2[4](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]

md13 : active raid1 sde4[4] sda4[0] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [5/5] [UUUUU]
bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sde1[4] sda1[0] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [5/5] [UUUUU]
bitmap: 0/65 pages [0KB], 4KB chunk

unused devices: <none>
[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sde3[5] sda3[0] sdd3[3] sdc3[2] sdb3[1]
7807782400 blocks level 5, 64k chunk, algorithm 2 [5/4] [UUUU_]
[=>...................] recovery = 6.8% (134224700/1951945600) finish=244.3min speed=123972K/sec

md5 : active raid1 sde2[2](S) sdd2[3](S) sdc2[4](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]

md13 : active raid1 sde4[4] sda4[0] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [5/5] [UUUUU]
bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sde1[4] sda1[0] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [5/5] [UUUUU]
bitmap: 0/65 pages [0KB], 4KB chunk

unused devices: <none>
[~] #


Getting up to 126MB/sec! Too bad I am limited by SATA I speeds, or maybe it could go even faster!

TS-670 v4.3.6 (20191212)
i7-3770S
16GB DDR3 1600MHz 1.35v (F3-1600C9D-16GRSL)
6 x 8TB WD (WD80EFZX)
1 x QM2-2P-384
2 x ADATA XPG SX8200 Pro 1.0TB

User avatar
marshalleq
Easy as a breeze
Posts: 302
Joined: Mon Jul 23, 2012 2:44 pm
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by marshalleq » Sun Sep 01, 2013 2:22 pm

BTW to get the blockdev package, you can install util-linux-ng from ipkg and to get watch install the procps package also from ipkg.
TS669-L (3GB RAM) fw 4.3.2.0050 Build 20170104 with 6x4TB Seagate ST4000VN000-1H41SC43 2x2TB Hitachi HDS723020BLA642 2x8TB ST8000VN0022-2EL112
QNAP UX-500P External 5 Bay USB3 Expansion Cabinet
Raspberry Pi 1,2 & 3
Mac Mini
Apple TV 3/4
Windows
Sabayon Linux
Ubuntu Linux
Android
Docker
Ryzen 1800x custom system

Web
https://www.tech-knowhow.com

Twitter
@Marshalleq
@TechKn0wH0w

User avatar
marshalleq
Easy as a breeze
Posts: 302
Joined: Mon Jul 23, 2012 2:44 pm
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by marshalleq » Tue Sep 03, 2013 2:06 pm

May as well add mine as a reference for others to compare to. 3x4TB drives in RAID-5. Rebuild of ejected disk went from 5MB/s to 60-70MB/s. It's a 669-L. Read ahead made a negative difference, adjusting raid min speed made the most difference and stripe size made a small difference. I confirmed these settings have no effect on mirror in my case.

Also confirmed having bitmaps on slows down straight copy (but obviously speeds up failed raid disk rebuild).
TS669-L (3GB RAM) fw 4.3.2.0050 Build 20170104 with 6x4TB Seagate ST4000VN000-1H41SC43 2x2TB Hitachi HDS723020BLA642 2x8TB ST8000VN0022-2EL112
QNAP UX-500P External 5 Bay USB3 Expansion Cabinet
Raspberry Pi 1,2 & 3
Mac Mini
Apple TV 3/4
Windows
Sabayon Linux
Ubuntu Linux
Android
Docker
Ryzen 1800x custom system

Web
https://www.tech-knowhow.com

Twitter
@Marshalleq
@TechKn0wH0w

lindsay.mathieson
Starting out
Posts: 11
Joined: Wed Sep 04, 2013 1:45 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by lindsay.mathieson » Thu Sep 05, 2013 1:45 pm

Just to add a data point - Setting up a TS-420 with 4*3TB WD Reds using RAID 10. Initial sync was painful, looking at a projected time of 2 days. Sync speed was around 40000k/sec

Bumped min up to 200000, max up to 400000 and set blockdev --setra 16384 /dev/md0, disabled all media and download services.

Sync speed is averaging 77000k/sec and the estimate to completion is 20 hours - big improvement.

Locked

Return to “Turbo Station Installation & Setup”