Page 7 of 13
Re: [HOWTO] How to increase raid rebuild speed
Posted: Sat May 11, 2013 8:40 am
by musashi77
Finnally finished, however the extra harddrive space is not recognised. It shows that all 4 drives are part of the RAID but on the space I originally had with 3. Have tried a "check" volume and a re-start of the unit. Any ideas?
Re: [HOWTO] How to increase raid rebuild speed
Posted: Sat May 11, 2013 10:20 am
by Don
musashi77 wrote:Finnally finished, however the extra harddrive space is not recognised. It shows that all 4 drives are part of the RAID but on the space I originally had with 3. Have tried a "check" volume and a re-start of the unit. Any ideas?
Off topic. Start a new thread.
Re: [HOWTO] How to increase raid rebuild speed
Posted: Sun May 12, 2013 10:02 pm
by AUSTraveler
AUSTraveler wrote:TS-809 Pro running FW 3.8.2. Upgraded hard drives this past week and now performing a RAID6 reshape from 4 drives to 5 drives (all Seagate ST4000DM000s). Started the process last night, and it was only 20% when I woke up (according to mdstat). Found this thread and tried a few different values for speed_limit_min, speed_limit_max, and stripe_cache_size.
CPU core 1 was between 19% and 25% for all values tested (running md0_raid6). CPU core 2 was between 4% and 7% (running md0_reshape). CPU has headroom, and I've stopped all non-esssential services.
Best speed I got was 23.8K/sec on the reshape, and that was with the default values (min: 50000, max: 100000, stripe_cache_size: 4096). Increasing the min had very little effect. Increasing the stripe_cache_size used more memory but dropped speed of reshape as follows
stripe_cache_size: 8192 --> speed: 22.2K/sec
stripe_cache_size: 16384 --> speed: 20K/sec
So, I've gone back to the defaults for now and will wait out the remaining ~37 hours.
Also, when I tried to lower stripe_cache_size, I found I had to set 4096 with the echo command several times before it would drop back down to 4096.
At about 70 or 75%, the speed was above 43K/sec with CPUs near 30% and 10%. So, the rebuild was faster at the end with no changes from my end.
Also, it failed to complete on its own, and I had to follow the steps on this thread to resize manually -
http://forum.qnap.com/viewtopic.php?t=12109
Re: [HOWTO] How to increase raid rebuild speed
Posted: Sat Jun 15, 2013 11:17 am
by marshalleq
You will get a bit more speed by:
# echo 8192 >/sys/block/md0/md/stripe_cache_size
You can get a bit more by using blockdev (which appears not to be installed by default on qnap by:
blockdev --report (Shows you what your readahead is set to then:
blockdev --setra 16384 /dev/md0
I have a 669-L and my CPU doesn't get past 50% with all this on. Though there are no users on it but me...
Re: [HOWTO] How to increase raid rebuild speed
Posted: Thu Jun 27, 2013 1:55 am
by fantomas
marshalleq wrote:You will get a bit more speed by:
# echo 8192 >/sys/block/md0/md/stripe_cache_size
I have already posted this recommendation. Note that this only applies for RAID5 and RAID6.
Re: [HOWTO] How to increase raid rebuild speed
Posted: Thu Jun 27, 2013 4:51 am
by marshalleq
Yes apologies I somehow missed it.
Re: [HOWTO] How to increase raid rebuild speed
Posted: Wed Jul 10, 2013 12:59 am
by Jorgje
Hint: DISABLE your download station
In my case, I was creating a new RAID1 volume with two empty disks (2x Western Digital WD30EFRX) on my TS-639 PRO.
Meanwhile, my download station was still running. Although I'm downloading data to another disk that is outside of the RAID array I was creating, in the past I always noticed that the download station has a lot of activity on all disks. This seems to be activity on the all-disk RAID1 volume that keeps the QNAP system config (/dev/md9, which is mounted /mnt/HDA_ROOT in my case).
To give you an idea:
Sync of the new RAID1 volume was originally running around 22 MB/s.
After raising the /proc/sys/dev/raid/speed_limit_min to 150000, RAID sync speeded up to 44 MB/s.
After disabling the download station (and as a result stop letting the download station bother all disks), RAID sync speeded up to around 100 MB/s.
Re: [HOWTO] How to increase raid rebuild speed
Posted: Fri Jul 19, 2013 7:20 pm
by takeanap
I think it's my first post - yeah - and just want to say thank you and provide some numbers on how good this HOWTO helped in getting my old 410er upgraded:
Situation: TS410 w/ 4 x 500G Seagate Barracuda (some Enterprise 24/7 Disks); RAID 5, EXT3 (no bitmap)
FW: 3.7.3 Build 20120801
Ordered: 4 x 2 TB Seagate ST2000VX000 (not so enterprise, but 24/7 ... maybe)
//cut
QNAP TS-410: 4 x 500G => 4 x 2T (ext3, no bitmap)
2013/07/02:
12:50: replaced disk 1
12:52: rebuilding/recovery (mdstat=> 20 MB/s) (rest 360min)
13:20: speed_limit_min => 7000 =>10000 => 30000 => _50000_ (default 5000) (MDstat => 24MB/s)
14:20: MDstat => 28MB/s (28%) (rest 200min)
14:50: MDstat => 28MB/s (38%) (rest ???min)
16:50: MDstat => 29MB/s (79%) (rest 58min)
17:52: Total: ~4:58
17:57: replaced disk 2
18:00: MDstat => 29MB/s ( 1%) (rest 290min)
2013/07/03
14:00: : resync started (estimated 1200min, 21 MB/s)
...
2013/07/04
09:20: : resync (estimated 10min, 21 MB/s)
09:43: Total 19:55
09:45: e2fsck ...
11:55: done
//cut
In short:
- overall it took ~2 days including nicht times where I did not swap disks
- bumped rebuild throughput by ~50%; limit was '50000' no firther improvement above that (w/ TS410)
- Overall time saved (my estimate) ~ 6 h
- CPU at ~100%
- no services runnnig/available (manually disabled all of them)
- Admin UI and ssh was possible.
Good night, and good luck
Re: [HOWTO] How to increase raid rebuild speed
Posted: Sat Aug 17, 2013 5:18 pm
by x3mgroup
So i just purchased two new QNAP's
TS-269L (with 2x 3TB/7200rpm HDD - Seagate ST3000DM001-1CH1CC27 )
TS-269 Pro (with 2 4TB/7200rpm HDD - Hitachi HDS724040ALE640 MJAO)
After playing with the PRO, i decided to return to "Factory Settings" and reinitialize the drives.
Meanwhile i was reading the forum, and came along this thread. And indeed i wouldn't mind to have the whole process of re-syncing the drives go a little quicker!
Doing some calc in would take about
10 HOURS!!!
So i followed the tips, and applied the tweaks
Code: Select all
echo 50000 >/proc/sys/dev/raid/speed_limit_min
But no performance increase, and the CPU stayed on an average of 12%
I changed the value (50000) up to
1000000 !!!, but no change in preformance and CPU usage
So it looks that this will not work on the latest devices, with
firmware version 4.0.2
It will take about 10 HOURS to finish
Does anyone else have had similar experience?
Does anyone know why?
Regards
Re: [HOWTO] How to increase raid rebuild speed
Posted: Tue Aug 20, 2013 6:46 pm
by fantomas
x3mgroup wrote:
TS-269L (with 2x 3TB/7200rpm HDD - Seagate ST3000DM001-1CH1CC27 )
These were reported to be buggy and fail too often (search forums for more info)
x3mgroup wrote:
TS-269 Pro (with 2 4TB/7200rpm HDD - Hitachi HDS724040ALE640 MJAO)
After playing with the PRO, i decided to return to "Factory Settings" and reinitialize the drives.
Meanwhile i was reading the forum, and came along this thread. And indeed i wouldn't mind to have the whole process of re-syncing the drives go a little quicker!
Doing some calc in would take about 10 HOURS!!!
copying 4TB of data will take much time. even on 100MB/s it takes 11 hours. 100MB/s is quite good performance with current disks. You have apparently encountered your NAS limits.
x3mgroup wrote:
So i followed the tips, and applied the tweaks
Code: Select all
echo 50000 >/proc/sys/dev/raid/speed_limit_min
But no performance increase, and the CPU stayed on an average of 12%
I changed the value (50000) up to
1000000 !!!, but no change in preformance and CPU usages
the speed_limit_min only changes how the disk performance is split between raid rebuilding and standard usage (clients, applications etc). It would only give bigger rebuild speed if clients were reading or writing the data on your NAS, slowing clients down. It does not make the disks faster.
Re: [HOWTO] How to increase raid rebuild speed
Posted: Wed Aug 21, 2013 6:04 am
by marshalleq
I don't agree with the third point, for me I noticed quite a performance increase for a single user. But like anything your mileage will vary. I have no facts to base this on, but given you have a 269 which I assume is similar hardware to my 669 and that I got performance improvements on both the 669 and the lower spec 412, I suspect the issue has more to do with your running raid 1. Quite possibly RAID-1 is less effected by these commands.
Re: [HOWTO] How to increase raid rebuild speed
Posted: Fri Aug 30, 2013 4:36 am
by chodaboy19
I wanted to confirm this works nicely with the TS-559 Pro+ and QTS v4.0.2:
[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sde3[5] sda3[0] sdd3[3] sdc3[2] sdb3[1]
7807782400 blocks level 5, 64k chunk, algorithm 2 [5/4] [UUUU_]
[=>...................] recovery = 6.6% (130539324/1951945600) finish=242.7min speed=125057K/sec
md5 : active raid1 sde2[2](S) sdd2[3](S) sdc2[4](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]
md13 : active raid1 sde4[4] sda4[0] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [5/5] [UUUUU]
bitmap: 0/57 pages [0KB], 4KB chunk
md9 : active raid1 sde1[4] sda1[0] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [5/5] [UUUUU]
bitmap: 0/65 pages [0KB], 4KB chunk
unused devices: <none>
[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sde3[5] sda3[0] sdd3[3] sdc3[2] sdb3[1]
7807782400 blocks level 5, 64k chunk, algorithm 2 [5/4] [UUUU_]
[=>...................] recovery = 6.8% (133306172/1951945600) finish=239.9min speed=126298K/sec
md5 : active raid1 sde2[2](S) sdd2[3](S) sdc2[4](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]
md13 : active raid1 sde4[4] sda4[0] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [5/5] [UUUUU]
bitmap: 0/57 pages [0KB], 4KB chunk
md9 : active raid1 sde1[4] sda1[0] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [5/5] [UUUUU]
bitmap: 0/65 pages [0KB], 4KB chunk
unused devices: <none>
[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md0 : active raid5 sde3[5] sda3[0] sdd3[3] sdc3[2] sdb3[1]
7807782400 blocks level 5, 64k chunk, algorithm 2 [5/4] [UUUU_]
[=>...................] recovery = 6.8% (134224700/1951945600) finish=244.3min speed=123972K/sec
md5 : active raid1 sde2[2](S) sdd2[3](S) sdc2[4](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]
md13 : active raid1 sde4[4] sda4[0] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [5/5] [UUUUU]
bitmap: 0/57 pages [0KB], 4KB chunk
md9 : active raid1 sde1[4] sda1[0] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [5/5] [UUUUU]
bitmap: 0/65 pages [0KB], 4KB chunk
unused devices: <none>
[~] #
Getting up to 126MB/sec! Too bad I am limited by SATA I speeds, or maybe it could go even faster!
Re: [HOWTO] How to increase raid rebuild speed
Posted: Sun Sep 01, 2013 2:22 pm
by marshalleq
BTW to get the blockdev package, you can install util-linux-ng from ipkg and to get watch install the procps package also from ipkg.
Re: [HOWTO] How to increase raid rebuild speed
Posted: Tue Sep 03, 2013 2:06 pm
by marshalleq
May as well add mine as a reference for others to compare to. 3x4TB drives in RAID-5. Rebuild of ejected disk went from 5MB/s to 60-70MB/s. It's a 669-L. Read ahead made a negative difference, adjusting raid min speed made the most difference and stripe size made a small difference. I confirmed these settings have no effect on mirror in my case.
Also confirmed having bitmaps on slows down straight copy (but obviously speeds up failed raid disk rebuild).
Re: [HOWTO] How to increase raid rebuild speed
Posted: Thu Sep 05, 2013 1:45 pm
by lindsay.mathieson
Just to add a data point - Setting up a TS-420 with 4*3TB WD Reds using RAID 10. Initial sync was painful, looking at a projected time of 2 days. Sync speed was around 40000k/sec
Bumped min up to 200000, max up to 400000 and set blockdev --setra 16384 /dev/md0, disabled all media and download services.
Sync speed is averaging 77000k/sec and the estimate to completion is 20 hours - big improvement.