[HOWTO] How to increase raid rebuild speed

Discussion on setting up QNAP NAS products.
Locked
User avatar
schumaku
Guru
Posts: 43664
Joined: Mon Jan 21, 2008 4:41 pm
Location: Kloten (Zurich), Switzerland -- Skype: schumaku
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by schumaku » Mon Mar 04, 2013 3:46 am

Well, all new NAS, no data? Crazy idea in my opinion not to start using the intended storage layout, and use Expand Capacity (Online RAID Capacity Expansion) at this stage.

In any RAID operation, all storage blocks are touched - being a resync, reshape, ... it does not matter of the RAID storage blocks are occupied or empty - the only exception is resync with Bitmaps enabled where only changed blocks are dealed with.

User avatar
Don
Guru
Posts: 12008
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: [HOWTO] How to increase raid rebuild speed

Post by Don » Mon Mar 04, 2013 5:50 am

Goal - it's possible you have a bad drive. This will increase rebuild times because of the increased errors recovery.
Read the Online Manuals and use the forum search feature before posting.

It is a recommended to use RAID and have external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

Submit bugs and feature requests to QNAP via their Helpdesk app.

NAS: TVS-882BR | F/W: 4.3.6.0895 | 40GB | 2 x M.2 SATA RAID 1 (System/VMs) | 4 x M.2 NMVe QM2-4P-384A RAID 5 (Cache) | 5 x 4TB HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-663 | F/W: 4.4.2.1320 | 16GB | 2 x M.2 NMVe QM2-2P RAID 1 (Cache) | 4 x 4TB RAID 5
Apps: Boinc, Squid, DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS, Entware, DLstation, +others

fantomas
Experience counts
Posts: 1549
Joined: Mon Feb 07, 2011 5:40 am
Location: Bratislava, Slovakia
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by fantomas » Wed Mar 06, 2013 4:20 am

gPaq wrote:Although this tip is very much appreciated, and all the commands work on my newly purchased TS-469 Pro, making the changes on "speed_limit_min" makes not a lick of difference, unfortunately. I increased this speed gradually from the default 5000 to 50000, up to 150000, and basically nothing happens. The processor utilization does not go up (hovers around 16% before and after changes), and after 11 hours of re-striping I'm at 17%, which puts my ETA at about 4 more days to completion. That's not reasonable anymore... Is there anything that puts a damper on this configuration in newer models?


These options just tune priority for RAID reshaping. You can tell the kernel how much of disk I/O it should reserver for resyncing and other processes. They won't make your NAS go faster than it can - it's limited by CPU and DISK speeds, you can not tune those by software.
experience with administration of UN*X (mostly linux) and applications on internet servers since 1994...

slickdawg
New here
Posts: 9
Joined: Fri Mar 15, 2013 10:53 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by slickdawg » Tue Mar 19, 2013 3:31 am

Well, for giggles I decided to try this out on my new TS869 which has been plagued with issues new out of the box.

8 * 4TB Hitachi 7200RPM drives, all as one big RAID6 volume. The synchronization of this takes 40+ hours


I've made subtle changes, with no significant results. So I decided to go crazy and see what happened.

Max = 800000
Min = 200000

[/proc/sys/dev/raid] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md8 : active raid1 sdh2[2](S) sdg2[3](S) sdf2[4](S) sde2[5](S) sdd2[6](S) sdc2[7](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]

md0 : active raid6 sdh3[7] sdg3[6] sdf3[5] sde3[4] sdd3[3] sdc3[2] sdb3[1] sda3[0]
23432697216 blocks super 1.0 level 6, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]
[======>..............] resync = 31.0% (1211472968/3905449536) finish=1668.9min speed=26902K/sec

md13 : active raid1 sdh4[3](S) sdg4[4](S) sdf4[5](S) sde4[6](S) sdd4[7](S) sdc4[2] sdb4[1] sda4[0]
458880 blocks [3/3] [UUU]
bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
530048 blocks [8/8] [UUUUUUUU]
bitmap: 1/65 pages [4KB], 4KB chunk

unused devices: <none>



CPU usage is hanging around 42% , temps are normal too. I imagine I'm waiting on disks.




If you are interested in my particular TS869 issue, here's that thread:

viewtopic.php?f=45&t=72988

fantomas
Experience counts
Posts: 1549
Joined: Mon Feb 07, 2011 5:40 am
Location: Bratislava, Slovakia
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by fantomas » Tue Mar 19, 2013 4:19 am

Have you tried to tune /sys/block/md0/md/stripe_cache_size if it helps? I think 26902K/sec is quite low for such disks
experience with administration of UN*X (mostly linux) and applications on internet servers since 1994...

slickdawg
New here
Posts: 9
Joined: Fri Mar 15, 2013 10:53 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by slickdawg » Tue Mar 19, 2013 5:08 am

It's at the default of 4096. Any suggestions of a value to use?

fantomas
Experience counts
Posts: 1549
Joined: Mon Feb 07, 2011 5:40 am
Location: Bratislava, Slovakia
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by fantomas » Tue Mar 19, 2013 4:58 pm

above in this thread I mentioned increasing to 16384 increased the build speed. you can try 8192, 16384, 32768 if they will help (wait some time after each change to let it work). You just need to be careful - it eats system memory, so when it stops looking better, leave it or change to last smaller value
experience with administration of UN*X (mostly linux) and applications on internet servers since 1994...

slickdawg
New here
Posts: 9
Joined: Fri Mar 15, 2013 10:53 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by slickdawg » Wed Mar 20, 2013 11:01 am

After another roughly 40 hours, the sync finished. I've enabled the bitmap, hopefully it's "good" this time.

AUSTraveler
New here
Posts: 6
Joined: Sun Jan 17, 2010 9:46 am

Re: [HOWTO] How to increase raid rebuild speed

Post by AUSTraveler » Mon May 06, 2013 12:11 am

TS-809 Pro running FW 3.8.2. Upgraded hard drives this past week and now performing a RAID6 reshape from 4 drives to 5 drives (all Seagate ST4000DM000s). Started the process last night, and it was only 20% when I woke up (according to mdstat). Found this thread and tried a few different values for speed_limit_min, speed_limit_max, and stripe_cache_size.

CPU core 1 was between 19% and 25% for all values tested (running md0_raid6). CPU core 2 was between 4% and 7% (running md0_reshape). CPU has headroom, and I've stopped all non-esssential services.

Best speed I got was 23.8K/sec on the reshape, and that was with the default values (min: 50000, max: 100000, stripe_cache_size: 4096). Increasing the min had very little effect. Increasing the stripe_cache_size used more memory but dropped speed of reshape as follows

stripe_cache_size: 8192 --> speed: 22.2K/sec
stripe_cache_size: 16384 --> speed: 20K/sec

So, I've gone back to the defaults for now and will wait out the remaining ~37 hours.

Also, when I tried to lower stripe_cache_size, I found I had to set 4096 with the echo command several times before it would drop back down to 4096.

User avatar
doktornotor
Ask me anything
Posts: 7519
Joined: Tue Apr 24, 2012 5:44 am

Re: [HOWTO] How to increase raid rebuild speed

Post by doktornotor » Mon May 06, 2013 12:17 am

AUSTraveler wrote:Upgraded hard drives this past week and now performing a RAID6 reshape from 4 drives to 5 drives (all Seagate ST4000DM000s).


No comment. :roll: :roll: :roll:
I'm gone from this forum till QNAP stop wasting volunteers' time. Get help from QNAP helpdesk instead.
Warning: offensive signature and materials damaging QNAP reputation follow:
QNAP's FW security issues
QNAP's hardware compatibility list madness
QNAP's new logo competition
Dear QNAP, kindly fire your clueless incompetent forum "admin" And while at it, don't forget the webmaster!

AUSTraveler
New here
Posts: 6
Joined: Sun Jan 17, 2010 9:46 am

Re: [HOWTO] How to increase raid rebuild speed

Post by AUSTraveler » Mon May 06, 2013 3:06 am

doktornotor wrote:
AUSTraveler wrote:Upgraded hard drives this past week and now performing a RAID6 reshape from 4 drives to 5 drives (all Seagate ST4000DM000s).


No comment. :roll: :roll: :roll:


Then why post?

musashi77
Starting out
Posts: 17
Joined: Mon Jan 28, 2008 8:02 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by musashi77 » Wed May 08, 2013 3:23 pm

There's one more option for rebuilding RAID5/RAID6 devices:

# cat /sys/block/md0/md/stripe_cache_size
4096

increasing this could help, last time I have tried 16384 (4x more).


What is the actual command to use to increase this value? my TS-409 is set to 256.

Changing the min speed to value 50000 hasnt changed anything

User avatar
schumaku
Guru
Posts: 43664
Joined: Mon Jan 21, 2008 4:41 pm
Location: Kloten (Zurich), Switzerland -- Skype: schumaku
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by schumaku » Thu May 09, 2013 1:31 am

musashi77 wrote:What is the actual command to use to increase this value? my TS-409 is set to 256.
most discussions hae are related to newer NAS models. Be aware things are changing on the newer firmware releases once more for the models still under maintenance.

Lack of a TS-409 ... is this the output you see the 256?

# cat /sys/block/md0/md/stripe_cache_size
256

To change, you can echo a number in:

# echo 512 > /sys/block/md0/md/stripe_cache_size
# cat /sys/block/md0/md/stripe_cache_size
...

musashi77 wrote:Changing the min speed to value 50000 hasnt changed anything
You are aware that your TS-409 is not a racing machine?

musashi77
Starting out
Posts: 17
Joined: Mon Jan 28, 2008 8:02 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by musashi77 » Thu May 09, 2013 8:15 am

Correct the output is 256.

# cat /sys/block/md0/md/stripe_cache_size
256

However the following cmmand does nothing to change it:

# echo 512 > /sys/block/md0/md/stripe_cache_size
# cat /sys/block/md0/md/stripe_cache_size
256

Im aware that the 409 isnt going to break any land speed records, and had anticaped it would take some time, however it was initially looking like it would take more then 2 weeks, that is really pushing the limitations of my (and the girlfriends) patience. Tweaking the speed_limit_min to 50000-80000 has doubled the speed and effectively halved the time. Still not great though, speed=5514K/sec.

EDIT: I mentioned earlier that changing the speed_limit_min to 50000 didnt change anything, however it did increase the speeds from around 2000K/sec up to 5-6000K/sec. Ill take whatever I can get at this stage.

User avatar
schumaku
Guru
Posts: 43664
Joined: Mon Jan 21, 2008 4:41 pm
Location: Kloten (Zurich), Switzerland -- Skype: schumaku
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by schumaku » Fri May 10, 2013 1:12 am

That's about all what is possible...

Locked

Return to “Turbo Station Installation & Setup”