[HOWTO] How to increase raid rebuild speed

[jad albert]

Same here - adjusted mine to 200000 and my CPU doesn't go over 2%, so I can only assume its not working!!

My assumption would instead be that your very fast CPU can easily manage this task already without changing this setting and that your disks are the performance bottlenecks.

I have a TVS-871T (i7 model)

This thread was started in a time when the fastest NAS CPUs had someting like 1/15th or 1/20th of the processing capacity your CPU have.

[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec

Looks decent to me. What's wrong with it?

Thanks for sharing!

[Trexx]

Just a note: For migrations from Raid1-Raid5, the first 50% of the rebuild will tend to be slow, not sure specially was going on behind the scenes,but once it hits the 50%+ your throughput should increase significantly. In my case, it was about 3-4x of that it was between 0-50%.

[avvidme]

Hi everyone, I'm doing a Raid 5 to 6 migration on a TS-853 Pro on new WD 4TB drives (7 in total).

Here's the status:
[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md1 : active raid6 sdf3[6] sdg3[0] sdd3[5] sdc3[4] sdb3[3] sda3[2] sdh3[1]
19485318080 blocks super 1.0 level 6, 64k chunk, algorithm 18 [7/6] [UUUUUU_]
[====>................] reshape = 21.0% (820684800/3897063616) finish=16520.9min speed=3103K/sec

No matter ANY setting I use such as
echo 800000 >/proc/sys/dev/raid/speed_limit_max
echo 400000 >/proc/sys/dev/raid/speed_limit_min

And cache size up to 32k. The speed of 3103k just won't change, and the CPU utilization is next to nothing.
After 4 days, I still have about 10 days to go!!!!!

HELP! is there ANY other way to increase the speed of this migration???
Yes I have Bitmap on, but is it safe to turn it off during the migration?

Help!

[Spider99]

missing disk is affecting rebuild/migration?

[P3R]

The speed of 3103k just won't change, and the CPU utilization is next to nothing.

I wouldn't worry about the low CPU utilization. Today's CPUs are much faster so are unlikely to be a bottleneck. It's usually the disks that's the limitation.

Do you have anything else active that access the disks? Active clients in the network and/or active services/apps on the NAS? Virtualization, torrent downloading or anything else?

After 4 days, I still have about 10 days to go!!!!!

That prediction isn't necessarily accurate.

[avvidme]

Thanks for your reply;

Do you have anything else active that access the disks? Active clients in the network and/or active services/apps on the NAS? Virtualization, torrent downloading or anything else?

After 4 days, I still have about 10 days to go!!!!!

That prediction isn't necessarily accurate.[/quote]

Nothing, I shut off all applications before I started the migration. Bare bones services running.

[avvidme]

missing disk is affecting rebuild/migration?

No missing disk, it's an online spare.

J

[P3R]

No missing disk, it's an online spare.

In the manual for the older cat1 models you can find this:
Note: A hot spare drive must be removed from the disk volume before executing the following action:
• Online RAID capacity expansion
• Online RAID level migration
• Adding a hard drive member to a RAID 5, RAID 6 or RAID 10 volume

I would have expected the same to apply to cat2 as well but I don't know.

[avvidme]

No missing disk, it's an online spare.

In the manual for the older cat1 models you can find this:
Note: A hot spare drive must be removed from the disk volume before executing the following action:
• Online RAID capacity expansion
• Online RAID level migration
• Adding a hard drive member to a RAID 5, RAID 6 or RAID 10 volume

I would have expected the same to apply to cat2 as well but I don't know.

Hi, sorry I should've been more specific - the drive is physically in the slot, but it's NOT configured for anything yet. It'll become an online spare once the RAID 6 migration is done.

I'm just blown away that I'm only getting 3100k from an 853. I even set the min/max to very high numbers (800000, etc) and no effect.
Just stumped here.

[P3R]

I even set the min/max to very high numbers (800000, etc) and no effect.

If a car doesn't have the performance to reach the speed limit, a raised limit won't make it go faster. The question is why your 853 moves at Trabant speed?

A bad disk slowing things down maybe? You could check the SMART data on all disks to see if that gives you any hint.

[avvidme]

A bad disk slowing things down maybe? You could check the SMART data on all disks to see if that gives you any hint.

Smart status looks perfect across all of them. They are the WD RED 4TB WD40EFRX drives, SATA3. Normal NAS performance (read/write) has been just fine under RAID 5.
But I decided to migrate to 6 just for better data protection. It's been 5 days running non-stop so far, I'm starting to believe it'll be another 9 days as estimated.

I guess there's no other speed adjustments outside of min/max/cache?
Thanks again
J

[avvidme]

Unfortunately, I have bitmap on right now which I'm sure isn't helping. Is it possible to turn it off WHILE the migration is running?

Doesn't seem to be an option in the GUI right now since a number of menu items aren't available during migration. If it's safe to do, is there a command line option?

Thanks again

[Spider99]

you are casting about in the dark - bitmap helps in raid rebuild

you have a bad disk or some other config issue thats slowing the reshape - if your array was "full" before the start that will have a detrimental effect

when i built my 8 disk raid 5 - new no data it took a day and a half on my 853 to sync - with speeding it up - you are reshaping - a lot more complicated and significant change and hence slower process - actually i have a hunch the speed up options have little or no effect on a reshape

I also suspect once it gets past 50% and /or the data it will speed up a lot

[chodaboy19]

Code: Select all

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md1 : active raid5 sdd3[11] sdc3[10] sdb3[9] sda3[8] sdf3[7] sde3[6]
      39020358080 blocks super 1.0 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
      [============>........]  resync = 60.7% (4741593912/7804071616) finish=602.8min speed=84663K/sec

md256 : active raid1 sdd2[7](S) sdc2[6] sdb2[2] sda2[3](S) sdf2[4](S) sde2[5](S)
      530112 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md13 : active raid1 sdd4[29] sdc4[28] sdb4[27] sda4[26] sdf4[25] sde4[24]
      458880 blocks super 1.0 [24/6] [UUUUUU__________________]
      bitmap: 1/1 pages [4KB], 65536KB chunk

md9 : active raid1 sdd1[29] sdc1[28] sdb1[27] sda1[26] sdf1[25] sde1[24]
      530112 blocks super 1.0 [24/6] [UUUUUU__________________]
      bitmap: 1/1 pages [4KB], 65536KB chunk

Upgrading from 6 x 4TB to 6 x 8TB, I set the min to 85000 and it's working nicely!

[avvidme]

Upgrading from 6 x 4TB to 6 x 8TB, I set the min to 85000 and it's working nicely!

I wasn't just expanding the raid, I was also converting from Raid 5 to 6.
J