[HOWTO] How to increase raid rebuild speed

Discussion on setting up QNAP NAS products.
Locked
jad albert
New here
Posts: 2
Joined: Thu Oct 20, 2016 10:53 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by jad albert »

[color=#000000]P3R[/color] wrote:
[color=#000000]Ifti[/color] wrote:Same here - adjusted mine to 200000 and my CPU doesn't go over 2%, so I can only assume its not working!!
My assumption would instead be that your very fast CPU can easily manage this task already without changing this setting and that your disks are the performance bottlenecks.
I have a TVS-871T (i7 model)
This thread was started in a time when the fastest NAS CPUs had someting like 1/15th or 1/20th of the processing capacity your CPU have.
[~] # egrep speed /proc/mdstat
[=>...................] resync = 9.1% (534795552/5850567168) finish=639.6min speed=138511K/sec
Looks decent to me. What's wrong with it?
Thanks for sharing!
Last edited by jad albert on Fri Oct 21, 2016 8:05 pm, edited 1 time in total.
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: [HOWTO] How to increase raid rebuild speed

Post by Trexx »

Just a note: For migrations from Raid1-Raid5, the first 50% of the rebuild will tend to be slow, not sure specially was going on behind the scenes,but once it hits the 50%+ your throughput should increase significantly. In my case, it was about 3-4x of that it was between 0-50%.
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
User avatar
avvidme
Know my way around
Posts: 185
Joined: Fri Jan 16, 2009 10:36 am
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by avvidme »

Hi everyone, I'm doing a Raid 5 to 6 migration on a TS-853 Pro on new WD 4TB drives (7 in total).

Here's the status:
[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md1 : active raid6 sdf3[6] sdg3[0] sdd3[5] sdc3[4] sdb3[3] sda3[2] sdh3[1]
19485318080 blocks super 1.0 level 6, 64k chunk, algorithm 18 [7/6] [UUUUUU_]
[====>................] reshape = 21.0% (820684800/3897063616) finish=16520.9min speed=3103K/sec

No matter ANY setting I use such as
echo 800000 >/proc/sys/dev/raid/speed_limit_max
echo 400000 >/proc/sys/dev/raid/speed_limit_min

And cache size up to 32k. The speed of 3103k just won't change, and the CPU utilization is next to nothing.
After 4 days, I still have about 10 days to go!!!!!

HELP! is there ANY other way to increase the speed of this migration???
Yes I have Bitmap on, but is it safe to turn it off during the migration?

Help!
_______________________________________________________________________
Checkout http://www.digitalgatehouse.com for more of my ramblings!
User avatar
Spider99
Experience counts
Posts: 1951
Joined: Fri Oct 21, 2011 11:14 pm
Location: UK

Re: [HOWTO] How to increase raid rebuild speed

Post by Spider99 »

missing disk is affecting rebuild/migration?
Tim

TS-853A(16GB): - 4.3.4.0483 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R »

avvidme wrote:The speed of 3103k just won't change, and the CPU utilization is next to nothing.
I wouldn't worry about the low CPU utilization. Today's CPUs are much faster so are unlikely to be a bottleneck. It's usually the disks that's the limitation.

Do you have anything else active that access the disks? Active clients in the network and/or active services/apps on the NAS? Virtualization, torrent downloading or anything else?
After 4 days, I still have about 10 days to go!!!!!
That prediction isn't necessarily accurate.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
User avatar
avvidme
Know my way around
Posts: 185
Joined: Fri Jan 16, 2009 10:36 am
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by avvidme »

Thanks for your reply;

Do you have anything else active that access the disks? Active clients in the network and/or active services/apps on the NAS? Virtualization, torrent downloading or anything else?
After 4 days, I still have about 10 days to go!!!!!
That prediction isn't necessarily accurate.[/quote]

Nothing, I shut off all applications before I started the migration. Bare bones services running.
_______________________________________________________________________
Checkout http://www.digitalgatehouse.com for more of my ramblings!
User avatar
avvidme
Know my way around
Posts: 185
Joined: Fri Jan 16, 2009 10:36 am
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by avvidme »

Spider99 wrote:missing disk is affecting rebuild/migration?
No missing disk, it's an online spare.

J
_______________________________________________________________________
Checkout http://www.digitalgatehouse.com for more of my ramblings!
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R »

avvidme wrote:No missing disk, it's an online spare.
In the manual for the older cat1 models you can find this:
Note: A hot spare drive must be removed from the disk volume before executing the following action:
• Online RAID capacity expansion
• Online RAID level migration
• Adding a hard drive member to a RAID 5, RAID 6 or RAID 10 volume


I would have expected the same to apply to cat2 as well but I don't know. :S
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
User avatar
avvidme
Know my way around
Posts: 185
Joined: Fri Jan 16, 2009 10:36 am
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by avvidme »

P3R wrote:
avvidme wrote:No missing disk, it's an online spare.
In the manual for the older cat1 models you can find this:
Note: A hot spare drive must be removed from the disk volume before executing the following action:
• Online RAID capacity expansion
• Online RAID level migration
• Adding a hard drive member to a RAID 5, RAID 6 or RAID 10 volume


I would have expected the same to apply to cat2 as well but I don't know. :S
Hi, sorry I should've been more specific - the drive is physically in the slot, but it's NOT configured for anything yet. It'll become an online spare once the RAID 6 migration is done.

I'm just blown away that I'm only getting 3100k from an 853. I even set the min/max to very high numbers (800000, etc) and no effect.
Just stumped here.
_______________________________________________________________________
Checkout http://www.digitalgatehouse.com for more of my ramblings!
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R »

avvidme wrote:I even set the min/max to very high numbers (800000, etc) and no effect.
If a car doesn't have the performance to reach the speed limit, a raised limit won't make it go faster. The question is why your 853 moves at Trabant speed?

A bad disk slowing things down maybe? You could check the SMART data on all disks to see if that gives you any hint.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
User avatar
avvidme
Know my way around
Posts: 185
Joined: Fri Jan 16, 2009 10:36 am
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by avvidme »

P3R wrote: A bad disk slowing things down maybe? You could check the SMART data on all disks to see if that gives you any hint.
Smart status looks perfect across all of them. They are the WD RED 4TB WD40EFRX drives, SATA3. Normal NAS performance (read/write) has been just fine under RAID 5.
But I decided to migrate to 6 just for better data protection. It's been 5 days running non-stop so far, I'm starting to believe it'll be another 9 days as estimated.

I guess there's no other speed adjustments outside of min/max/cache?
Thanks again
J
_______________________________________________________________________
Checkout http://www.digitalgatehouse.com for more of my ramblings!
User avatar
avvidme
Know my way around
Posts: 185
Joined: Fri Jan 16, 2009 10:36 am
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by avvidme »

Unfortunately, I have bitmap on right now which I'm sure isn't helping. Is it possible to turn it off WHILE the migration is running?

Doesn't seem to be an option in the GUI right now since a number of menu items aren't available during migration. If it's safe to do, is there a command line option?

Thanks again
_______________________________________________________________________
Checkout http://www.digitalgatehouse.com for more of my ramblings!
User avatar
Spider99
Experience counts
Posts: 1951
Joined: Fri Oct 21, 2011 11:14 pm
Location: UK

Re: [HOWTO] How to increase raid rebuild speed

Post by Spider99 »

you are casting about in the dark - bitmap helps in raid rebuild

you have a bad disk or some other config issue thats slowing the reshape - if your array was "full" before the start that will have a detrimental effect

when i built my 8 disk raid 5 - new no data it took a day and a half on my 853 to sync - with speeding it up - you are reshaping - a lot more complicated and significant change and hence slower process - actually i have a hunch the speed up options have little or no effect on a reshape

I also suspect once it gets past 50% and /or the data it will speed up a lot
Tim

TS-853A(16GB): - 4.3.4.0483 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
chodaboy19
Know my way around
Posts: 224
Joined: Thu Nov 26, 2009 4:28 am

Re: [HOWTO] How to increase raid rebuild speed

Post by chodaboy19 »

Code: Select all

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md1 : active raid5 sdd3[11] sdc3[10] sdb3[9] sda3[8] sdf3[7] sde3[6]
      39020358080 blocks super 1.0 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
      [============>........]  resync = 60.7% (4741593912/7804071616) finish=602.8min speed=84663K/sec

md256 : active raid1 sdd2[7](S) sdc2[6] sdb2[2] sda2[3](S) sdf2[4](S) sde2[5](S)
      530112 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md13 : active raid1 sdd4[29] sdc4[28] sdb4[27] sda4[26] sdf4[25] sde4[24]
      458880 blocks super 1.0 [24/6] [UUUUUU__________________]
      bitmap: 1/1 pages [4KB], 65536KB chunk

md9 : active raid1 sdd1[29] sdc1[28] sdb1[27] sda1[26] sdf1[25] sde1[24]
      530112 blocks super 1.0 [24/6] [UUUUUU__________________]
      bitmap: 1/1 pages [4KB], 65536KB chunk
Upgrading from 6 x 4TB to 6 x 8TB, I set the min to 85000 and it's working nicely!

TVS-672N h5.0.1.2248 build 20221215
i7-8700T | 64GB DDR4 (2 x M471A4G43MB1-CTD) | 2 x Samsung 970 Evo Plus 1.0TB | 6 x 18TB HGST HC550
User avatar
avvidme
Know my way around
Posts: 185
Joined: Fri Jan 16, 2009 10:36 am
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by avvidme »

chodaboy19 wrote:Upgrading from 6 x 4TB to 6 x 8TB, I set the min to 85000 and it's working nicely!
I wasn't just expanding the raid, I was also converting from Raid 5 to 6.
J
_______________________________________________________________________
Checkout http://www.digitalgatehouse.com for more of my ramblings!
Locked

Return to “Turbo Station Installation & Setup”