[HOWTO] How to increase raid rebuild speed

Discussion on setting up QNAP NAS products.
Locked
User avatar
avvidme
Know my way around
Posts: 185
Joined: Fri Jan 16, 2009 10:36 am
Contact:

Re: [HOWTO] How to increase raid rebuild speed

Post by avvidme »

Well, it finally finished. TS-859 Raid 5 to 6 migration plus 1 extra drive (expansion) = 17 days.
All services turned off.

And yet my throughput performance is now back to normal.

Unfreakin believable.
_______________________________________________________________________
Checkout http://www.digitalgatehouse.com for more of my ramblings!
solomonshv
Starting out
Posts: 24
Joined: Thu Feb 02, 2017 4:20 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by solomonshv »

i can't thank you enough for finding this. I am upgrading my TS 451 from WD 5TB Red drives (that are coming out of warranty) to 8TB Red Pro drives. i was swapping the drives 1 at a time. i set min to 150,000 and max to 300,000. this increased the rebuild speed THIRTY FOLD!!! as in, 30 times faster. and the CPU load only hovers between 40 to 50%.

i'm blown away. not by how much this helped or how easy it was, but blown away by how f**king stupid QNAP are for this oversight. the default minimum on my 451 was 5000 (5MB/s) and that's pretty much the speed it was moving at. who thought this was a good idea??? at this rate, if a drive failed, there is a pretty good chance a second drive will fall out of RAID in the WEEKS it would take to rebuild
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R »

solomonshv wrote:I am upgrading my TS 451 from WD 5TB Red drives (that are coming out of warranty)...
Well everything is coming out of warranty eventually but doesn't a 5 TB WD Red have at the very least 5 month left on the 3 year warranty even if you were among the very first customers to receive it?

When did you buy yours?
...at this rate, if a drive failed, there is a pretty good chance a second drive will fall out of RAID in the WEEKS it would take to rebuild
I disagree about it beeing "...a pretty good chance...".

Faster is of course better if your NAS have nothing better to do than to devote almost all resources toward a rebuild but I don't think that hours, days or even weeks for a rebuild will affect the risk for an additional disk failure more than very marginally. The big risk with a rebuild is that any of the other disks have bad sectors already and the few per cent of additional time even a multi week rebuild would affect that risk are only those few per cent. I don't see how that could be the same as "...a pretty good chance...".
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
rafarataneneces
New here
Posts: 5
Joined: Sun Mar 19, 2017 3:23 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by rafarataneneces »

avvidme wrote:Well, it finally finished. TS-859 Raid 5 to 6 migration plus 1 extra drive (expansion) = 17 days.
All services turned off.

And yet my throughput performance is now back to normal.

Unfreakin believable.
Did this happened with the default values?

/proc/sys/dev/raid/speed_limit_max
200000
/proc/sys/dev/raid/speed_limit_min
1000
User avatar
Spider99
Experience counts
Posts: 1951
Joined: Fri Oct 21, 2011 11:14 pm
Location: UK

Re: [HOWTO] How to increase raid rebuild speed

Post by Spider99 »

this trick only works on a rebuild not a reshape so will not affect a change from r5 to r6
Tim

TS-853A(16GB): - 4.3.4.0483 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
User avatar
oyvindo
Experience counts
Posts: 1399
Joined: Tue May 19, 2009 2:08 am
Location: Norway, Oslo

Re: [HOWTO] How to increase raid rebuild speed

Post by oyvindo »

This tip worked great on my TS-543Mini still - 8 years after first post. :D
Great Smile....
ImageImageImage
User avatar
JohnSmith755
New here
Posts: 3
Joined: Wed Jun 14, 2017 4:47 pm

Re: [HOWTO] How to increase raid rebuild speed

Post by JohnSmith755 »

I'm glad I found this important information, thanks for posting!
“Lesser, greater, middling, it's all the same. Proportions are negotiated, boundaries blurred. I'm not a pious hermit, I haven't done only good in my life. But if I'm to choose between one evil and another, then I prefer not to choose at all.”
― Andrzej Sapkowski, The Last Wish
Amedee
Starting out
Posts: 16
Joined: Sun Jan 13, 2013 1:54 am

Re: [HOWTO] How to increase raid rebuild speed

Post by Amedee »

Don wrote:I found this article on the web. Use it at your own risk as I have not tested it and make no guarantees. If yo do try it please post your results here so others may benefit.
Increase the speed of Linux Software RAID reconstruction

If you are in a situation where you sit in front of the console (or on a remote ssh connection) waiting for a Linux software RAID to finish rebuilding (either you added a new drive, or you replaced a failed one, etc.) then you might be frustrated by how slow this process is running. You are running cat on /proc/mdstat repeatedly (you should really use watch in this case ;)), and this seems to never finish… Obviously that there is a logical reason for this ‘slowness‘ and on a production system you should leave it running with the defaults. But in case you want to speed up this process here is how you can do it. This will place a much higher load on the system so you should use it with care.

To see your Linux kernel speed limits imposed on the RAID reconstruction use:

Code: Select all

cat /proc/sys/dev/raid/speed_limit_max
200000
cat /proc/sys/dev/raid/speed_limit_min
1000
In the system logs you can see something similar to:

Code: Select all

md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.
This means that the minimum guaranteed speed of the rebuild of the array is approx 1MB/s. The actual speed will be higher and will depend on the system load and what other processes are running at that time.
In case you want to increase this minimum speed you need to enter a higher value in speed_limit_min. For example to set this to approx 50 megabytes per second as minimum use:

Code: Select all

echo 50000 >/proc/sys/dev/raid/speed_limit_min
The results are instant… you can return to the watch window to see it running, and hope that this will finish a little faster (this will really depend on the system you are running, the HDDs, controllers, etc.):

Code: Select all

watch cat /proc/mdstat
There is no watch program on the NAS so that part won't work. You will have to play with the numbers until you get one that works for you.

Article is located here --> http://www.ducea.com/2006/06/25/increas ... struction/
You can still use watch. I suppose that most of y'all are Windows users, so first install Linux-on-Windows, which gives you an Ubuntu inside Windows.
For those of you who are already on Linux, you're almost there.

First you need to install sshpass (in Linux, or in Linux-on-Windows):

Code: Select all

sudo apt install sshpass
Then you type:

Code: Select all

SSHPASS=<YOUR_SUPER_SECRET_PASSWORD> \
NAS_IP=<YOUR_NAS_IP_ADDRESS> \
watch sshpass -e ssh -o PubkeyAuthentication=no -o StrictHostKeyChecking=no admin@$NAS_IP "cat /proc/mdstat"
Output:

Code: Select all

Every 2,0s: sshpass -e ssh -o PubkeyAuthentication=no -o StrictHostKeyChecking=no admin@192.168.178.34 cat /proc/mdstat

Personalities : [raid1] [linear] [raid0] [raid10] [raid6] [raid5] [raid4] 
md0 : active raid6 sda3[0] sdd3[3] sdc3[2] sdb3[1]
		 3903891072 blocks super 1.0 level 6, 64k chunk, algorithm 2 [4/4] [UUUU]
		 [==========>..........]  resync = 53.4% (1044127808/1951945536) finish=923.6min speed=16380K/sec
		 
md4 : active raid1 sdd2[4](S) sdc2[3](S) sdb2[2] sda2[0]
		 530128 blocks super 1.0 [2/2] [UU]
		 
md13 : active raid1 sda4[0] sdc4[5] sdd4[4] sdb4[3]
		 458880 blocks super 1.0 [4/4] [UUUU]
		 bitmap: 0/8 pages [0KB], 32KB chunk

md9 : active raid1 sda1[0] sdc1[6] sdd1[5] sdb1[4]
		 530112 blocks super 1.0 [4/4] [UUUU]
		 bitmap: 1/9 pages [4KB], 32KB chunk

unused devices: <none>
SECURITY CONSIDERATIONS
First and foremost, users of sshpass should realize that ssh's insistance on only getting the password interactively is not without reason. It is close to impossible to securely store the
password, and users of sshpass should consider whether ssh's public key authentication provides the same end-user experience, while involving less hassle and being more secure.
Oscar_I
New here
Posts: 3
Joined: Mon Jan 04, 2016 2:22 am

Re: [HOWTO] How to increase raid rebuild speed

Post by Oscar_I »

After reading this thread, I still haven't managed to get my raid-rebuild speed to increase to acceptable level

The problem is that the rebuild speed is only between 1000-1800k/Sec (with a time to finish of about 22days!!??)


I had a QNAP TS-453-Pro configured as RAID10 with four 4TB WD-red disks and I am now in the process of replacing them to four 8 TB disks (currently at ~93% disk usage)
The first two disks went fairly fast (although it took a few days for each disk), but now after I inserted the third, the speed seems to be even slower than before.

All disks are in "good" shape according to the internal disk monitor.
I have upgraded the internal firmware about a month ago.
The NAS is not used for anything else than backup, and I have paused the backup during the raid-rebuild.

I have tried different values for /proc/sys/dev/raid/speed_limit_min - but it doesn't seem to have much impact
When I run "top" the stat for MD1_resync is set as DWN - and almost no CPU usage.

Does the number of files impact the rebuild speed? I have a fair amount of rather small files (~32k), could that be a reason?
Does anyone have any clue what might be wrong or have any ideas of what I can try to I can increase the rebuild speed?



Thanks
Oscar
User avatar
dolbyman
Guru
Posts: 35015
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: [HOWTO] How to increase raid rebuild speed

Post by dolbyman »

maybe the raidpair drive has issues..raid10 rebuild is a simple copy process no parity calc operations needed.

And the first two disks went ok speedwise you said.
So be patient and hope the other drive does not bite the dust.
Oscar_I
New here
Posts: 3
Joined: Mon Jan 04, 2016 2:22 am

Re: [HOWTO] How to increase raid rebuild speed

Post by Oscar_I »

But wouldn't that be shown in the disk monitor ?
All disks are in "good shape" according to the QNAP webinterface
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R »

Oscar_I wrote:But wouldn't that be shown in the disk monitor ?
Often but not always.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
Oscar_I
New here
Posts: 3
Joined: Mon Jan 04, 2016 2:22 am

Re: [HOWTO] How to increase raid rebuild speed

Post by Oscar_I »

P3R wrote:
Oscar_I wrote:But wouldn't that be shown in the disk monitor ?
Often but not always.
Any suggestion of where I can get more info about the status (and possibly determine what drive that is that has issues)

When I looked at the output from

Code: Select all

>top
it seems as the rebuild process is pretty much dormant...

Code: Select all

Mem: 1738052K used, 192488K free, 203784K shrd, 217152K buff, 227064K cached
CPU:  7.6% usr 25.7% sys  0.0% nic 37.7% idle 28.2% io  0.0% irq  0.6% sirq
Load average: 20.59 19.58 16.43 3/569 29681
PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
...
20283     2 admin    DWN      0  0.0   3  0.1 [md1_resync]
it seem as the "DWN" means (found on a different forum)

Code: Select all

D - uninterruptible sleep (usually IO)
W - paging (not valid since the 2.6.xx kernel)
N - low-priority (nice to other users)
Is this a sign of a faulty hard drive?
or can I force the QNAP TS-453 Pro to increase the priority?
(the NAS is currently not being used due to the low diskspace)

Thanks!
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [HOWTO] How to increase raid rebuild speed

Post by P3R »

Oscar_I wrote:Any suggestion of where I can get more info about the status (and possibly determine what drive that is that has issues)
I would begin with comparing the raw values of the detailed SMART data (Storage Manager, Storage, Disks/VJBOD, [select the disk in question], Disk Health, SMART Information) between all the disks to see if something is significantly different with this disk.

Other than that unfortunately the only suggestion I have would be to run the diagnostic software of the manufacturer on the disk but that would have to be done in a different computer so isn't the best now in the middle of a current storage expansion... :cry:

I would probably contact the proper channel for a possible warranty replacement of the disk (reseller or disk manufacturer depending on where in the world you're located) and ask for their advice.

Are these 8 TB disks also WD Red? If not, what brand and model?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
gandalf67
Starting out
Posts: 20
Joined: Tue Mar 06, 2018 6:04 am

Re: [HOWTO] How to increase raid rebuild speed

Post by gandalf67 »

Hello,

Just purchased a TS-431P, updated to the latest firmware, installed 4 x 8TB drives (Seagate) and configured a Raid-5 volume. Initial sync is awfully slow, and speed does not change when I change the sync priority:

cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md1 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[0]
23412214272 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
[>....................] resync = 2.1% (166559964/7804071424) finish=103654.7min speed=1228K/sec

That's 71 days!

I tried to increase the min and max speed as described here, no result. CPU is almost idle, with the md1_raid5 process using between 0.5 and 3% cpu.

Web UI is slow too, everytime I commit a change, it takes a long time, or even times out/fails, like trying to change the admin password, or trying to reboot. Think I'm going to try a hard reset.

Edit: finally managed to reboot the NAS, it restarted rebuild, this time between 15 and 30 Mbytes/s, which of course had changed more or less 71 days to 71 hours, much better, even if it's a bit slow, but on my previous NAS (Lenovo ix4-300D) it took 2 days for 4x4TB Raid-5 so this seems consistent.

Thanks.
Locked

Return to “Turbo Station Installation & Setup”