[HOWTO] How to increase raid rebuild speed
- avvidme
- Know my way around
- Posts: 185
- Joined: Fri Jan 16, 2009 10:36 am
- Contact:
Re: [HOWTO] How to increase raid rebuild speed
Well, it finally finished. TS-859 Raid 5 to 6 migration plus 1 extra drive (expansion) = 17 days.
All services turned off.
And yet my throughput performance is now back to normal.
Unfreakin believable.
All services turned off.
And yet my throughput performance is now back to normal.
Unfreakin believable.
_______________________________________________________________________
Checkout http://www.digitalgatehouse.com for more of my ramblings!
Checkout http://www.digitalgatehouse.com for more of my ramblings!
-
- Starting out
- Posts: 24
- Joined: Thu Feb 02, 2017 4:20 pm
Re: [HOWTO] How to increase raid rebuild speed
i can't thank you enough for finding this. I am upgrading my TS 451 from WD 5TB Red drives (that are coming out of warranty) to 8TB Red Pro drives. i was swapping the drives 1 at a time. i set min to 150,000 and max to 300,000. this increased the rebuild speed THIRTY FOLD!!! as in, 30 times faster. and the CPU load only hovers between 40 to 50%.
i'm blown away. not by how much this helped or how easy it was, but blown away by how f**king stupid QNAP are for this oversight. the default minimum on my 451 was 5000 (5MB/s) and that's pretty much the speed it was moving at. who thought this was a good idea??? at this rate, if a drive failed, there is a pretty good chance a second drive will fall out of RAID in the WEEKS it would take to rebuild
i'm blown away. not by how much this helped or how easy it was, but blown away by how f**king stupid QNAP are for this oversight. the default minimum on my 451 was 5000 (5MB/s) and that's pretty much the speed it was moving at. who thought this was a good idea??? at this rate, if a drive failed, there is a pretty good chance a second drive will fall out of RAID in the WEEKS it would take to rebuild
-
- Guru
- Posts: 13192
- Joined: Sat Dec 29, 2007 1:39 am
- Location: Stockholm, Sweden (UTC+01:00)
Re: [HOWTO] How to increase raid rebuild speed
Well everything is coming out of warranty eventually but doesn't a 5 TB WD Red have at the very least 5 month left on the 3 year warranty even if you were among the very first customers to receive it?solomonshv wrote:I am upgrading my TS 451 from WD 5TB Red drives (that are coming out of warranty)...
When did you buy yours?
I disagree about it beeing "...a pretty good chance..."....at this rate, if a drive failed, there is a pretty good chance a second drive will fall out of RAID in the WEEKS it would take to rebuild
Faster is of course better if your NAS have nothing better to do than to devote almost all resources toward a rebuild but I don't think that hours, days or even weeks for a rebuild will affect the risk for an additional disk failure more than very marginally. The big risk with a rebuild is that any of the other disks have bad sectors already and the few per cent of additional time even a multi week rebuild would affect that risk are only those few per cent. I don't see how that could be the same as "...a pretty good chance...".
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
-
- New here
- Posts: 5
- Joined: Sun Mar 19, 2017 3:23 pm
Re: [HOWTO] How to increase raid rebuild speed
Did this happened with the default values?avvidme wrote:Well, it finally finished. TS-859 Raid 5 to 6 migration plus 1 extra drive (expansion) = 17 days.
All services turned off.
And yet my throughput performance is now back to normal.
Unfreakin believable.
/proc/sys/dev/raid/speed_limit_max
200000
/proc/sys/dev/raid/speed_limit_min
1000
- Spider99
- Experience counts
- Posts: 1951
- Joined: Fri Oct 21, 2011 11:14 pm
- Location: UK
Re: [HOWTO] How to increase raid rebuild speed
this trick only works on a rebuild not a reshape so will not affect a change from r5 to r6
Tim
TS-853A(16GB): - 4.3.4.0483 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
TS-853A(16GB): - 4.3.4.0483 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
- oyvindo
- Experience counts
- Posts: 1399
- Joined: Tue May 19, 2009 2:08 am
- Location: Norway, Oslo
Re: [HOWTO] How to increase raid rebuild speed
This tip worked great on my TS-543Mini still - 8 years after first post.
Great Smile....
Great Smile....
- JohnSmith755
- New here
- Posts: 3
- Joined: Wed Jun 14, 2017 4:47 pm
Re: [HOWTO] How to increase raid rebuild speed
I'm glad I found this important information, thanks for posting!
“Lesser, greater, middling, it's all the same. Proportions are negotiated, boundaries blurred. I'm not a pious hermit, I haven't done only good in my life. But if I'm to choose between one evil and another, then I prefer not to choose at all.”
― Andrzej Sapkowski, The Last Wish
― Andrzej Sapkowski, The Last Wish
-
- Starting out
- Posts: 16
- Joined: Sun Jan 13, 2013 1:54 am
Re: [HOWTO] How to increase raid rebuild speed
You can still use watch. I suppose that most of y'all are Windows users, so first install Linux-on-Windows, which gives you an Ubuntu inside Windows.Don wrote:I found this article on the web. Use it at your own risk as I have not tested it and make no guarantees. If yo do try it please post your results here so others may benefit.
There is no watch program on the NAS so that part won't work. You will have to play with the numbers until you get one that works for you.Increase the speed of Linux Software RAID reconstruction
If you are in a situation where you sit in front of the console (or on a remote ssh connection) waiting for a Linux software RAID to finish rebuilding (either you added a new drive, or you replaced a failed one, etc.) then you might be frustrated by how slow this process is running. You are running cat on /proc/mdstat repeatedly (you should really use watch in this case ), and this seems to never finish… Obviously that there is a logical reason for this ‘slowness‘ and on a production system you should leave it running with the defaults. But in case you want to speed up this process here is how you can do it. This will place a much higher load on the system so you should use it with care.
To see your Linux kernel speed limits imposed on the RAID reconstruction use:
In the system logs you can see something similar to:Code: Select all
cat /proc/sys/dev/raid/speed_limit_max 200000 cat /proc/sys/dev/raid/speed_limit_min 1000
This means that the minimum guaranteed speed of the rebuild of the array is approx 1MB/s. The actual speed will be higher and will depend on the system load and what other processes are running at that time.Code: Select all
md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc. md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.
In case you want to increase this minimum speed you need to enter a higher value in speed_limit_min. For example to set this to approx 50 megabytes per second as minimum use:The results are instant… you can return to the watch window to see it running, and hope that this will finish a little faster (this will really depend on the system you are running, the HDDs, controllers, etc.):Code: Select all
echo 50000 >/proc/sys/dev/raid/speed_limit_min
Code: Select all
watch cat /proc/mdstat
Article is located here --> http://www.ducea.com/2006/06/25/increas ... struction/
For those of you who are already on Linux, you're almost there.
First you need to install sshpass (in Linux, or in Linux-on-Windows):
Code: Select all
sudo apt install sshpass
Code: Select all
SSHPASS=<YOUR_SUPER_SECRET_PASSWORD> \
NAS_IP=<YOUR_NAS_IP_ADDRESS> \
watch sshpass -e ssh -o PubkeyAuthentication=no -o StrictHostKeyChecking=no admin@$NAS_IP "cat /proc/mdstat"
Code: Select all
Every 2,0s: sshpass -e ssh -o PubkeyAuthentication=no -o StrictHostKeyChecking=no admin@192.168.178.34 cat /proc/mdstat
Personalities : [raid1] [linear] [raid0] [raid10] [raid6] [raid5] [raid4]
md0 : active raid6 sda3[0] sdd3[3] sdc3[2] sdb3[1]
3903891072 blocks super 1.0 level 6, 64k chunk, algorithm 2 [4/4] [UUUU]
[==========>..........] resync = 53.4% (1044127808/1951945536) finish=923.6min speed=16380K/sec
md4 : active raid1 sdd2[4](S) sdc2[3](S) sdb2[2] sda2[0]
530128 blocks super 1.0 [2/2] [UU]
md13 : active raid1 sda4[0] sdc4[5] sdd4[4] sdb4[3]
458880 blocks super 1.0 [4/4] [UUUU]
bitmap: 0/8 pages [0KB], 32KB chunk
md9 : active raid1 sda1[0] sdc1[6] sdd1[5] sdb1[4]
530112 blocks super 1.0 [4/4] [UUUU]
bitmap: 1/9 pages [4KB], 32KB chunk
unused devices: <none>
SECURITY CONSIDERATIONS
First and foremost, users of sshpass should realize that ssh's insistance on only getting the password interactively is not without reason. It is close to impossible to securely store the
password, and users of sshpass should consider whether ssh's public key authentication provides the same end-user experience, while involving less hassle and being more secure.
-
- New here
- Posts: 3
- Joined: Mon Jan 04, 2016 2:22 am
Re: [HOWTO] How to increase raid rebuild speed
After reading this thread, I still haven't managed to get my raid-rebuild speed to increase to acceptable level
The problem is that the rebuild speed is only between 1000-1800k/Sec (with a time to finish of about 22days!!??)
I had a QNAP TS-453-Pro configured as RAID10 with four 4TB WD-red disks and I am now in the process of replacing them to four 8 TB disks (currently at ~93% disk usage)
The first two disks went fairly fast (although it took a few days for each disk), but now after I inserted the third, the speed seems to be even slower than before.
All disks are in "good" shape according to the internal disk monitor.
I have upgraded the internal firmware about a month ago.
The NAS is not used for anything else than backup, and I have paused the backup during the raid-rebuild.
I have tried different values for /proc/sys/dev/raid/speed_limit_min - but it doesn't seem to have much impact
When I run "top" the stat for MD1_resync is set as DWN - and almost no CPU usage.
Does the number of files impact the rebuild speed? I have a fair amount of rather small files (~32k), could that be a reason?
Does anyone have any clue what might be wrong or have any ideas of what I can try to I can increase the rebuild speed?
Thanks
Oscar
The problem is that the rebuild speed is only between 1000-1800k/Sec (with a time to finish of about 22days!!??)
I had a QNAP TS-453-Pro configured as RAID10 with four 4TB WD-red disks and I am now in the process of replacing them to four 8 TB disks (currently at ~93% disk usage)
The first two disks went fairly fast (although it took a few days for each disk), but now after I inserted the third, the speed seems to be even slower than before.
All disks are in "good" shape according to the internal disk monitor.
I have upgraded the internal firmware about a month ago.
The NAS is not used for anything else than backup, and I have paused the backup during the raid-rebuild.
I have tried different values for /proc/sys/dev/raid/speed_limit_min - but it doesn't seem to have much impact
When I run "top" the stat for MD1_resync is set as DWN - and almost no CPU usage.
Does the number of files impact the rebuild speed? I have a fair amount of rather small files (~32k), could that be a reason?
Does anyone have any clue what might be wrong or have any ideas of what I can try to I can increase the rebuild speed?
Thanks
Oscar
- dolbyman
- Guru
- Posts: 35275
- Joined: Sat Feb 12, 2011 2:11 am
- Location: Vancouver BC , Canada
Re: [HOWTO] How to increase raid rebuild speed
maybe the raidpair drive has issues..raid10 rebuild is a simple copy process no parity calc operations needed.
And the first two disks went ok speedwise you said.
So be patient and hope the other drive does not bite the dust.
And the first two disks went ok speedwise you said.
So be patient and hope the other drive does not bite the dust.
-
- New here
- Posts: 3
- Joined: Mon Jan 04, 2016 2:22 am
Re: [HOWTO] How to increase raid rebuild speed
But wouldn't that be shown in the disk monitor ?
All disks are in "good shape" according to the QNAP webinterface
All disks are in "good shape" according to the QNAP webinterface
-
- Guru
- Posts: 13192
- Joined: Sat Dec 29, 2007 1:39 am
- Location: Stockholm, Sweden (UTC+01:00)
Re: [HOWTO] How to increase raid rebuild speed
Often but not always.Oscar_I wrote:But wouldn't that be shown in the disk monitor ?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
-
- New here
- Posts: 3
- Joined: Mon Jan 04, 2016 2:22 am
Re: [HOWTO] How to increase raid rebuild speed
Any suggestion of where I can get more info about the status (and possibly determine what drive that is that has issues)P3R wrote:Often but not always.Oscar_I wrote:But wouldn't that be shown in the disk monitor ?
When I looked at the output from
Code: Select all
>top
Code: Select all
Mem: 1738052K used, 192488K free, 203784K shrd, 217152K buff, 227064K cached
CPU: 7.6% usr 25.7% sys 0.0% nic 37.7% idle 28.2% io 0.0% irq 0.6% sirq
Load average: 20.59 19.58 16.43 3/569 29681
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
...
20283 2 admin DWN 0 0.0 3 0.1 [md1_resync]
Code: Select all
D - uninterruptible sleep (usually IO)
W - paging (not valid since the 2.6.xx kernel)
N - low-priority (nice to other users)
or can I force the QNAP TS-453 Pro to increase the priority?
(the NAS is currently not being used due to the low diskspace)
Thanks!
-
- Guru
- Posts: 13192
- Joined: Sat Dec 29, 2007 1:39 am
- Location: Stockholm, Sweden (UTC+01:00)
Re: [HOWTO] How to increase raid rebuild speed
I would begin with comparing the raw values of the detailed SMART data (Storage Manager, Storage, Disks/VJBOD, [select the disk in question], Disk Health, SMART Information) between all the disks to see if something is significantly different with this disk.Oscar_I wrote:Any suggestion of where I can get more info about the status (and possibly determine what drive that is that has issues)
Other than that unfortunately the only suggestion I have would be to run the diagnostic software of the manufacturer on the disk but that would have to be done in a different computer so isn't the best now in the middle of a current storage expansion...
I would probably contact the proper channel for a possible warranty replacement of the disk (reseller or disk manufacturer depending on where in the world you're located) and ask for their advice.
Are these 8 TB disks also WD Red? If not, what brand and model?
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.
All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
-
- Starting out
- Posts: 20
- Joined: Tue Mar 06, 2018 6:04 am
Re: [HOWTO] How to increase raid rebuild speed
Hello,
Just purchased a TS-431P, updated to the latest firmware, installed 4 x 8TB drives (Seagate) and configured a Raid-5 volume. Initial sync is awfully slow, and speed does not change when I change the sync priority:
cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md1 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[0]
23412214272 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
[>....................] resync = 2.1% (166559964/7804071424) finish=103654.7min speed=1228K/sec
That's 71 days!
I tried to increase the min and max speed as described here, no result. CPU is almost idle, with the md1_raid5 process using between 0.5 and 3% cpu.
Web UI is slow too, everytime I commit a change, it takes a long time, or even times out/fails, like trying to change the admin password, or trying to reboot. Think I'm going to try a hard reset.
Edit: finally managed to reboot the NAS, it restarted rebuild, this time between 15 and 30 Mbytes/s, which of course had changed more or less 71 days to 71 hours, much better, even if it's a bit slow, but on my previous NAS (Lenovo ix4-300D) it took 2 days for 4x4TB Raid-5 so this seems consistent.
Thanks.
Just purchased a TS-431P, updated to the latest firmware, installed 4 x 8TB drives (Seagate) and configured a Raid-5 volume. Initial sync is awfully slow, and speed does not change when I change the sync priority:
cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md1 : active raid5 sdd3[3] sdc3[2] sdb3[1] sda3[0]
23412214272 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
[>....................] resync = 2.1% (166559964/7804071424) finish=103654.7min speed=1228K/sec
That's 71 days!
I tried to increase the min and max speed as described here, no result. CPU is almost idle, with the md1_raid5 process using between 0.5 and 3% cpu.
Web UI is slow too, everytime I commit a change, it takes a long time, or even times out/fails, like trying to change the admin password, or trying to reboot. Think I'm going to try a hard reset.
Edit: finally managed to reboot the NAS, it restarted rebuild, this time between 15 and 30 Mbytes/s, which of course had changed more or less 71 days to 71 hours, much better, even if it's a bit slow, but on my previous NAS (Lenovo ix4-300D) it took 2 days for 4x4TB Raid-5 so this seems consistent.
Thanks.