RAID 5 rebuild very slow on TS-412 (160-300k/sec)

[tangelo]

I recently had one of the drives in my 4-drive Raid 5 array (4 x 1.5TB) in my Qnap TS-412 change status from Good to Normal. It appeared to have some bad blocks and I decided to replace the drive. I replaced it with a 2TB drive and the array started rebuilding.

I assumed it would take a day or two and wasn't worried about how slowly it seemed to be progressing when I monitored it through the webgui. However, after a day or so I think it was only at 25% and progress appeared to have slowed down. I also started noticing that access to shared drives was failing (unable to connect), and then that the webgui was becoming unresponsive - it would take a long time to load pages sometimes, or just not load them at all. This steadily became worse to the point where it is now practically unusable - I have moved to terminal access but that can be sluggish at times also.

In any case, the rebuild started on Nov 9 (about 8 days ago) and is currently at 55%. A sample of the output from cat /proc/mdstat is:

Code: Select all

[~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md4 : active raid1 sda2[2](S) sdd2[0] sdc2[3](S) sdb2[1]
      530048 blocks [2/2] [UU]
     
md0 : active raid5 sda3[4] sdd3[3] sdc3[2] sdb3[1]
      4390708800 blocks level 5, 64k chunk, algorithm 2 [4/3] [_UUU]
      [===========>.........]  recovery = 55.2% (809173732/1463569600) finish=60374.7min speed=180K/sec
     
md13 : active raid1 sda4[0] sdd4[3] sdc4[2] sdb4[1]
      458880 blocks [4/4] [UUUU]
      bitmap: 2/57 pages [8KB], 4KB chunk

md9 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
      530048 blocks [4/4] [UUUU]
      bitmap: 8/65 pages [32KB], 4KB chunk

unused devices: <none>

I have followed the post here (viewtopic.php?t=10268) on how to increase the min speed for the rebuild. The results for this (after increasing the min speed to 75000) are:

Code: Select all

[~] # cat /proc/sys/dev/raid/speed_limit_max
200000
[~] # cat /proc/sys/dev/raid/speed_limit_min
75000

But the speeds I am seeing are in the order of 160-300K/sec, nowhere near the 75MB/sec I set - or even the much lower value that was there originally (5MB/s I think it was).

I have tried stopping a number of services (individually and using /etc/init.d/services.sh stop) but the following output from 'top' suggests that there isn't anything hogging CPU or memory in any case:

Code: Select all

Mem: 226620K used, 29040K free, 0K shrd, 53156K buff, 19032K cached
Load average: 8.89, 9.69, 9.20    (State: S=sleeping R=running, W=waiting)

  PID USER     STATUS   RSS  PPID %CPU %MEM COMMAND
20261 admin    R        916 19766  0.3  0.3 top
17532 admin    D       4168     1  0.0  1.6 smbd
19751 admin    S       2344  5808  0.0  0.9 sshd
 6185 admin    S <     2200     1  0.0  0.8 iscsid
 1637 admin    S       2076  5808  0.0  0.8 sshd
 1523 admin    S       1788     1  0.0  0.6 qsmartd
 2959 admin    S       1504     1  0.0  0.5 picd
 2486 admin    S       1376     1  0.0  0.5 _thttpd_
19766 admin    S       1340 19751  0.0  0.5 sh
23801 admin    D       1308     1  0.0  0.5 vs_refresh
 1642 admin    S       1308  1637  0.0  0.5 sh
 6145 admin    S       1188     1  0.0  0.4 qLogEngined
 2314 admin    S       1120     1  0.0  0.4 cupsd
 1519 admin    S       1096     1  0.0  0.4 hotswap
 4312 admin    S        896     1  0.0  0.3 hd_util
 1941 admin    S        868     1  0.0  0.3 upnpd
 2883 admin    S        736     1  0.0  0.2 hwmond
 5878 admin    S        668     1  0.0  0.2 upnpcd
 5808 admin    S        620     1  0.0  0.2 sshd
 2810 admin    S        600     1  0.0  0.2 gpiod
 2068 admin    S        580     1  0.0  0.2 mDNSResponderPo
 4315 admin    S        536     1  0.0  0.2 gen_bandwidth
  809 admin    S        532     1  0.0  0.2 daemon_mgr.nvr
 2933 admin    S        524  2925  0.0  0.2 centerim
 3315 admin    S        504     1  0.0  0.1 stunnel

If anyone has some suggestions for what to investigate next it would be much appreciated. I am not particularly linux proficient but I could make my way around under direction. Of course it could be something simple - I don't mind looking stupid if it get this problem sorted! At this stage, at say 180K/sec, it looks like the rebuild won't complete for another 41 days! (60000 min / 60 / 24)

Cheers
Paul

[ledf]

I´m in the same situation. Did you solve this issue?

[P3R]

@tangelo,

Maybe another disk is on the brink of a failure?

In your situation I would contact Qnap support.
'