FSCK on TS-410 still running after 264 hours

[pah96]

Hello!

Unfortunately I had to hard shutdown my TS-410 after one rsync job and some other services stopped working. The box wasn't accessable any more.

System is Raid5 Ext4 with 4x3GB Hitachi-Disks. Additionally disk 3 showed warning because of read failure at smart on 1 sector. 8TB MD-Volume was about 50% in use with data.

After booting up everything was fine execpt the system advised to perform a fsck. I knew this will not be completed for some time. Never the less I started the process because haven't done this for long time. And also because of the smart error in disk 3.

Now it is running about 38 hours and still shows 0% status. I know I've to be patient ...

But I'm not sure at the moment if this will ever end. Because I can see in the webui that the disks have about 28°C which is really little. Normally they have 46-48°C constant. So they nearly turn very slow I guess. Or even not?

And meanwhile disk 3 shows more errors: 46 current_pending_sectors & 4 reallocated_sector_ct - that is completely new status.

Hope this will end in a happy way.

I know I should have changed disk 3 earlier. But I thought this single error is not a big problem some weeks ago. Planned it this week because of holidays but it mostly comes different as you think or plan.

Anyway. Can any one calm me down only to be more patient? How can I additionally check if the fsck is this going on and just takes more time? Status change of disk 3 may be one way. But I like control more real time and with reliable method

BR,
pah96

[dolbyman]

are the disks nas disks ? If not, missing TLER would slow everything to a crawl

pop the busted disk out and see if it comes back to normal speeds

hope you have backups

[pah96]

Thanks for reply!

Is this really the only hint/possibility?

Ok, let me update you with following additional informations:

Code: Select all

[~] # cat /proc/mdstat
Personalities : [raid1] [linear] [raid0] [raid10] [raid6] [raid5] [raid4]
md0 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1]
                 8786092608 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

[~] # cat /etc/mdadm.conf
ARRAY /dev/md0 devices=/dev/sda3,/dev/sdb3,/dev/sdc3,/dev/sdd3

[~] # mdadm -Q /dev/md0
/dev/md0: 8379.07GiB raid5 4 devices, 0 spares. Use mdadm --detail for more detail.

[~] # mdadm -D /dev/md0
/dev/md0:
        Version : 01.00.03
  Creation Time : Fri Jun 13 18:56:43 2014
     Raid Level : raid5
     Array Size : 8786092608 (8379.07 GiB 8996.96 GB)
  Used Dev Size : 2928697536 (2793.02 GiB 2998.99 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sat May  8 15:35:31 2021
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           Name : 0
           UUID : 63ed0a23:5bc99009:aa4a639d:1215a52b
         Events : 770265

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3
       2       8       35        2      active sync   /dev/sdc3
       3       8       51        3      active sync   /dev/sdd3

Yes, these are NAS disks - HDS723030ALA640

And yes, there is a backup (an a TS-412).

And/But - how can it be otherwise - not everything is rsynced as I decided meanwhile 7 years ago but now it would be a painful loss of data

Thanks for any further hint!

BR,
pah96

[dolbyman]

Cant find any NAS specification for the disk model...says deskstar (not deskstar nas)

the mdstat only says that the raid has not failed yet..not if it is busy waiting

I cannot decipher the rest of your post

[pah96]

Indeed it is some days ago but you can find it still here:

https://www.qnap.com/en/compatibility/? ... category=1

For sure, deskstar not "nasstar", that's right. Not even Enterprise. But recommended as well.

But the only question was:

How can I check the real status of the fsck process - the webui isn't reliable.

Thanks a lot!

pah96

BTW: in hope it has finished meanwhile: don't really know but still no new log entrys

[dolbyman]

try e2fsck with -c flag on the md0 device

[pah96]

Update after 60hrs with status 0.0%

Code: Select all

[~] # iostat -Dx
                             extended device statistics
device mgr/s mgw/s    r/s    w/s    kr/s    kw/s   size queue   wait svc_t  %b
mtdblock4     0     0    0.0    0.0     0.0     0.0   16.0   0.0   40.0  40.0   0
mtdblock5     0     0    0.0    0.0     0.0     0.0    3.9   0.0   11.4  11.4   0
sda        1     1    1.2    0.5    17.0     6.1   13.2   0.2  112.7   7.0   1
sdb        1     1    1.1    0.5    13.7     8.5   13.9   0.2  113.4   7.4   1
sdc        0     0    0.6    0.3    11.3     2.7   15.3   0.0   29.8  10.2   1
sdd        0     0    0.6    0.3    11.7     2.7   15.9   0.0   20.7   9.5   1
sds        0     0    0.1    0.0     0.9     0.0    6.2   0.0   11.5   2.3   0
md9        0     0    0.1    0.0     0.8     0.2    8.9   0.0    0.0   0.0   0
md13       0     0    0.9    0.4    20.3     1.6   16.4   0.0    0.0   0.0   0
md4        0     0    1.5    0.8     6.2     3.4    4.0   0.0    0.0   0.0   0
md0        0     0    2.6    0.1    18.4     0.8    7.1   0.0    0.0   0.0   0
sdwa       0     0    0.0    0.0     0.1     0.0   13.7   0.0    5.2   0.9   0

doesn't change on anytime.

I guess there is nothing going on any more - am I right?

e2fsck -c # Check for bad blocks and add them to the badblock list

should/can this be done without stopping any process or rebooting?

I'm really surprised how less number of tools on qnap (NAS!) are available specially for disk mgmt.

br,
pah96

[pah96]

Update:

after 264 hours nothing changed, nothing new.