I always get a failed to check Filesystem message

[Tony Feliu]

Hi,
I will try to explain my problem.
Some hours ago I got the message:
[Storage & Snapshots] Set volume "DataVol1" to read-only. Storage pool "1" is full. Ensure the pool has 32.00 GB of free space.
I have a total capacity of 18TB with 9TB free.
The system asked to me to do a filesystem check.
There wasn't any abnormal shutdown and nothing similar.
I tried to make a Filesystem check and I get some messages like:
[Storage & Snapshots] Failed to check file system for volume "DataVol1". Storage pool: 1. Volume could not be unmounted by process "avahi-daemon".
After this I stopped the filesystem check and I rebooted my NAS.
After this I tried to do filesystem check and It never ends correctly, some times I reach 28% others 50% and now finally can't go beyond 1%
I always get the message:
[Storage & Snapshots] Failed to check file system for volume "DataVol1". Storage pool: 1.

My QNAP model is TS-453B with the firmware 4.5.4.1723 and 4 8TB disk in RAID5.

Any help or idea would be welcome.
Thanks

[Tony Feliu]

Hi guys,
Here's an upgrade of my problem. Any help would be appreciated.
After the check file system the volume get always unmounted so I couldn't access to my data.
QNAP support managed to mount the volume to let me save all my data, so I did (thank god).
Now all my data is save except an encrypted folder that I wasn't able to unlock with the password, I always get an "Unknown error" (see image).
I'm sure I'm writing the right password and I've also tried from the Shared Folders menu but I didn't see any shared folder (see image).
My admin user was doing strange things and seemed to do not have permissions for anything so I decided to reboot.
After rebooting my admin user was ok but then the problem was that the file system check fails as always an the volume is unmounted again.
QNAP support tried to mount the volume again for me to decrypt the folder an save the data but they couldn't.
Here is what I tried at the moment and some more information commands:

md_checker results:

Code: Select all

[~] # md_checker

Welcome to MD superblock checker (v2.0) - have a nice day~

Scanning system...


RAID metadata found!
UUID:           fdb1284b:df7ea5c3:4c104f1d:8ac02f22
Level:          raid5
Devices:        4
Name:           md1
Chunk Size:     64K
md Version:     1.0
Creation Time:  May 27 08:36:16 2018
Status:         ONLINE (md1) [UUUU]
===============================================================================================
 Enclosure | Port | Block Dev Name | # | Status |   Last Update Time   | Events | Array State
===============================================================================================
 NAS_HOST       1        /dev/sdc3   0   Active   Jul 25 11:01:22 2021   193064   AAAA
 NAS_HOST       2        /dev/sdd3   1   Active   Jul 25 11:01:22 2021   193064   AAAA
 NAS_HOST       4        /dev/sdb3   2   Active   Jul 25 11:01:22 2021   193064   AAAA
 NAS_HOST       3        /dev/sda3   3   Active   Jul 25 11:01:22 2021   193064   AAAA
===============================================================================================

pvdisplay results:

Code: Select all

[~] # pvdisplay
  --- Physical volume ---
  PV Name               /dev/drbd1
  VG Name               vg1
  PV Size               21.80 TiB / not usable 100.00 KiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              5715871
  Free PE               0
  Allocated PE          5715871
  PV UUID               FrWs4j-8Pwo-k3jo-Sdyg-uejO-83Gh-22T9g5

vgdisplay results:

Code: Select all

[~] # vgdisplay
  --- Volume group ---
  VG Name               vg1
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  5826
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                4
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               21.80 TiB
  PE Size               4.00 MiB
  Total PE              5715871
  Alloc PE / Size       5715871 / 21.80 TiB
  Free  PE / Size       0 / 0
  VG UUID               zMXX9u-Y0bC-7Eqe-6Qo2-PgIy-hyuX-heu0Hq

lvdisplay results:

Code: Select all

[~] # lvdisplay
  --- Logical volume ---
  LV Name                tp1
  VG Name                vg1
  LV UUID                kNQhzX-kKQ6-hNoq-UJOJ-a4I2-sPjk-hCIK3S
  LV Write Access        read/write
  LV Creation host, time NAS, 2018-05-27 08:36:19 +0200
  LV Pool metadata       tp1_tmeta
  LV Pool data           tp1_tierdata_0
  LV Status              available
  # open                 2
  LV Size                21.60 TiB
  Allocated pool data    100.00%
  Allocated pool chunks  45293216
  Allocated metadata     2.35%
  Current LE             5661652
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     4096
  Block device           253:5

  --- Logical volume ---
  LV Path                /dev/vg1/lv1
  LV Name                lv1
  VG Name                vg1
  LV UUID                rqAN6g-7d9L-RSuY-YMUv-1Egr-rvCy-9MEFbx
  LV Write Access        read/write
  LV Creation host, time NAS, 2018-05-27 08:38:18 +0200
  LV Pool name           tp1
  LV Status              available
  # open                 1
  LV Size                18.33 TiB
  Mapped size            100.00%
  Mapped sectors         39354368000
  Current LE             4804000
  Segments               1
  Allocation             inherit
  Read ahead sectors     8192
  Block device           253:7

  --- Logical volume ---
  LV Path                /dev/vg1/lv1312
  LV Name                lv1312
  VG Name                vg1
  LV UUID                PTdmFh-KjcZ-ov24-umCO-fQk8-mlpA-rJpV9J
  LV Write Access        read/write
  LV Creation host, time NAS, 2018-05-28 04:00:12 +0200
  LV Status              available
  # open                 0
  LV Size                2.23 GiB
  Current LE             571
  Segments               3
  Allocation             inherit
  Read ahead sectors     8192
  Block device           253:8

  --- Logical volume ---
  LV Path                /dev/vg1/lv544
  LV Name                lv544
  VG Name                vg1
  LV UUID                nbFxOP-I0s0-IZXS-znRz-XwpD-rpfw-iqpAAa
  LV Write Access        read/write
  LV Creation host, time NAS, 2018-05-28 04:00:13 +0200
  LV Status              NOT available
  LV Size                144.00 GiB
  Current LE             36864
  Segments               2
  Allocation             inherit
  Read ahead sectors     8192

trying to do an e2fsck after stop the services and also rebooted in maintenance:

Code: Select all

[~] # e2fsck -fyv -t -C 0 /dev/mapper/cachedev1
e2fsck 1.43.9 (8-Feb-2018)
DataVol1: recovering journal
Superblock needs_recovery flag is clear, but journal has data.
Run journal anywaye2fsck: Unknown code ____ 251 while recovering journal of DataVol1
e2fsck: unable to set superblock flags on DataVol1


DataVol1: ********** WARNING: Filesystem still has errors **********

With the backup superblock:

Code: Select all

[~] # e2fsck -fyv -t -C 0 -b 32768 /dev/mapper/cachedev1
e2fsck 1.43.9 (8-Feb-2018)
Superblock needs_recovery flag is clear, but journal has data.
Recovery flag not set in backup superblock, so running journal anyway.
DataVol1: recovering journal
Superblock needs_recovery flag is clear, but journal has data.
Recovery flag not set in backup superblock, so running journal anyway.
e2fsck: Unknown code ____ 251 while recovering journal of DataVol1
Superblock needs_recovery flag is clear, but journal has data.
Recovery flag not set in backup superblock, so running journal anyway.
e2fsck: unable to set superblock flags on DataVol1


DataVol1: ***** FILE SYSTEM WAS MODIFIED *****

DataVol1: ********** WARNING: Filesystem still has errors **********

Trying to mount the volume:

Code: Select all

[~] # /etc/init.d/init_lvm.sh
Changing old config name...
Reinitialing...
Detect disk(8, 48)...
dev_count ++ = 0Detect disk(8, 16)...
dev_count ++ = 1Detect disk(253, 0)...
ignore non-root enclosure disk(253, 0).
Detect disk(8, 64)...
ignore non-root enclosure disk(8, 64).
Detect disk(8, 32)...
dev_count ++ = 2Detect disk(8, 0)...
dev_count ++ = 3Detect disk(8, 48)...
Detect disk(8, 16)...
Detect disk(253, 0)...
ignore non-root enclosure disk(253, 0).
Detect disk(8, 64)...
ignore non-root enclosure disk(8, 64).
Detect disk(8, 32)...
Detect disk(8, 0)...
/bin/cat: /sys/bus/pci/devices/0000:00:14.0/0000:07:00.0/subsystem_vendor: No such file or directory
/bin/cat: /sys/bus/pci/devices/0000:00:14.0/0000:07:00.0/subsystem_vendor: No such file or directory
sys_startup_p2:got called count = -1
  WARNING: duplicate PV FrWs4j8Pwok3joSdyguejO83Gh22T9g5 is being used from both devices /dev/drbd1 and /dev/md1
  Found duplicate PV FrWs4j8Pwok3joSdyguejO83Gh22T9g5: using /dev/drbd1 not /dev/md1
  Using duplicate PV /dev/drbd1 from subsystem DRBD, ignoring /dev/md1
sh: /sys/block/dm-5/dm/pool/tier/relocation_rate: Permission denied
Done

At this point the volume stays on warning for a few seconds but after remains unmounted again.

Any idea or help will be welcome.
Thanks.

[Tony Feliu]

I belief your pool is wrongly configured.
Cant's say why or how. Something with the treshold...

Had it in the past.

Thanks for you reply RandomMen, I'll do some searches about it.

[Tony Feliu]

More info in case it helps:

Dmesg after an e2fsck -fyv -t -C 0 /dev/mapper/cachedev1:

Code: Select all

[87114.626924] buffer_io_error: 301 callbacks suppressed
[87114.632005] Buffer I/O error on dev dm-9, logical block 0, lost async page write
[87114.639406] Buffer I/O error on dev dm-9, logical block 1, lost async page write
[87114.646814] Buffer I/O error on dev dm-9, logical block 3, lost async page write
[87114.654228] Buffer I/O error on dev dm-9, logical block 4, lost async page write
[87114.661641] Buffer I/O error on dev dm-9, logical block 5, lost async page write
[87114.669060] Buffer I/O error on dev dm-9, logical block 6, lost async page write
[87114.676472] Buffer I/O error on dev dm-9, logical block 7, lost async page write
[87114.683965] Buffer I/O error on dev dm-9, logical block 8, lost async page write
[87114.691421] Buffer I/O error on dev dm-9, logical block 9, lost async page write
[87114.698827] Buffer I/O error on dev dm-9, logical block 10, lost async page write
[87124.839414] buffer_io_error: 4137 callbacks suppressed
[87124.839417] Buffer I/O error on dev dm-9, logical block 605585408, lost async page write

Dmesg after an e2fsck -fyv -t -C 0 -b 32768 /dev/mapper/cachedev1

Code: Select all

[87501.695416] Buffer I/O error on dev dm-9, logical block 1, lost async page write
[87501.702837] Buffer I/O error on dev dm-9, logical block 2, lost async page write
[87501.710241] Buffer I/O error on dev dm-9, logical block 3, lost async page write
[87501.717658] Buffer I/O error on dev dm-9, logical block 4, lost async page write
[87501.725078] Buffer I/O error on dev dm-9, logical block 5, lost async page write
[87501.732539] Buffer I/O error on dev dm-9, logical block 6, lost async page write
[87501.739998] Buffer I/O error on dev dm-9, logical block 7, lost async page write
[87501.747450] Buffer I/O error on dev dm-9, logical block 8, lost async page write
[87501.754877] Buffer I/O error on dev dm-9, logical block 9, lost async page write
[87501.762330] Buffer I/O error on dev dm-9, logical block 10, lost async page write

Dmesg after an /etc/init.d/init_lvm.sh

Code: Select all

[87653.512056] buffer_io_error: 7971 callbacks suppressed
[87653.512059] Buffer I/O error on dev dm-9, logical block 0, lost async page write
[87653.524648] Buffer I/O error on dev dm-9, logical block 1, lost async page write
[87653.532057] Buffer I/O error on dev dm-9, logical block 3, lost async page write
[87653.539468] Buffer I/O error on dev dm-9, logical block 4, lost async page write
[87653.546891] Buffer I/O error on dev dm-9, logical block 5, lost async page write
[87653.554302] Buffer I/O error on dev dm-9, logical block 6, lost async page write
[87653.561738] Buffer I/O error on dev dm-9, logical block 7, lost async page write
[87653.569154] Buffer I/O error on dev dm-9, logical block 8, lost async page write
[87653.576591] Buffer I/O error on dev dm-9, logical block 9, lost async page write
[87653.584021] Buffer I/O error on dev dm-9, logical block 10, lost async page write
[87653.654125] JBD2: recovery failed
[87653.657464] EXT4-fs (dm-9): error loading journal
[87654.716559] EXT4-fs (dm-9): INFO: recovery required on readonly filesystem
[87654.723463] EXT4-fs (dm-9): write access will be enabled during recovery
[87654.784342] JBD2: recovery failed
[87654.787725] EXT4-fs (dm-9): error loading journal
[87655.616659] EXT4-fs (dm-9): can't mount with data=, fs mounted w/o journal
[87658.627889] device-mapper: tier: set allocation tier failed

Thanks