md0 raid won't start anymore

[pjakobs]

I've posted this as a query on a thread that seemed to describe a similar issue, but I'll repost it here since it may or may not be related:

My QT-412 boots, enables the interface and is accessible via ssh but not (at least not in reasonable time) through the web interface.
The trouble started when I tried to create a RAID1 with two external drives using normal mdadm commands, I would assume that something in the automatic RAID config got broke by that. (I know, my bad, but who knew?)

After that, the webui showed the RAID status as "failed" although it was still active and I could access it through nfs, /proc/mdstat was showing a perfectly healthy RAIDset, too.
When I rebooted, the system did come up with a dhcp ip address and I had to re-configure the interface through qfinder.

After that, I could login with admin and my ssh key, so that data was preserved, but I found the system in the state as shown below minus /dev/md0

As you can see, I can activate my md0 just fine, even through a mdadm --assemble --scan, so md seems to be happy with the supberblocks.

My drives:

Code: Select all

/sbin/hdparm -i /dev/sd[a-d] 2>/dev/null |grep Model
 Model=WDC WD30EFRX-68EUZN0                    , FwRev=82.00A82, SerialNo=     WD-WCC4N1CZP0D7
 Model=WDC WD30EFRX-68EUZN0                    , FwRev=82.00A82, SerialNo=     WD-WCC4N7VXZN79
 Model=WDC WD30EFRX-68EUZN0                    , FwRev=80.00A80, SerialNo=     WD-WCC4NRD7DV7P
 Model=WDC WD30EFRX-68EUZN0                    , FwRev=82.00A82, SerialNo=     WD-WMC4N0F1CXD5

My Arrays:

Code: Select all

 cat /proc/mdstat 
Personalities : [raid1] [linear] [raid0] [raid10] [raid6] [raid5] [raid4] 
md0 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1]
                 8786092608 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
                 bitmap: 0/11 pages [0KB], 131072KB chunk

md13 : active raid1 sdb4[0] sdd4[3] sdc4[2] sda4[1]
                 458880 blocks [4/4] [UUUU]
                 bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
                 530048 blocks [4/4] [UUUU]
                 bitmap: 0/65 pages [0KB], 4KB chunk

(I have manually assembled /dev/md0 with mdadm --assemble /dev/md0 /dev/sd[a-d]3 which works just fine - but not upon system startup)

My Superblocks:

Code: Select all

mdadm --examine /dev/sda3    
/dev/sda3:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : d4b34e44:00da9f33:712188f4:d2ef62ca
           Name : 0
  Creation Time : Sun May 10 08:58:28 2015
     Raid Level : raid5
   Raid Devices : 4

  Used Dev Size : 5857395112 (2793.02 GiB 2998.99 GB)
     Array Size : 17572185216 (8379.07 GiB 8996.96 GB)
      Used Size : 5857395072 (2793.02 GiB 2998.99 GB)
   Super Offset : 5857395368 sectors
          State : clean
    Device UUID : 9f1c7674:8e8ac1f1:2b2a4122:ec2d55c6

Internal Bitmap : 2 sectors from superblock
    Update Time : Sat Oct 31 14:17:22 2015
       Checksum : f0fef5b2 - correct
         Events : 9

         Layout : left-symmetric
     Chunk Size : 64K

    Array Slot : 0 (0, 1, 2, 3, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
   Array State : Uuuu 380 failed

Code: Select all

mdadm --examine /dev/sdb3
/dev/sdb3:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : d4b34e44:00da9f33:712188f4:d2ef62ca
           Name : 0
  Creation Time : Sun May 10 08:58:28 2015
     Raid Level : raid5
   Raid Devices : 4

  Used Dev Size : 5857395112 (2793.02 GiB 2998.99 GB)
     Array Size : 17572185216 (8379.07 GiB 8996.96 GB)
      Used Size : 5857395072 (2793.02 GiB 2998.99 GB)
   Super Offset : 5857395368 sectors
          State : clean
    Device UUID : be20431e:831f7db0:b35dd759:781f541a

Internal Bitmap : 2 sectors from superblock
    Update Time : Sat Oct 31 14:17:27 2015
       Checksum : e51cb3df - correct
         Events : 9

         Layout : left-symmetric
     Chunk Size : 64K

    Array Slot : 1 (0, 1, 2, 3, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
   Array State : uUuu 380 failed

Code: Select all

mdadm --examine /dev/sdc3
/dev/sdc3:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : d4b34e44:00da9f33:712188f4:d2ef62ca
           Name : 0
  Creation Time : Sun May 10 08:58:28 2015
     Raid Level : raid5
   Raid Devices : 4

  Used Dev Size : 5857395112 (2793.02 GiB 2998.99 GB)
     Array Size : 17572185216 (8379.07 GiB 8996.96 GB)
      Used Size : 5857395072 (2793.02 GiB 2998.99 GB)
   Super Offset : 5857395368 sectors
          State : clean
    Device UUID : 921c0251:c24ca13b:30b9f050:5a21bc34

Internal Bitmap : 2 sectors from superblock
    Update Time : Sat Oct 31 14:17:57 2015
       Checksum : b4813a70 - correct
         Events : 9

         Layout : left-symmetric
     Chunk Size : 64K

    Array Slot : 2 (0, 1, 2, 3, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
   Array State : uuUu 380 failed

Code: Select all

mdadm --examine /dev/sdd3
/dev/sdd3:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : d4b34e44:00da9f33:712188f4:d2ef62ca
           Name : 0
  Creation Time : Sun May 10 08:58:28 2015
     Raid Level : raid5
   Raid Devices : 4

  Used Dev Size : 5857395112 (2793.02 GiB 2998.99 GB)
     Array Size : 17572185216 (8379.07 GiB 8996.96 GB)
      Used Size : 5857395072 (2793.02 GiB 2998.99 GB)
   Super Offset : 5857395368 sectors
          State : clean
    Device UUID : 32539d79:99d15121:f1976784:f0de07f8

Internal Bitmap : 2 sectors from superblock
    Update Time : Sat Oct 31 14:18:17 2015
       Checksum : b98f9254 - correct
         Events : 9

         Layout : left-symmetric
     Chunk Size : 64K

    Array Slot : 3 (0, 1, 2, 3, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
   Array State : uuuU 380 failed

Seems like something has majorly messed up the superblocks and they're reporting many more RAID slots than the drive really has.

For reasons unknown,

Code: Select all

getcfg system version

yields nothing. Not a thing, but I've recently upgraded to 4.2

Code: Select all

mdadm --examine --scan       
ARRAY /dev/md9 level=raid1 num-devices=4 UUID=82234069:44f6464a:543cfa98:bdf63dbc
ARRAY /dev/md/4 level=raid1 metadata=1.0 num-devices=2 UUID=a2b283af:1c3f61a5:bbede3ff:29e9e0af name=4
ARRAY /dev/md/0 level=raid5 metadata=1.0 num-devices=4 UUID=d4b34e44:00da9f33:712188f4:d2ef62ca name=0
ARRAY /dev/md13 level=raid1 num-devices=4 UUID=7ef8bc9b:8c8c662b:80b39f00:a2dda617

sounds reasonable to me

Code: Select all

cat /etc/config/raidtab
raiddev /dev/md0
        raid-level      5
        nr-raid-disks   4
        nr-spare-disks  0
        chunk-size      4
        persistent-superblock   1
        device  /dev/sda3
        raid-disk       0
        device  /dev/sdb3
        raid-disk       1
        device  /dev/sdc3
        raid-disk       2
        device  /dev/sdd3
        raid-disk       3

is at least right about md0

Code: Select all

[VOLUME 1]
device name = /dev/md0
raid level = 5
raid disks = 1,2,3,4
spare raid disks =
status = 0
record_time = Fri May  8 17:01:25 2015

filesystem = 104
[Global]
Available Disk = 4

won't fault that one either.

Code: Select all

[~] # dmesg                              
EXT3-fs (md9): mounted filesystem with writeback data mode
[   30.211518] md: md13 stopped.
[   30.571732] md: bind<sda4>
[   30.572045] md: bind<sdc4>
[   30.572320] md: bind<sdd4>
[   30.572589] md: bind<sdb4>
[   30.576076] md/raid1:md13: active with 4 out of 4 mirrors
[   30.576315] created bitmap (57 pages) for device md13
[   30.583960] md13: bitmap initialized from disk: read 4/4 pages, set 0 of 114720 bits
[   30.640547] md13: detected capacity change from 0 to 469893120
[   30.668358]  md13: unknown partition table
[   32.216266] kjournald starting.  Commit interval 5 seconds
[   32.350891] EXT3-fs (md9): using internal journal
[   32.350923] EXT3-fs (md9): mounted filesystem with writeback data mode
[   33.113997] kjournald starting.  Commit interval 5 seconds
[   33.190925] EXT3-fs (md9): using internal journal
[   33.190958] EXT3-fs (md9): mounted filesystem with ordered data mode
[   37.950841] kjournald starting.  Commit interval 5 seconds
[   38.010776] EXT3-fs (md13): using internal journal
[   38.011130] EXT3-fs (md13): recovery complete
[   38.011157] EXT3-fs (md13): mounted filesystem with writeback data mode
[   99.355282] fuse init (API version 7.18)
[   99.609118] NET: Registered protocol family 5
[   99.910860] md: linear personality registered for level -1
[   99.956722] md: raid0 personality registered for level 0
[   99.985540] md: raid10 personality registered for level 10
[  100.025233] xor: measuring software checksum speed
[  100.070360]    arm4regs  :  1067.200 MB/sec
[  100.120355]    8regs     :   759.200 MB/sec
[  100.170353]    32regs    :   904.800 MB/sec
[  100.170374] xor: using function: arm4regs (1067.200 MB/sec)
[  100.630501] raid6: int32x1     87 MB/s
[  100.800453] raid6: int32x2    120 MB/s
[  100.970367] raid6: int32x4    132 MB/s
[  101.140579] raid6: int32x8    119 MB/s
[  101.140600] raid6: using algorithm int32x4 (132 MB/s)
[  101.140620] raid6: using intx1 recovery algorithm
[  101.164391] async_tx: api initialized (async)
[  101.262466] md: raid6 personality registered for level 6
[  101.262502] md: raid5 personality registered for level 5
[  101.262522] md: raid4 personality registered for level 4
[  101.584439] device-mapper: ioctl: 4.22.1-ioctl (2012-06-01) initialised: dm-devel@redhat.com
[  101.847607] MV-CESA:Fallback driver 'hmac(sha1)' could not be loaded!
[  101.847652] alg: hash: Failed to load transform for mv-hmac-sha1: -2
[  101.899627] cryptodev: driver 0.9 loaded.
[  102.030709] usbcore: registered new interface driver usbhid
[  102.030747] usbhid: USB HID core driver
[  102.083424] usbcore: registered new interface driver usblp
[  102.112224] sysinfo: Module loaded.
[  102.241752] jnl: driver (lke_9.2.0 QNAP, LBD=ON) loaded at bf1f6000
[  102.282946] ufsd: module license 'Commercial product' taints kernel.
[  102.282986] Disabling lock debugging due to kernel taint
[  102.310541] ufsd: driver (lke_9.2.0 QNAP, build_host("BuildServer35"), acl, ioctl, bdi, sd2(0), fua, bz, rsrc) loaded at bf204000
[  102.310554] NTFS support included
[  102.310558] Hfs+/HfsJ support included
[  102.310562] optimized: speed
[  102.310566] Build_for__QNAP_TS-120_k3.4.6_2014-11-19_lke_9.2.0_r245986_b9
[  102.310572] 
[  102.488564] RPC: Registered named UNIX socket transport module.
[  102.488604] RPC: Registered udp transport module.
[  102.488623] RPC: Registered tcp transport module.
[  102.488641] RPC: Registered tcp NFSv4.1 backchannel transport module.
[  102.681605] NFS: Registering the id_resolver key type
[  102.858331] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[  103.162455] fnotify: Load file notify kernel module.
[  103.222021] lib80211: common routines for IEEE802.11 drivers
[  103.222144] lib80211_crypt: registered algorithm 'NULL'
[  103.395191] cfg80211: Calling CRDA to update world regulatory domain
[  105.030696] usbcore: registered new interface driver snd-usb-audio
[  105.056340] usbcore: registered new interface driver snd-usb-caiaq
[  105.148070] Linux video capture interface: v2.00
[  105.317500] uvcvideo: Unable to create debugfs directory
[  105.325753] usbcore: registered new interface driver uvcvideo
[  105.325791] USB Video Class driver (1.1.1)
[  106.524408] eth0: link down
[  106.524454] eth0: started
[  106.551529] eth1: started
[  106.734683] eth0: stopped
[  106.740704] eth0: mac address changed
[  106.767299] eth0: started
[  106.797466] eth1: stopped
[  106.803473] eth1: mac address changed
[  106.830000] eth1: started
[  109.632942] eth0: link up, full duplex, speed 1 Gbps
[  119.141147] EXT2-fs (mtdblock5): warning: mounting unchecked fs, running e2fsck is recommended
[  122.332927] kjournald starting.  Commit interval 5 seconds
[  122.370858] EXT3-fs (md9): using internal journal
[  122.370889] EXT3-fs (md9): mounted filesystem with ordered data mode
[  125.915009] active port 0 :139
[  125.915043] active port 1 :445
[  125.915058] active port 2 :20
[  130.447859] EXT2-fs (mtdblock5): warning: mounting unchecked fs, running e2fsck is recommended
[  133.308539] Unable to find swap-space signature
[  140.769140] Unable to find swap-space signature
[  147.598801] Unable to find swap-space signature
[  153.798697] Unable to find swap-space signature
[  157.199328] md: md0 stopped.
[  158.269406] md: md0 stopped.
[  160.929244] md: md0 stopped.
[  166.279267] md: md0 stopped.
[  169.789560] md: md0 stopped.
[  184.415222] eth0: stopped
[  184.421946] eth1: stopped
[  186.563646] eth0: link down
[  186.563691] eth0: started
[  186.700877] eth1: started
[  189.764528] eth0: link up, full duplex, speed 1 Gbps
[  209.017714] eth0: stopped
[  209.044286] eth0: link down
[  209.044322] eth0: started
[  209.119479] eth1: stopped
[  209.142917] eth1: started
[  211.795606] eth0: link up, full duplex, speed 1 Gbps
[  236.934595] active port 0 :139
[  236.934629] active port 1 :445
[  236.934644] active port 2 :20
[  249.290230] Initiate iscsi target log successfully.
[  249.482598] iscsi_log_rcv_msg: get log pid = 5596.
[  254.583961] EXT2-fs (mtdblock5): warning: mounting unchecked fs, running e2fsck is recommended
[  258.323254] warning: process `pic_raw' used the deprecated sysctl system call with 8.1.2.
[  260.256451] rule type=2, num=0
[  260.450688] MAC:00:08:9B:CF:60:1C
[  260.451360] page 3, reg 18  = 4a85
[  260.452637] WOL enable
[  260.463762] MAC:00:08:9B:CF:60:1D
[  260.464405] page 3, reg 18  = 4a85
[  260.465682] WOL enable
[  263.793880] Loading iSCSI transport class v2.0-871.
[  264.319684] iscsi: registered transport (tcp)
[  265.448890] iscsid (6325): /proc/6325/oom_adj is deprecated, please use /proc/6325/oom_score_adj instead.
[  323.788853] iscsi target log cleanup successfully.
[  727.830678] md: md0 stopped.
[  727.979983] md: bind<sdb3>
[  727.980247] md: bind<sdc3>
[  727.981529] md: bind<sdd3>
[  727.981784] md: bind<sda3>
[  727.985482] md/raid:md0: device sda3 operational as raid disk 0
[  727.985520] md/raid:md0: device sdd3 operational as raid disk 3
[  727.985543] md/raid:md0: device sdc3 operational as raid disk 2
[  727.985566] md/raid:md0: device sdb3 operational as raid disk 1
[  728.101788] md/raid:md0: allocated 67840kB
[  728.103132] md/raid:md0: raid level 5 active with 4 out of 4 devices, algorithm 2
[  728.103169] RAID conf printout:
[  728.103177]  --- level:5 rd:4 wd:4
[  728.103187]  disk 0, o:1, dev:sda3
[  728.103195]  disk 1, o:1, dev:sdb3
[  728.103203]  disk 2, o:1, dev:sdc3
[  728.103211]  disk 3, o:1, dev:sdd3
[  728.116663] created bitmap (11 pages) for device md0
[  728.118285] md0: bitmap initialized from disk: read 1/1 pages, set 0 of 22345 bits
[  728.170611] md0: detected capacity change from 0 to 8996958830592
[  966.318906]  md0: unknown partition table

I think that the md0 start around 720s is my manual mdadm --assemble --scan.

so - to me, it's either something with the superblock themselves or with the way they are being interpreted.

Any ideas?

kind regards

pj

[pjakobs]

I've manually re-installed the 4.2.0 firmware and now I get

Code: Select all

[~] # getcfg system version
4.2.0

but still no working RAID.

[pjakobs]

just tried this

Code: Select all

[~] # storage_boot_init 2     
storage_boot_init 2 ...
storage_boot_init.c: Check filesystem on /dev/sda3.
mdadm: /dev/md0 not identified in config file.
mdadm: stopped /dev/md0
mdadm: /dev/md0 not identified in config file.
mdadm: stopped /dev/md0
storage_boot_init.c: Stop raid device /dev/md0 succeeded
storage_boot_init.c: Stop raid device /dev/md0 succeeded
storage_boot_init.c: Start_Raid_Force failed on /dev/md0
storage_boot_init.c:     /dev/md0 is not active, stop init!!
storage_boot_init.c: check_last_degrade_error...

and

Code: Select all

[~] # mdadm --assemble --scan
mdadm: /dev/md0 has been started with 4 drives.
[~] # cat /proc/mdstat 
Personalities : [raid1] [linear] [raid0] [raid10] [raid6] [raid5] [raid4] 
md0 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1]
                 8786092608 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
                 bitmap: 0/11 pages [0KB], 131072KB chunk

md13 : active raid1 sdb4[0] sdd4[3] sdc4[2] sda4[1]
                 458880 blocks [4/4] [UUUU]
                 bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
                 530048 blocks [4/4] [UUUU]
                 bitmap: 0/65 pages [0KB], 4KB chunk

unused devices: <none>

now running e2fsck /dev/md0 - file system seems to be ok.

jup, it is

Code: Select all

[~] # e2fsck /dev/md0
e2fsck 1.42.6 (21-Sep-2012)

/dev/md0: clean, 2583439/549134336 files, 1406681135/2196523152 blocks

any ideas?

[Don]

QNAP does not support raid on external devices unless it one of QNAP's expansion units. There are multiple partitions and raids on all QNAP internal drives that it uses for its own purposes. Since you decided to muck around with mdadm I suggest you open a support ticket with QNAP to see if they can undo whatever it is you did.

[pjakobs]

This forum sure is a bit rough around the edges, isn't it?

For those who care to know:

it really wasn't the superblocks, the "FAIL" states were a red herring. It was a rather common failure mode on the Qnap devices:
uLinux.conf was partially overwritten during a reconfig / reboot.

I've had something similar before, if it happens to you, you may want to go ahead and first make sure your /etc/config/uLinux.conf looks ok (starting with a [System] section and actually being rather long (ok ok, I know this is not a testable criteria).)

I found help in the German qnapclub forum, thanks dr_mike there. The solution was rather simple:
There's a copy of a working uLinux.conf on /dev/mtdblock5 (one of the internal flash devices)
Copying it over fixed the issue in my case. YMMV

Code: Select all

mount -t ext2 /dev/mtdblock5 /tmp/config
cp /mnt/HDA_ROOT/.config/uLinux.conf /mnt/HDA_ROOT/.config/uLinux.bak
cp /tmp/config/uLinux.conf /mnt/HDA_ROOT/.config/uLinux.conf
umount /tmp/config
reboot

I'm actually pretty sure now that "what I did" was not causing this issue, which doesn't mean that running an external md-raid would be necessarily a great idea as that's something that Qnap uses to tier their systems.

pj

[tdoyle]

Hi pj,

I hear you about being rough around the edges.

Just wanted to say that you absolutely saved my bacon. I have had some issues with my nas in the past (419p-ii) mostly due to Seagate drives that became unsupported. I had a drive fail 2 weeks ago and replaced it and rebuilt the array without issue. This morning I rebooted in preparation to upgrade to the latest 4.2.0 build but it never came back up. I was able to log in through the browser after realizing that my fixed IP was now dynamic. The name of the nas was also reset but some of my other settings were still intact. While I do have backups, I dread copying that much data (again). In any case, I saw your post in another thread and your symptoms were exactly the same as mine. I then found this thread, tried your solution and I cannot tell you how happy I was when the nas came back to life. I would have never looked at those config files.

Again thanks for following up and posting your resolution here, it certainly helped me and I hope it will help others.

Keep up the good work,

Travis

[pickers]

pj - just wanted to say thanks so much for your solution which got me up and running again after several days of downtime and searching for solutions. My TS-110 hadn't come back properly after a power failure - had lost IP settings and failed to mount its single disk volume. All is well now, thanks to you - thanks so much for taking the time to come back and post your solution.

[MrVideo]

For the future, it is strongly recommended that you get a UPS for your NAS. Get one that allows a USB connection so that the NAS can be warned when the main power fails, so that the NAS can shut down gracefully after a certain amount of time. APC makes units that are compatible.

[pickers]

Thanks, yes - I have learned my lesson on this and am currently in the process of choosing a suitable device