RAID Repair?

[asteynberg]

Hi all,

facing a bit of a problem so hopefully someone can advise.

I had a TS-459u+ NAS with 2x WD 4tb Reds, the NAS notified me of an error on the HDD (not a major one it seems) and I recently got a replacement.
Thing is, as I went to replace the "faulty" HDD with the new HDD, both hard drives were lit up green and according to QNAP, no errors whatsoever.

So rather than replace the drive, I added the drive and expanded the RAID.
This is where things went sideways, see below :

114 Information 2018-06-29 08:33:46 System 127.0.0.1 localhost Drive 2 plugged in.
115 Information 2018-06-29 13:05:40 System 10.0.0.250 localhost [HDD SMART] HDD 4 Rapid Test started.
116 Information 2018-06-29 13:07:43 System 127.0.0.1 localhost [HDD SMART] HDD 4 Rapid Test result: Completed without error(s).
117 Information 2018-06-29 13:11:59 System 127.0.0.1 localhost [Mirror Disk Volume: Drive 3 4] Do RAID Migration to RAID5.
118 Information 2018-06-29 13:18:24 System 127.0.0.1 localhost [Qsync] Database self-recovery started.
119 Information 2018-06-29 17:44:48 System 127.0.0.1 localhost [RAID5 Disk Volume: Drive 3 4 2] Start rebuilding.
120 Warning 2018-06-29 17:45:53 System 127.0.0.1 localhost [Harddisk 3] I/O error, sense_key=0x3, asc=0x11, ascq=0x4, CDB=28 00 00 6f 75 88 00 04 00 00 .

So at number 117 is where the RAID migration started and at number 120 things went wrong. It seems the migrating has not started although the NAS thinks its in RAID 5 already?
When I got back to the NAS the raid was inactive, the previous "faulty" hard drive was now showing up as most definitely faulty and the whole thing was just a mess.

Question is, is this recoverable at all? I'm not sure what info to provide as its such a mess at the moment.
I've tried recovering the RAID via the Raid Management (this just fails without explanation), can anyone shed some light on this?

[storageman]

As you've found out really bad idea to migrate when a drive shows faulty.
What does "md_checker" show in Putty/SSH?

Do you want to delete the other post?

[asteynberg]

Yip not doing that again, ok ran md_checker :

Legacy Firmware Detected!
Scanning disks...

RAID metadata found!
UUID: 7f6cbc7a:ddab3d8d:cc843aa8:84c679b4
Level: raid5
Devices: 3
Name: md3
Chunk Size: 64K
md Version: 1.0
Creation Time: Jun 29 13:12:05 2018
Status: OFFLINE
===============================================================================
Disk | Device | # | Status | Last Update Time | Events | Array State
===============================================================================
3 /dev/sdc3 0 Active Jul 2 07:33:28 2018 25463 Uuu
2 /dev/sdb3 1 Active Jul 2 07:33:28 2018 25463 uUu
4 /dev/sdd3 2 Rebuild Jul 2 07:33:28 2018 25463 uuU
===============================================================================


Took care of the double post..

[asteynberg]

Also forgot to mention, I do have a backup settings file somewhere before this whole fiasco took place, if that might help

[storageman]

Try running this
"/etc/init.d/init_lvm.sh"
Does it start to rebuild?

[asteynberg]

-sh: /etc/init.d/init_lvm.sh: No such file or directory

Not sure why?

[storageman]

ok try this
"mdadm -AfR /dev/md3 /dev/sdc3 /dev/sdb3 /dev/sdd3"


This kicks a pending rebuild state.
"echo repair > /sys/block/md3/md/sync_action"

[asteynberg]

EDIT : Sorry didnt see first part of your post, will try again now

[storageman]

What firmware are you on?

[asteynberg]

Darn, no luck.
mdadm -AfR /dev/md3 /dev/sdc3 /dev/sdb3 /dev/sdd3 = mdadm: device /dev/md3 already active - cannot assemble it
echo repair > /sys/block/md3/md/sync_action = no such file etc...
Firmware is 4.2.6 Build 20180531

[storageman]

Hmm, out of ideas.

try "mdadm --stop /dev/md3"
Then "mdadm -AfR /dev/md3 /dev/sdc3 /dev/sdb3 /dev/sdd3" again

If this fails, whichever of these disks is faulty try:

3 /dev/sdc3 0 Active Jul 2 07:33:28 2018 25463 Uuu
2 /dev/sdb3 1 Active Jul 2 07:33:28 2018 25463 uUu
4 /dev/sdd3 2 Rebuild Jul 2 07:33:28 2018 25463 uuU


"mdadm -AfR /dev/md3 /dev/sdc3 /dev/sdb3 /dev/sdd3" replacing with the word "missing"

[asteynberg]

Thanks for your help so far in anyways, no this is like a one in a million problem!

Ok so first result :
[~] # mdadm -AfR /dev/md3 /dev/sdc3 /dev/sdb3 /dev/sdd3
mdadm: failed to RUN_ARRAY /dev/md3: Input/output error

Then the missing one :
mdadm: missing has no superblock - assembly aborted

Fire off a help ticket so QNAP as well, just seems if this RAID is totally screwed...

[storageman]

Sorry, I think commands I have are failing because this is Cat1 model.
I would start again and rebuild from backups.

[asteynberg]

Ok cool, no problem, thanks very much for the help I really do appreciate it!

[Cpt_Killepi]

@Storageman is there chance to get your help with this incident?
viewtopic.php?f=25&t=161255&p=789705&hi ... RP#p789705