Recovering RAID5 with missing superblocks

[mikkelsj]

Hi all,

Yesterday, my 869 PRO rebooted itself with 2 disks shown as missing in the storage manager.
I had seen one disk missing before, with the solution being to eject the drive, reboot and insert it again - this time it did not help.

Troubleshooting a lot over the night, yielded a somewhat cause for this behaviour. Seems that 5 of 8 drives have missing superblock.
Running e2fsck with a backup superblock has taken over 20 hours now, and I am not sure it will fix anything.

I CAN still assemble and mount the array, but i am unsure how valid the data is now.

How can i proceed to get my data up and running? At least to do a backup and then recreate the array.

Any help is much appreciated

P.S. I have already submitted a request to QNAP support, but seeing as this is a weekend, I am not expecting a reply before tomorrow.

[pwilson]

Hi all,

Yesterday, my 869 PRO rebooted itself with 2 disks shown as missing in the storage manager.
I had seen one disk missing before, with the solution being to eject the drive, reboot and insert it again - this time it did not help.

Troubleshooting a lot over the night, yielded a somewhat cause for this behaviour. Seems that 5 of 8 drives have missing superblock.
Running e2fsck with a backup superblock has taken over 20 hours now, and I am not sure it will fix anything.

I CAN still assemble and mount the array, but i am unsure how valid the data is now.

How can i proceed to get my data up and running? At least to do a backup and then recreate the array.

Any help is much appreciated

P.S. I have already submitted a request to QNAP support, but seeing as this is a weekend, I am not expecting a reply before tomorrow.

You sure you want our help.... You have a strange way of asking for help....

• No firmware version information provided.
• No Make/Model of drive information provided.
• No Drive Configuration information provided.

Please review article: When you're asking a question, please include the following.

[mikkelsj]

You sure you want our help.... You have a strange way of asking for help....

• No firmware version information provided.
• No Make/Model of drive information provided.
• No Drive Configuration information provided.

Please review article: When you're asking a question, please include the following.

Easy now, all you have to do is ask...

Firmware is 4.0.2.

Harddrives are WD Green 2TB.

Drive configuration is RAID5.

This is output of mdadm -D:

/dev/md0:
Version : 00.90.03
Creation Time : Wed Aug 10 21:41:21 2011
Raid Level : raid5
Array Size : 13663619200 (13030.64 GiB 13991.55 GB)
Used Dev Size : 1951945600 (1861.52 GiB 1998.79 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Sun Nov 17 18:23:44 2013
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

UUID : 1708ac76:d56fdf46:915c1d7d:42b8c178
Events : 0.11368539

Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 8 19 1 active sync /dev/sdb3
2 8 35 2 active sync /dev/sdc3
3 8 51 3 active sync /dev/sdd3
4 8 83 4 active sync /dev/sdf3
5 8 67 5 active sync /dev/sde3
6 8 115 6 active sync /dev/sdh3
7 8 99 7 active sync /dev/sdg3

[teracow]

.

[mikkelsj]

Harddrives are WD Green 2TB.

This is a little vague...

Please login via SSH as "admin" and run the following to identify your drives:

Code: Select all

echo ; hdparm -i /dev/sd[a-z] 2>/dev/null | grep -E "Model|/dev/" | sed -e ':a; /:$/ { N; s/:\n/ ,/; ba; }' | sed 's/Model=//;s/FwRev=//;s/  */\ /g' | awk 'BEGIN { FS=","; printf "%-15s %-30s%s\n", "device ID", "make \& model", "firmware version" ; printf "%-15s %-30s%s\n", "--------------", "----------------------------", "---------------" } ; { printf "%-15s%-30s%s\n",$1,$2,$3 }' ; echo

Please post the output back here.

Sorry, they are Seagate - green though

/dev/sda ST2000DL003-9VT166 CC32
/dev/sdb ST2000DL003-9VT166 CC32
/dev/sdc ST2000DL003-9VT166 CC32
/dev/sdd ST2000DL003-9VT166 CC32
/dev/sde ST2000DL003-9VT166 CC3C
/dev/sdf ST2000DL003-9VT166 CC3C
/dev/sdg ST2000DL003-9VT166 CC3C
/dev/sdh ST2000DL003-9VT166 CC3C

[teracow]

.

[mikkelsj]

/dev/sda ST2000DL003-9VT166 CC32

Your drives are Not Recommended.

You will continue having trouble with those drives as they are known to have issues. You may not be able to recover that unit. This is why backups should be performed regularly. No good trying to recover the array so you can perform a backup.

Well thanks - but that **

When were these drives added to the "not recommended"-list?
I bought mine about a year ago and do not seem to remember them being there.
Very convenient of QNAP to add some of the most used drives to the list, AFTER users have been having issues...

How difficult would it be to recover the RAID? Including data recovery services.

[teracow]

.

[pwilson]

/dev/sda ST2000DL003-9VT166 CC32

Your drives are Not Recommended.

You will continue having trouble with those drives as they are known to have issues. You may not be able to recover that unit. This is why backups should be performed regularly. No good trying to recover the array so you can perform a backup.

When were these drives added to the "not recommended"-list?

They have been non-recommended since at least 2010.

I bought mine about a year ago and do not seem to remember them being there.

I'd recommend always checking the QNAP 3.5" HDD Compatibility List before purchasing your drives going forward. It is makes you feel any better I made the same mistake - I too owned 4 of these horrible Seagate ST2000DL003 drives. Since replacing them with recommended drives from the list, I have not experienced any further issues.

[adprom]

Hi all,

Yesterday, my 869 PRO rebooted itself with 2 disks shown as missing in the storage manager.
I had seen one disk missing before, with the solution being to eject the drive, reboot and insert it again - this time it did not help.

Troubleshooting a lot over the night, yielded a somewhat cause for this behaviour. Seems that 5 of 8 drives have missing superblock.
Running e2fsck with a backup superblock has taken over 20 hours now, and I am not sure it will fix anything.

I CAN still assemble and mount the array, but i am unsure how valid the data is now.

What 2 drive bays are showing as missing?

The reason I ask is that from what you have described it is highly unlikely the issue is related to the specific drives as others have suggested and much more likely to be related to an issue a couple of us have had. In every case, two drives disappear.

See http://forum.qnap.com/viewtopic.php?f=25&t=87089 and http://forum.qnap.com/viewtopic.php?f=142&t=83730

If this is what is happening on yours then it validates that this is likely a firmware issue and not a drive issue! Get in contact with QNAP official support.

For those who can't extend their focus behind drive model, have you actually considered the chances of 2 drives missing at the exact same time on reboot, that were working before? The chances due to simultaneous drive failure, where they were working fine before is infinitesimally small.

It would appear that this particular issue relates specifically to the 869 pro model more than anything else.

[mikkelsj]

What 2 drive bays are showing as missing?

The reason I ask is that from what you have described it is highly unlikely the issue is related to the specific drives as others have suggested and much more likely to be related to an issue a couple of us have had. In every case, two drives disappear.

See http://forum.qnap.com/viewtopic.php?f=25&t=87089 and http://forum.qnap.com/viewtopic.php?f=142&t=83730

If this is what is happening on yours then it validates that this is likely a firmware issue and not a drive issue! Get in contact with QNAP official support.

For those who can't extend their focus behind drive model, have you actually considered the chances of 2 drives missing at the exact same time on reboot, that were working before? The chances due to simultaneous drive failure, where they were working fine before is infinitesimally small.

It would appear that this particular issue relates specifically to the 869 pro model more than anything else.

Thanks for the insigts!

For me, it was drives 7 and 8 though.
Tried my former approach of ejecting, rebooting and inserting again - this made the raid, disks and volume disappear for the WebGUI.

Running dmesg just shows a lot of segfaults, as well as sunning config_util and storage_boot_init manually.

In relation to FW, i was wondering why it said to upgrade to 4.0.4 one day, and the next it didn't. Did QNAP retract it? Any others notice this?

[adprom]

For me, it was drives 7 and 8 though.
Tried my former approach of ejecting, rebooting and inserting again - this made the raid, disks and volume disappear for the WebGUI.

Running dmesg just shows a lot of segfaults, as well as sunning config_util and storage_boot_init manually.

In relation to FW, i was wondering why it said to upgrade to 4.0.4 one day, and the next it didn't. Did QNAP retract it? Any others notice this?

That's interesting. That is in line with what we are seeing where it is always consecutive drive bays which is consistent with what myself and the other poster saw. Yes, 4.0.4 was retracted.

Definitely contact qnap support.

[mikkelsj]

For me, it was drives 7 and 8 though.
Tried my former approach of ejecting, rebooting and inserting again - this made the raid, disks and volume disappear for the WebGUI.

Running dmesg just shows a lot of segfaults, as well as sunning config_util and storage_boot_init manually.

In relation to FW, i was wondering why it said to upgrade to 4.0.4 one day, and the next it didn't. Did QNAP retract it? Any others notice this?

That's interesting. That is in line with what we are seeing where it is always consecutive drive bays which is consistent with what myself and the other poster saw. Yes, 4.0.4 was retracted.

Definitely contact qnap support.

A short update: I managed to get the RAID shown in the WebGUI, by deleting the contents of /etc/raidtab and /etc/config/mdmadm.conf and rebooting(of course with a backup, as per QNAP's own guide: http://unofficialqnapsupport.net/?page_id=1336)

Still getting missing drives 7 and 8 from the RAID, but running a bad blocks scan now on the one disk that shows in physical - it worked before.

Any other ideas are still welcome, while i wait for QNAP support.

[fantomas]

The only idea is to replace failing drives ASAP instead of pulling them out and back in.
It's not QNAP thinking drives are bad - it's drives not responding or returning errors.

I'd advise you buy new drives and configure new RAID - with 8 drives, I recommend using RAID6 when possible.