Raid 5 array not active - what can I do?

[pkf]

Hello.

I've got a problem with one of my old Qnaps. It's 5 bay TS-509. After restarting some time ago there appeared that 5th disk is totally broken and 4th reports smart error. Raid volume was degreaded but working. So, we tried to replace 5th disk first. And something strange happened - disk number 1 reported read error too. Having nothing to loose, old 5th disk was placed again and I restarted whole thing. Now the situation is like that:

opera_2019-10-21_13-23-01.png

So, generally like at the start point with the exception that the array is now in 'not active' instead of degreaded mode. The question in what should I do now to try to save some of my data. Try to replace 5th disk again or to test some commands mentioned in this topic: viewtopic.php?f=25&t=142897 ? Would be grateful if someone more familiar gave me some hints.

And also: why I don't have md_checker? Is my device too old?

Big thanks.

[dolbyman]

your device is too old for md_checker

bring your qnap unit to a data recovery compnay...you have 2 (or more) busted drives in raid5 ...thats it

that is why you need to run backups..raid is not a backup

[mlbarnes]

The second device that you say is 'busted' is only reporting SMART errors!
Some drives with SMART errors run for years like that.
If 4 out of 5 of those disks surely it isn't time to be paying for piecemeal recovery?
This is RAID 5 - should be working with only one drive down, let alone a write off!

[mlbarnes]

I'm no expert (wouldn't have known md_checker was not going to work), but I would be happy to see some output from tests of some sort...
Maybe... SSH and run

Code: Select all

blkid

then post that output? (let me know if you don't know how to SSH!).

Also, much respect my colleague above, he is probably right, but I would try running

Code: Select all

md_checker

too and confirming.
No offence - I have just had bad advice before...

Another output that might be informative -

Code: Select all

qcli_storage

- post results.

Like I say, I'm learning because of a QNAP Failure (my second significant data loss with this box...) and trying to figure out how to salvage it myself.
I'm pretty sure there's no harm at least in my advice so far though!

Good luck...

[dolbyman]

OP had a defective drive (5) and and two drives with smart/io errors (4&1) , that is three devices not working in a RAID5 ... the array is bust

a recovery service might be able to assemble data from the broken/error drives into a working array, but that is outside of the scope of this forum (and any additional uptime and fondling of the drives might reduce recovery chances even further)

DO BACKUPS people

[pkf]

Here are some outputs of commands mentioned above:

Code: Select all

[~] # blkid
/dev/sdx1: UUID="d020c7e7-6fa6-40e5-9a12-1f336ba99cd6" TYPE="ext2"
/dev/sdx2: LABEL="QTS_BOOT_PART2" UUID="2599eda2-7af4-4a69-96b9-9010423dcd0d" TYPE="ext2"
/dev/sdx3: LABEL="QTS_BOOT_PART3" UUID="418f0566-29f4-4eb1-b376-b2d7a32f0b78" TYPE="ext2"
/dev/sdx5: UUID="651e3bf1-81be-46c8-bcfe-c3b23b0b11a7" TYPE="ext2"
/dev/sdx6: UUID="d0936675-ba61-4bf1-abf1-fc3e319d167c" TYPE="ext2"
/dev/sdd1: UUID="48637433-d7cb-e402-fa37-45c78b36df19" TYPE="mdraid"
/dev/sdd2: UUID="c75679eb-3066-4f20-9aec-ae3e397c5cdc" TYPE="swap"
/dev/sdd3: UUID="b846f80a-c5bb-ddb4-f007-511c2d285aaf" TYPE="mdraid"
/dev/sdd4: UUID="8684f0e3-a9e8-55f8-36b7-3bc167faab2d" TYPE="mdraid"
/dev/sdc1: UUID="48637433-d7cb-e402-fa37-45c78b36df19" TYPE="mdraid"
/dev/sdc2: UUID="08e1ccfd-ec61-396b-5f8f-fd03f3da3c80" TYPE="mdraid"
/dev/sdc3: UUID="b846f80a-c5bb-ddb4-f007-511c2d285aaf" TYPE="mdraid"
/dev/sdc4: UUID="8684f0e3-a9e8-55f8-36b7-3bc167faab2d" TYPE="mdraid"
/dev/sdb1: UUID="48637433-d7cb-e402-fa37-45c78b36df19" TYPE="mdraid"
/dev/sdb2: UUID="c75679eb-3066-4f20-9aec-ae3e397c5cdc" TYPE="swap"
/dev/sdb3: UUID="b846f80a-c5bb-ddb4-f007-511c2d285aaf" TYPE="mdraid"
/dev/sdb4: UUID="8684f0e3-a9e8-55f8-36b7-3bc167faab2d" TYPE="mdraid"
/dev/sda1: UUID="48637433-d7cb-e402-fa37-45c78b36df19" TYPE="mdraid"
/dev/sda2: UUID="c75679eb-3066-4f20-9aec-ae3e397c5cdc" TYPE="swap"
/dev/sda3: UUID="b846f80a-c5bb-ddb4-f007-511c2d285aaf" TYPE="mdraid"
/dev/sdareal4: UUID="8684f0e3-a9e8-55f8-36b7-3bc167faab2d" TYPE="mdraid"
/dev/sde1: UUID="48637433-d7cb-e402-fa37-45c78b36df19" TYPE="mdraid"
/dev/sde2: UUID="dfb79306-8ae4-47ee-9357-23fb2e906ede" TYPE="swap"
/dev/sde3: UUID="b846f80a-c5bb-ddb4-f007-511c2d285aaf" TYPE="mdraid"
/dev/sde4: UUID="8684f0e3-a9e8-55f8-36b7-3bc167faab2d" TYPE="mdraid"
/dev/md9: UUID="2b4cae97-de1e-434e-a1c2-b09892121aee" TYPE="ext3"
/dev/sda4: UUID="4b148e5d-3650-43b7-a66e-02a636a22d44" TYPE="ext3"

[~] # md_checker
-sh: md_checker: command not found

[~] # qcli_storage
-sh: qcli_storage: command not found

Yes, I know that raid is not a backup and I have some backups but my life would be much easier if I hadn't to make use of them... Now only one drive is totally defective and one reporting smart error. I don't konw what happened to the other one but now it is not reporting errors. So, maybe there would be a chance... Thank you.

[pkf]

I replaced 5th disk with new one. After system restart, now the volume and my data is visible! But in degreaded mode. So, everything looks like this:

opera_2019-10-22_17-10-46.png

What does that mean - I did not touch disk 1 this time. What to do first? Wouldn't like to make any stupid move now...

[mlbarnes]

Sorry for the slow reply - I hadn't watched this topic, but I would be glad to know if all is now fully resolved?

Yep - I still seem to be at odds with dolbyman, but I think I can see where / why now, and we are both right.
I'm right that you still have hope to salvage your data, which I think you have already realised - it being back up and running and all...
I think your recent post backs up my opinion, but I still say don't take my word for anything!
As far as I can see, what he said about your drives 4&1 were based on what you said had happened before, and was not the current issue.
It makes me worry that nothign is stable though, so for goodness sake get some backups or get some disks replaced ASAP, preferably both.
If you have a simple external drive you could make a backup to, I would do that right now, while the array is rebuilding!

More specifics...
Your first screenshot only showed one disk not working (admittedly another disk had SMART errors).
There was nothing about IO errors (SMART errors are rarely IO errors), but you did say that you had some?
Anyway, based on the screenshot, a RAID 5 array like yours should still be working albeit degraded.
That seems to now be exactly what you have. So all of your data should be fine for now and you just have to wait until the array rebuilds itself.

You should think about (others would probably say DO IT!!) changing that disk that is still reporting SMART errors, but only after the array has completely rebuilt and everything (apart from the smart amber alert on drive 4) shows green.
Then get a new drive, and with the machine still running (yep, nobody tells you that loud enough until you've lost data)...
...eject the amber alert one in slot 4 (check I have this right!) and replace it with the new one. Then hopefully all will be perfect again!

There might be some validity to what DM said about both your drives 1 as well as 4 - yegods, this sounds like an expensive hobby doesn't it?
Are you sure there were io errors on both those before?
WD Red drives I would generally say were my favourite and as solid as any I have used.

Anyway, our thoughts on disks 1 and 4 were where DM and I were differing - as a desperate soul with similar previous experiences I was being optimistic (hopefully not just in denial!) about salvage.
He was being proactive about making you more cautious... and was (hopefully) too quick to tell you to write off all your data (which is the rule on this forum!).

Do keep backups somehow though (you can't argue with the "told you so, it's oblivion for you and there's nothing you can do about it" crew!)...
I myself have worked on the assumption that a working RAID array would be a safe place for stuff I wouldn't be devastated about losing.
I keep backups, but only a small proportion of my data.
Twice now I have nonetheless mourned my data badly enough, last time was my whole life of stuff I'm not currently using.
One or two powercuts and boom 5TB of my life gone despite perfectly healthy disks....
On an up note, I now have a lot of empty storage space!
- once I truly do give up on the salvage that is, until then they are useless precious artefacts gathering dust while I delude myself I will re-decipher them one day....

My whole attitude to QNAP servers has changed (too late) in fact - I think their only role is to keep data reliably online and accessible, not to secure it.
Backups (multiple) are your security...
Which is difficult when you're running a non-profit on a shoestring (why does it asterisk out shoe-string?) and figured £3k was enough to spend on securing 3TB of teaching videos...
As everyone else here will tell you though, you need to spend much more... evidently.
A tape drive probably and shelf storage full of tapes! Assuming you have another £2k+...
I don't, so I'm working on making a Linux box with more flexible and tolerant old disk array solutions and probably SnapRAID and learning much more about data salvage...
Thinking about going on a course in fact - if anyone in the UK (** / London) is actually reading my rambles and has any suggestions let me know...
(Ess-ex gets ** too? - wow, I am foul mouthed it seems...)
Maybe a CompTIA A+ course, but I suspect I will be bored with the less specific components...

Please let us know what has happened since!!

[mlbarnes]

Shorter reply to your what to do now question (bet you already did stuff right? - sorry I'm slow)...
Don't change anything...
While you can access your data, copy it to an external disk.
Then start troubleshooting the reasons for your problems as above...

[dolbyman]

no need for complicated LTO/RDX/etc drives

a simple 8TB backup disk 3.5" with separate power brick) costs 150 bucks
a simple 5TB backup disk 2.5" with only one USB connection costs 120 bucks

I fail to see how that is too expensive to rotate out once in a while to a safe location

Losing data is not a great experience, but pretending there was nothing that could have been done without spending a lot of $$$ or time beforehand is wrong.

And if the data was really really important, a data rescue company can surely recover it, endless trail and errors, executing wrong commands can and will make recovery more expensive or even impossible

[mlbarnes]

Yep, ok.
Agreed. Just always seemed better to put it in the array than to use it as an external additional backup
I was wrong with that. Anyway, sorry - a bit off thread and that's my fault!

I may stick with the QNAP and add a monthly backup...
Plenty of disks to do it now!
Good advice too late learned!

Those tape drives are very tempting though too - I mean £30 on one media to backup everything I have - I would buy ten!
Until you realise the drives are the price of a hundred of them...
Maybe when I have re-accumulated something worth backing up!

Hope things are going well with yours though pkf?

[pkf]

Hi everyone.
Glad that my data on the array is visible, I'm copying it to external drives. Goes really slow as the array is in degreaded mode. Tha plan is to copy everything and set whole thing up from scratch again as I have no idea what to do with that one uninitialized drive and how to make it working on the array together.
As the discussion went to backups etc, I'll tell you what I do on other Qnap where the really important data is used. I use five external 6TB drives, for every day of week and copy everything from Qnap once a day. As on that location there is no one who would change drives manually, we've set up a solution which powers on certain drive for a time needed for copy to be made and than powers the drive off. About once or twice a month there is another copy made by human and taken out of location.

[mlbarnes]

Thanks for the update and glad it is going well.
I'm not sure I can afford a drive for every day of the week, but two backups might be a good idea I guess...
I assume the backup task is automated too?
Probably figure myself a solution if I look into backup station... I did have OffSite backups to another QNAP, but the connection never seemed reliable and of course I was moving everything around (as you are now!) with just the one copy when the blasted thing decided it couldn't read its own arrays despite perfectly working disks...
Sounds like you have less critical data, but be careful clearing everything when you have just that one backup finished - I too feel happier with a 'clean install' type of approach, but two copies of data at all times is a lot more reassuring! Even for data you think isn't that critical - until you lose it ....

[pkf]

Yes, the everyday backup task is automated ofcourse Maybe some day we'll think about one more Qnap...
Regards.