RAID5 - One disk Error, second SMART Warning - State : clean, degraded

[mikolajj]

Hello,
I have TS-451 with Firmware version: QTS 5.0.1.2346, 8GB RAM
I created One RAID Group 4 x 2TB.

Last week I got SMART 197 Current_Pending_Sector on my Drive 4

197 Current_Pending_Sector error.png

So I've open web browser, looked for replace procedure here
and fallow the "Replacing a faulty disk when there is no spare disk in the enclosure' - take drive out, put new one, wait for beep, check if Rebuilding started automaticly - it did.

Next day after work I was checking Rebuilding status and I saw two things.
- Drive 4 was marked as Error
- Drive 2 had a SMART warning - also "197 Current_Pending_Sector"

Not thinking to much I took next HDD went to my Qnap and replaced Drive 4 (the one with Error, I put there yesterday) - same procedure as before.
But this time I got a different info in Qnap web panel.
RAID was degraded - all my shares disappear, all syncs with cloud reported an error.
So it looks like HDD was disconnected and I should "Recover RAID group"
When I try to "Recover RAID group" I got en error like this:

failed to recovery.png

I'm thinking that beside this SMART Warning only one drive failed.
All data should be on
- Disk 1
- Disk 2
- Disk 3
and RAID should function as always (with no protection) - but it's not.

What I wanted to do was:
- to make sure all data is synced between Disk 1-3
- add drive 4, wait for Rebuild
- replace Drive 2 with new drive

Now my status looks like this:
Drive 1 - Good
Drive 2 - SMART Warning
Drive 3 - Good
Drive 4 - Good <-- this one is the new one and empty

And I can't find any options in qnap web panel to Rebuild (even with SMART error)
I googled a lot about RAID5 and SMART Errors at the same time but no solution came well described.

I have a backup
Please don't write an answer that I should cleanup everything and build everything from scratch - I know I can do that - but I also want to learn sth from this case.
I would like to know what probably happened, and is it possible to repair the system from command-line using md_checker, mdadm or other tools?

So the question is what can I do in such a situation?

Best regards and thanks in advance for your tips and comments
MJ

PS I'm a little disappointed with Qnap as one of the best NAS solutions ( lets say at least top 3 ) - I think system should handle this with more user friendly way

[dolbyman]

- So you had a faulty drive (4)
- swapped the drive (4)
- During the rebuild another drive threw errors (2)
- The RAID is busted
- Rebuild the RAID with good disks and restore from backups

[mikolajj]

My situation was almost like you wrote about it:

- So you had a faulty drive (4)
- swapped the drive (4)
- During the rebuild same swapped drive threw errors (4)
- swapped the drive (4) again
- then a SMART Warning appear (2)
- The RAID is busted
- Rebuild the RAID with good disks and restore from backups

So the drive with Warning should be fine - at least for a moment to take data out of it, rebuild raid from disks 1,2,3
then add disk 4, rebuild, replace drive 2, rebuild and done.

I can see you are a super active guru here ; - ) do you have enough time to explain what happened during first and second rebuild- more step by step way?

[diwiwi]

Disk 2 was found faulty during the RAID rebuild after replacing the drive 4.
The RAID 5 cannot rebuild with 2 failed drives.
The RAID cannot recover.
That's it.

But fortunately you have backups

-

[mikolajj]

OK, is there any way to force rebuild even if there is a SMART Warning?
Even if rebuild wont recover one file it's still less work then recreate everything from backup

[dolbyman]

The sectors are pending .. if the drive decides that these are readable and can be relocated, you might be able to rebuild the RAID.

You could also clone that drive onto a new drive and hope that the defect sectors are not anywhere near data (you would use the cloned drive without the smart errors to reassemble the RAID)