RAID1 broke with faulty disk, now shows "inactive" state

Questions about SNMP, Power, System, Logs, disk, & RAID.
Albercik
Starting out
Posts: 11
Joined: Fri Aug 27, 2021 1:46 am

RAID1 broke with faulty disk, now shows "inactive" state

Post by Albercik »

Hello All!

I have a older QNAP model TS-219P (two bay). It was working fine with one 4TB WD RED hard drive. But...
I wanted the data to more secure, so I've bought a second hard drive (exactly the same, as the first one), insterted it into bay 2 and the drive was recognized.
Then, I've upgraded the volume from "Single disk" to "RAID1". The volume started to rebuild and that was OK.
Suddenly - it turned out, that my new hard drive is faulty (during RAID building) and QNAP posted a "read/write error" (also there were some errors in SMART) and it dropped the rebuild.
So... I've turned off the NAS, took the faulty HDD out and sent it back to RMA.
But then, when the NAS has started with only the first hard drive - it showed the RAID status "inactive" and I can't recover it.
After clicking a "recover" button - the error is thrown: [Mirror Disk Volume: Drive 1 2] RAID Recovery failed: RAID level does not match.
I know it does not match, as the RAID1 requires at least 2 HDDs, but why doesn't it run in degraded mode, but instead - it's inactive?

I've tried to SSH/SCP to the NAS, but I still can't access my files (nothing unusual - the raid is inactive, so the data won't be visible...).

Can you please help my out on how can I recover my files?

I'm waiting for the new hard drive to come, but it will take few days and I'm stuck now :(
User avatar
dolbyman
Guru
Posts: 35024
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: RAID1 broke with faulty disk, now shows "inactive" state

Post by dolbyman »

Always have backups .. a tear filled lesson many have learned the hard way (a RAID is NOT a backup)

To access your files, use a linux live boot to read your existing disk, save the files to another location. (that only works because your NAs is a decade old, with a new NAS, this is not an option)

After that is done, troubleshooting of the current NAS can begin.
Albercik
Starting out
Posts: 11
Joined: Fri Aug 27, 2021 1:46 am

Re: RAID1 broke with faulty disk, now shows "inactive" state

Post by Albercik »

Hi dolbyman and thank you for the reply.

I use this NAS to store my movies/music/photos (strictly personal use) so making a backup of it is quite ridicolous. I also have another QNAP for my businness stuff and it has 2x3TB HDD and another one 6TB that's performing as backup (backup is created every week from the two 3TB RAID1 HDDs). But for some personal stuff it's quite pointless to have a another NAS to backup stuff... At least my wife wouldn't be happy to have another buzzing brick in home :D

OK, back to the topic. I have the disk connected to the Raspberry Pi running Debian, but still have trouble recovering the files. I can't mount the drive as it's "linux_raid_member" type, and I can't start RAID with mdadm because "mdadm: failed to RUN_ARRAY /dev/md1: Invalid argument" and "mdadm: Not enough devices to start the array.". I remember, some time ago I've recovered the data from my customer's QNAP by connecting it to a Windows 2008R2 server machine, but I ain't go no access to it at the moment.

Any advise (except of "do backup next time") will be much appreciated!
User avatar
dolbyman
Guru
Posts: 35024
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: RAID1 broke with faulty disk, now shows "inactive" state

Post by dolbyman »

Well if the files are not important .. a backup is probably not needed...an external USB drive for backup is certainly not a big extra

Windows will not be able to read any of this by itself, it can't handle the linux fs.

what is the output of

Code: Select all

mdadm --assemble --scan
Albercik
Starting out
Posts: 11
Joined: Fri Aug 27, 2021 1:46 am

Re: RAID1 broke with faulty disk, now shows "inactive" state

Post by Albercik »

I've inserted the drive back to QNAP.

Qnap says

Code: Select all

mdadm: No suitable drives found for /dev/md0
Debian was properly assembling raid for SDA1, SDA2 and SDA4 (they're few houndred MBs), but it was saying that the raid cannot be started for /dev/sda3 (my 3,6TB volume) because of a drive missing.

Gosh, why the heck, did I tried to do that in a first place? Awww yeah - I wanted the data to be more secure :D :D

[EDIT]
Oh, and

Code: Select all

cat /proc/mdstat
says:

Code: Select all

md2 : active raid1 sda2[0]
                 530128 blocks super 1.0 [2/1] [U_]

md13 : active raid1 sda4[0]
                 458880 blocks super 1.0 [2/1] [U_]
                 bitmap: 6/8 pages [24KB], 32KB chunk

md9 : active raid1 sda1[0]
                 530112 blocks super 1.0 [2/1] [U_]
                 bitmap: 8/9 pages [32KB], 32KB chunk

unused devices: <none>
User avatar
dolbyman
Guru
Posts: 35024
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: RAID1 broke with faulty disk, now shows "inactive" state

Post by dolbyman »

Any output for my command given ?

You converted the single to RAID1 for failure tolerance, not to make the data more secure .. common misconception (also never do backups internally, always external .. in regards to your business NAS )
Albercik
Starting out
Posts: 11
Joined: Fri Aug 27, 2021 1:46 am

Re: RAID1 broke with faulty disk, now shows "inactive" state

Post by Albercik »

dolbyman wrote: Sat Aug 28, 2021 2:06 am Any output for my command given ?

You converted the single to RAID1 for failure tolerance, not to make the data more secure .. common misconception (also never do backups internally, always external .. in regards to your business NAS )
As per your question - I've posted the output in my previous reply.

By "more secure" I meant exacly "failure tolerance" ;) It's more secure because of higher fail-safe protection, and not because of the permissions/security case.
Albercik
Starting out
Posts: 11
Joined: Fri Aug 27, 2021 1:46 am

Re: RAID1 broke with faulty disk, now shows "inactive" state

Post by Albercik »

OK, finally got somwhere closer.

After connecting it to my raspberry and entered:

Code: Select all

mdadm --create /dev/md1 --metadata=1.0 --level=1 --raid-devices=2 /dev/sda3 missing
And the Linux said:

Code: Select all

mdadm: array /dev/md1 started.
Then I've perfomed:

Code: Select all

mount -o ro /dev/md1 /mnt/qnap
And the Linux said:
nothing (meaning no errors found).

I went to /mnt/qnap and I can see the file structure, but... I can't access any for my shared folders - they're seen as files?

Code: Select all

root@raspbx:/mnt/qnap# ls -l
ls: cannot access 'Multimedia': Structure needs cleaning
ls: cannot access 'Download': Structure needs cleaning
ls: cannot access 'Web': Structure needs cleaning
ls: cannot access 'Public': Structure needs cleaning
ls: cannot access 'homes': Structure needs cleaning
ls: cannot access 'Video': Structure needs cleaning
ls: cannot access 'Music': Structure needs cleaning
ls: cannot access 'Maciek_stona': Structure needs cleaning
total 24
-rw------- 1 root root  6144 Jan 11  2021 aquota.user
d????????? ? ?    ?        ?            ? Download
d????????? ? ?    ?        ?            ? homes
drwx------ 2 root root 16384 Jan 11  2021 lost+found
d????????? ? ?    ?        ?            ? Maciek_stona
d????????? ? ?    ?        ?            ? Multimedia
d????????? ? ?    ?        ?            ? Music
d????????? ? ?    ?        ?            ? Public
d????????? ? ?    ?        ?            ? Video
d????????? ? ?    ?        ?            ? Web
I don't quite understand what does the "Structure needs cleaning" message means and I'd also prefer not to take any action on the file structure as I ain't got no backup of this disk (I don't have another 4TB drive as my second one is still on RMA to clone the disk).
Any thoughts?
User avatar
dolbyman
Guru
Posts: 35024
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: RAID1 broke with faulty disk, now shows "inactive" state

Post by dolbyman »

Buy another disk disk and clone on that (keep the new RMA disk as cold spare or better, as external backup disk)
Albercik
Starting out
Posts: 11
Joined: Fri Aug 27, 2021 1:46 am

Re: RAID1 broke with faulty disk, now shows "inactive" state

Post by Albercik »

dolbyman wrote: Fri Sep 03, 2021 7:21 pm Buy another disk disk and clone on that (keep the new RMA disk as cold spare or better, as external backup disk)
Unfortunately - I will have money back, as that model is no longer available on the market (WE RED 4TB WDEFRX), so I'll definitly have to buy another one. The only problem is that I'd like the new disk to work in RAID 1 with the old one, so I needs to be exactly the same size in bytes (or bigger) and It's quite hard to find as they're all just "4TB" (to be exact - it's 4 Tebi Bytes)...

Anyway - I still ned to gain an access to my files, so It would be really nice if somebody could help me out with accessing those folders :)

[EDIT]
What's even more odd - I've disconnected my HDD from the RPi and have insertted it back to my QNAP. Now, the raid is no longer in "Inactive" state, rather than works in "Degraded" mode, however... I still cannot access my files. File station says that "The file or directory does not exists" and "ls -l" gives this output:

Code: Select all

[/share/MD0_DATA] # ls -l
ls: ./Multimedia: Input/output error
ls: ./Download: Input/output error
ls: ./Web: Input/output error
ls: ./Public: Input/output error
ls: ./homes: Input/output error
ls: ./Video: Input/output error
ls: ./Music: Input/output error
ls: ./Maciek_stona: Input/output error
What to do next?

[EDIT2]
I've checked via fdisk on qnap and it says:

Code: Select all

Disk /dev/md0: 0 MB, 0 bytes
2 heads, 4 sectors/track, 0 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Also - mdstat says:

Code: Select all

Personalities : [raid1] [linear] [raid0] [raid6] [raid5] [raid4]
md0 : active raid1 sda3[0]
                 3905449536 blocks super 1.0 [2/1] [U_]
                 bitmap: 4/30 pages [16KB], 65536KB chunk

md2 : active raid1 sda2[0]
                 530128 blocks super 1.0 [2/1] [U_]

md13 : active raid1 sda4[0]
                 458880 blocks super 1.0 [2/1] [U_]
                 bitmap: 6/8 pages [24KB], 32KB chunk

md9 : active raid1 sda1[0]
                 530112 blocks super 1.0 [2/1] [U_]
                 bitmap: 8/9 pages [32KB], 32KB chunk

unused devices: <none>
It seems that the /dev/sda3 is in raid /dev/md0, but the system claims it's size is 0B.
Albercik
Starting out
Posts: 11
Joined: Fri Aug 27, 2021 1:46 am

Re: RAID1 broke with faulty disk, now shows "inactive" state

Post by Albercik »

Hi Everyone!

Finally got my money back for the faulty harddrive so I've quickly bought a new one (Hitachi this time). I wanted to make a backup of the existing partition to the new drive, before trying to recover anything from it, and... well... this was a mistake!
I've connected both disks to one of my PCs and ran Gparted Live CD to copy the partition. What I didn't know back then is that this son of a b**** will perform a FSCK first and will try to repair the disk without even asking me if I want to do this and... simply deleted all the stuff it thought was broken!

So I've ended up with almost completely clean hard drive...

A the moment I'm trying to save everything that can be saved using Photorec, but it's output is a real mess! All of my websites/databases are lost and all of the files are named 'f12345687.ext' (.ext is a proper extension). Gosh... The only positive thing is that this NAS was holding my movies and music, and all of the important files (documents, photos) are on the second NAS, that's working fine. Never again!
Thisisnotmyname
Easy as a breeze
Posts: 447
Joined: Mon Nov 19, 2018 1:21 am

Re: RAID1 broke with faulty disk, now shows "inactive" state

Post by Thisisnotmyname »

It's tough to learn the lesson this way but backups are important. Any time you have data in a single location (even if that location is a NAS) you run the chance of losing it. NAS systems with RAID (other than RAID0) provide some tolerance for hardware failures themselves but there's still the chance that you can lose the entire array and thus your data if it does not exist anywhere else. For most home users the 3-2-1 back-up strategy works best:

THREE copies of your data
in TWO different media types (i.e. one on hard drive and one on tape, in the home space most accept local drive and RAID arrayed NAS as two different media these days)
at least ONE of which is in a different location.

Simplest approach would be to have your originals on your primary device (e.g. computer) a second copy on a local external device (typically a NAS for anyone here but could be as simple as an external drive) and then back all of that up off site to a cloud service such as Amazon S3.

Sorry that you are getting bitten, the good news is that now you know and can take steps to avoid a repeat in the future.
Albercik
Starting out
Posts: 11
Joined: Fri Aug 27, 2021 1:46 am

Re: RAID1 broke with faulty disk, now shows "inactive" state

Post by Albercik »

Heh, i have a daily backups made from my PC to my other NAS, but the most critical data exists on OneDrive anyway, so I can say I have a tripple backup... I use both of my NAS'es as an archive so there's no point to backup them again to some external media. I only wanted to add some fault tolerance to the other one...
Thisisnotmyname
Easy as a breeze
Posts: 447
Joined: Mon Nov 19, 2018 1:21 am

Re: RAID1 broke with faulty disk, now shows "inactive" state

Post by Thisisnotmyname »

then why are you using photorec?
Albercik
Starting out
Posts: 11
Joined: Fri Aug 27, 2021 1:46 am

Re: RAID1 broke with faulty disk, now shows "inactive" state

Post by Albercik »

Thisisnotmyname wrote: Fri Sep 24, 2021 9:24 am then why are you using photorec?
Because my PC is making backup to my first NAS and the data that was stored on my second NAS was lost. They were not "mission critical", but it would took me huge amount of time to recreate them. Especially - there were the very first songs performed by my son, it's sentimental value was quite big :)

Anyway - after recovering the files with Photorec, I gave a chance to R-Linux running on LinuxMX Live CD. It has recovered almost 80% of my files with the correct folders structure! It has also recovered the data that I wanted the most. I've copied the recovered data to my second 4TB HDD (it's 1.1TB total) and reinstalled the WD hard drive back to NAS. Then I've formatted it and setup the folder's structure as it was before. Of course - still the RAID runs in degraded mode (there's one drive still missing).

What I think I'm gonna do now is to copy the recovered data back to my NAS, then I'll lend/buy a 2TB HDD to make a backup of it. After that - I'll clean the other 4TB HDD and install it in the NAS and let the RAID rebuild. Am I thinking correctly?
Post Reply

Return to “System & Disk Volume Management”