Page 1 of 48

"Disk Failed", then "Disk Unplugged" errors

Posted: Sat Jul 15, 2017 1:32 am
by Sturmie
I purchased my TS-563 back in April this year. Last night, I updated to the firmware from 4.3.3.0210(20170606) to 4.3.3.0238(2017073) and then got a "Host: Disk 3 failed" followed by "Host: Disk 3 unplugged" and then "RAID Group 1 is in degraded mode" (obviously), but after a shutdown/pull drive, put back in/reboot, all disks showed up fine and the RAID group started to rebuild. It got all the way to ~60% and then it happened again this morning...same set of errors and the disk 3 is showing as "not installed" now (I'm at the office and can't pull the drive and re-insert it like I did last night).

Is this a failing drive (these are 8TB Red NAS drives from WD and are less than 3 months old each) or is this a problem with the backplane in my QNAP? It's not unheard of for drives to go bad this quickly, but it's definitely unlikely. I'm fine getting another 8TB Red drive locally and keeping the WD warranty replaced drive as a cold spare, but before I drop $280 on a drive, I want to make sure it's not the QNAP that's the issue.

Thanks.

Re: "Disk Failed", then "Disk Unplugged" errors

Posted: Sat Jul 15, 2017 1:40 am
by dolbyman
is there any SMART attributes on the disk that look out of the ordinary ?

Re: "Disk Failed", then "Disk Unplugged" errors

Posted: Sat Jul 15, 2017 1:49 am
by Sturmie
dolbyman wrote:is there any SMART attributes on the disk that look out of the ordinary ?
Since it says it's "unplugged" in the QNAP interface, I can't get any info on it. This is my first QNAP NAS (my previous experience was with Thecus), but why would the drive "unplug" as opposed to simply removing itself from the RAID? I would think that since the drive is still in the QNAP, I should be able to access it even though it's failed and not part of the RAID group.

Re: "Disk Failed", then "Disk Unplugged" errors

Posted: Sat Jul 15, 2017 10:49 am
by hscorpio15201
Sturmie wrote:I purchased my TS-563 back in April this year. Last night, I updated to the firmware from 4.3.3.0210(20170606) to 4.3.3.0238(2017073) and then got a "Host: Disk 3 failed" followed by "Host: Disk 3 unplugged" and then "RAID Group 1 is in degraded mode" (obviously), but after a shutdown/pull drive, put back in/reboot, all disks showed up fine and the RAID group started to rebuild. It got all the way to ~60% and then it happened again this morning...same set of errors and the disk 3 is showing as "not installed" now (I'm at the office and can't pull the drive and re-insert it like I did last night).

Is this a failing drive (these are 8TB Red NAS drives from WD and are less than 3 months old each) or is this a problem with the backplane in my QNAP? It's not unheard of for drives to go bad this quickly, but it's definitely unlikely. I'm fine getting another 8TB Red drive locally and keeping the WD warranty replaced drive as a cold spare, but before I drop $280 on a drive, I want to make sure it's not the QNAP that's the issue.

Thanks.
I'm glad this was posted. After the last update, I'm having the same exact issue. I have a very similar setup using the TS-653A with 6 x 8 TB WD Red Drives. Basically it starts with drive 3 falling out of the array and then causes drives 4, 5, 6 to follow. I rebooted and it rebuilt fine but after about 12 hours, it occurred again. I opened a support ticket but havent heard back yet. Not hijacking your thread but just wanted to share the similar issue.

Re: "Disk Failed", then "Disk Unplugged" errors

Posted: Sat Jul 15, 2017 10:10 pm
by Sturmie
Oh man, that's not great to hear that almost the exact same scenario is happening to you after the last update. Did you lose any data? For me, only drive 3 dropped out, so *knock on wood* all of my data was still safe.

I actually went and purchased another 8TB Red locally after work (luckily, they pricematched to Newegg) and the array has been rebuilding since last night - currently at 86%. I too opened a ticket and they recommended trying another drive first, which I was obviously doing already. I'll keep you posted.

Re: "Disk Failed", then "Disk Unplugged" errors

Posted: Sat Jul 15, 2017 11:56 pm
by hscorpio15201
Luckily enough I havent lost any data. Its like it forgets that drive 3 exists and then kicks 4 , 5, 6 out. I rebuilt it twice and the data was fine. I'm curious to what support comes back with before I make a call on drive replacement.

Re: "Disk Failed", then "Disk Unplugged" errors

Posted: Sun Jul 16, 2017 11:29 pm
by Sturmie
Well, so far so good. The rebuild finished and the scrubbing process is at 77%. It's looking like it actually was probably the drive itself.

Re: "Disk Failed", then "Disk Unplugged" errors

Posted: Mon Jul 17, 2017 4:31 am
by hscorpio15201
Sturmie wrote:Well, so far so good. The rebuild finished and the scrubbing process is at 77%. It's looking like it actually was probably the drive itself.
Thats great to hear! Since I'm impatient and have a backup, I took a stab at reloading the 6/24 firmware and that went well. Havent had the drive drop issue and was able to make it through multiple file system tests / disk check of number 3. I'm going to monitor but making it a full 24 hours without the issue is a big step. I'll keep the ticket open since I'm curious

Re: "Disk Failed", then "Disk Unplugged" errors

Posted: Mon Jul 17, 2017 5:35 pm
by ServalCat
Hi all. I too have had the same issue. After upgrading QTS disk 3 and 4 dropped. However, the issue may be more complex than just QTS upgrade.
I lost contact with Bay 3 and 4 (brand new 3 GB WD red drives), reinserted them and it was ok for a while. Then went on holiday for a week (shut down NAS first), and couldn't get them back after starting up. I've just reverted a couple of QTS firmwares (from 4.3.3.0238 build 20170703 back to build 20170606) and the disks reappear (but vol 3 has been tagged with abnormal i/o status).

However, I found that people also have had identical sounding problems with the same model as mine QNAP 451A when upgrading RAM. I did this recently also from 2 GB to 8GB. I even bought the expensive Qnap RAM rather than generic laptop brand to avoid problems.

There is more about this here
viewtopic.php?f=182&t=125803

Not sure if any of you folks have upgraded RAM, but it might also influence things. The solution, sadly, was to downgrade RAM back to 2 GB, then no problems.

Re: "Disk Failed", then "Disk Unplugged" errors

Posted: Mon Jul 17, 2017 8:14 pm
by P3R
This isn't something this community can solve. Instead please submit a ticket with Qnap support.

Re: "Disk Failed", then "Disk Unplugged" errors

Posted: Tue Jul 18, 2017 10:51 am
by MrVideo
ServalCat wrote:Not sure if any of you folks have upgraded RAM, but it might also influence things. The solution, sadly, was to downgrade RAM back to 2 GB, then no problems.
Interesting. I have not upgraded my RAM and no issues with drive 3. I do not have a drive 4. That might also make a difference.

Re: "Disk Failed", then "Disk Unplugged" errors

Posted: Tue Jul 18, 2017 6:32 pm
by P3R
ServalCat wrote:The solution, sadly, was to downgrade RAM back to 2 GB, then no problems.
That's not a solution. It can at best be considered a work around to avoid a hardware fault or a huge software bug that in my opinion is unacceptable. If you even have Qnap RAM, you have a completely supported configuration that simply should work. I very much recommend that you open a case with Qnap support as I suggested previously.

Re: "Disk Failed", then "Disk Unplugged" errors

Posted: Tue Jul 18, 2017 9:21 pm
by hscorpio15201
Yea I dont believe the ram was the issue too. I upgraded my memory too but the issue subsided when I reverted the firmware prior to the 7/6 update. I have a ticket open to see what they say but since I reverted, the NAS has been rock solid and all my disk passed their checks.

Re: "Disk Failed", then "Disk Unplugged" errors

Posted: Tue Jul 18, 2017 9:34 pm
by Trexx
From just reading the thread, the common denominator appears to be WD Red drives (varying capacities). Again as was recommended earlier, make sure you all have Helpdesk tickets open with QNAP. It could be a compatibility issue of some kind related to WD firmware.

Re: "Disk Failed", then "Disk Unplugged" errors

Posted: Wed Jul 19, 2017 4:06 am
by ServalCat
Yep. I agree with that all. And learned that I should put quotes around the word "solution". ;-D
My TS451A is back to normal now and has survived for 24 h post error checking without the disk going down again. Like you all, I really want to work out if there is a common denominator to make sure my help ticket does not just get brushed off with the usual "factory reset" approach.
Yes, all my disks are WD reds 2TB in bays 1 and 2 and 3 TBs in bays 3 and 4.

What I did.
1) Rolled back QTS from 4.3.3.0238 build 20170703 to the previous build, without any noticeable effect (bay 3 and 4 still "unplugged").
2) Further rollback to build 20170606 and bay 3 and 4 now exist again but disk 3 is tagged for abnormal i/o activity.
3) Ran SMART diagnostic on 3. Rapid version worked showed no errors. Long version got stuck at 90% sometime after 3 hours (left it overnight).
4) cancelled SMART scan.
5) ran disk scan looking for bad blocks. That went fine and it cleared the bad i/o activity status.

I'm pretty new to these NAS boxes, so I am really grateful to see your thoughts on this.
I think the RAM issue is a red herring and probably also "disk bay 3". I'll put in my ticket with details of the disks and QTS versions etc.

Thanks
Duane
-----------------------------------------
Model: TS451A 8GB (single Qnap RAM board) Firmware version 4.3.3.0210 Build 20170606 (primarily as fileserver)
Disks: 2 + 2 + 3 + 3 TB as RAID 5 (will likely revert to a safer configuration at some point).
Disk models: (model firmware) 2x (WDC WD20EFRX-68EUZN0 82.00A82), 2x (WDC WD30EFRX-68EUZN0 82.00A82)
UPS: CPS Value800EIGP Network: 1Gb Ethernet from NAS port 1 to FRITZ!Box router.