"Disk Failed", then "Disk Unplugged" errors

Questions about SNMP, Power, System, Logs, disk, & RAID.
Post Reply
Sturmie
New here
Posts: 7
Joined: Sat Apr 29, 2017 10:05 pm

"Disk Failed", then "Disk Unplugged" errors

Post by Sturmie » Sat Jul 15, 2017 1:32 am

I purchased my TS-563 back in April this year. Last night, I updated to the firmware from 4.3.3.0210(20170606) to 4.3.3.0238(2017073) and then got a "Host: Disk 3 failed" followed by "Host: Disk 3 unplugged" and then "RAID Group 1 is in degraded mode" (obviously), but after a shutdown/pull drive, put back in/reboot, all disks showed up fine and the RAID group started to rebuild. It got all the way to ~60% and then it happened again this morning...same set of errors and the disk 3 is showing as "not installed" now (I'm at the office and can't pull the drive and re-insert it like I did last night).

Is this a failing drive (these are 8TB Red NAS drives from WD and are less than 3 months old each) or is this a problem with the backplane in my QNAP? It's not unheard of for drives to go bad this quickly, but it's definitely unlikely. I'm fine getting another 8TB Red drive locally and keeping the WD warranty replaced drive as a cold spare, but before I drop $280 on a drive, I want to make sure it's not the QNAP that's the issue.

Thanks.

dolbyman
Guru
Posts: 14820
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: "Disk Failed", then "Disk Unplugged" errors

Post by dolbyman » Sat Jul 15, 2017 1:40 am

is there any SMART attributes on the disk that look out of the ordinary ?

Sturmie
New here
Posts: 7
Joined: Sat Apr 29, 2017 10:05 pm

Re: "Disk Failed", then "Disk Unplugged" errors

Post by Sturmie » Sat Jul 15, 2017 1:49 am

dolbyman wrote:is there any SMART attributes on the disk that look out of the ordinary ?


Since it says it's "unplugged" in the QNAP interface, I can't get any info on it. This is my first QNAP NAS (my previous experience was with Thecus), but why would the drive "unplug" as opposed to simply removing itself from the RAID? I would think that since the drive is still in the QNAP, I should be able to access it even though it's failed and not part of the RAID group.

hscorpio15201
New here
Posts: 6
Joined: Sat Jul 15, 2017 10:39 am

Re: "Disk Failed", then "Disk Unplugged" errors

Post by hscorpio15201 » Sat Jul 15, 2017 10:49 am

Sturmie wrote:I purchased my TS-563 back in April this year. Last night, I updated to the firmware from 4.3.3.0210(20170606) to 4.3.3.0238(2017073) and then got a "Host: Disk 3 failed" followed by "Host: Disk 3 unplugged" and then "RAID Group 1 is in degraded mode" (obviously), but after a shutdown/pull drive, put back in/reboot, all disks showed up fine and the RAID group started to rebuild. It got all the way to ~60% and then it happened again this morning...same set of errors and the disk 3 is showing as "not installed" now (I'm at the office and can't pull the drive and re-insert it like I did last night).

Is this a failing drive (these are 8TB Red NAS drives from WD and are less than 3 months old each) or is this a problem with the backplane in my QNAP? It's not unheard of for drives to go bad this quickly, but it's definitely unlikely. I'm fine getting another 8TB Red drive locally and keeping the WD warranty replaced drive as a cold spare, but before I drop $280 on a drive, I want to make sure it's not the QNAP that's the issue.

Thanks.


I'm glad this was posted. After the last update, I'm having the same exact issue. I have a very similar setup using the TS-653A with 6 x 8 TB WD Red Drives. Basically it starts with drive 3 falling out of the array and then causes drives 4, 5, 6 to follow. I rebooted and it rebuilt fine but after about 12 hours, it occurred again. I opened a support ticket but havent heard back yet. Not hijacking your thread but just wanted to share the similar issue.

Sturmie
New here
Posts: 7
Joined: Sat Apr 29, 2017 10:05 pm

Re: "Disk Failed", then "Disk Unplugged" errors

Post by Sturmie » Sat Jul 15, 2017 10:10 pm

Oh man, that's not great to hear that almost the exact same scenario is happening to you after the last update. Did you lose any data? For me, only drive 3 dropped out, so *knock on wood* all of my data was still safe.

I actually went and purchased another 8TB Red locally after work (luckily, they pricematched to Newegg) and the array has been rebuilding since last night - currently at 86%. I too opened a ticket and they recommended trying another drive first, which I was obviously doing already. I'll keep you posted.

hscorpio15201
New here
Posts: 6
Joined: Sat Jul 15, 2017 10:39 am

Re: "Disk Failed", then "Disk Unplugged" errors

Post by hscorpio15201 » Sat Jul 15, 2017 11:56 pm

Luckily enough I havent lost any data. Its like it forgets that drive 3 exists and then kicks 4 , 5, 6 out. I rebuilt it twice and the data was fine. I'm curious to what support comes back with before I make a call on drive replacement.

Sturmie
New here
Posts: 7
Joined: Sat Apr 29, 2017 10:05 pm

Re: "Disk Failed", then "Disk Unplugged" errors

Post by Sturmie » Sun Jul 16, 2017 11:29 pm

Well, so far so good. The rebuild finished and the scrubbing process is at 77%. It's looking like it actually was probably the drive itself.

hscorpio15201
New here
Posts: 6
Joined: Sat Jul 15, 2017 10:39 am

Re: "Disk Failed", then "Disk Unplugged" errors

Post by hscorpio15201 » Mon Jul 17, 2017 4:31 am

Sturmie wrote:Well, so far so good. The rebuild finished and the scrubbing process is at 77%. It's looking like it actually was probably the drive itself.


Thats great to hear! Since I'm impatient and have a backup, I took a stab at reloading the 6/24 firmware and that went well. Havent had the drive drop issue and was able to make it through multiple file system tests / disk check of number 3. I'm going to monitor but making it a full 24 hours without the issue is a big step. I'll keep the ticket open since I'm curious

User avatar
ServalCat
New here
Posts: 3
Joined: Mon Jul 17, 2017 5:14 pm

Re: "Disk Failed", then "Disk Unplugged" errors

Post by ServalCat » Mon Jul 17, 2017 5:35 pm

Hi all. I too have had the same issue. After upgrading QTS disk 3 and 4 dropped. However, the issue may be more complex than just QTS upgrade.
I lost contact with Bay 3 and 4 (brand new 3 GB WD red drives), reinserted them and it was ok for a while. Then went on holiday for a week (shut down NAS first), and couldn't get them back after starting up. I've just reverted a couple of QTS firmwares (from 4.3.3.0238 build 20170703 back to build 20170606) and the disks reappear (but vol 3 has been tagged with abnormal i/o status).

However, I found that people also have had identical sounding problems with the same model as mine QNAP 451A when upgrading RAM. I did this recently also from 2 GB to 8GB. I even bought the expensive Qnap RAM rather than generic laptop brand to avoid problems.

There is more about this here
viewtopic.php?f=182&t=125803

Not sure if any of you folks have upgraded RAM, but it might also influence things. The solution, sadly, was to downgrade RAM back to 2 GB, then no problems.

P3R
Guru
Posts: 10981
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: "Disk Failed", then "Disk Unplugged" errors

Post by P3R » Mon Jul 17, 2017 8:14 pm

This isn't something this community can solve. Instead please submit a ticket with Qnap support.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

User avatar
MrVideo
Experience counts
Posts: 4654
Joined: Fri May 03, 2013 2:26 pm

Re: "Disk Failed", then "Disk Unplugged" errors

Post by MrVideo » Tue Jul 18, 2017 10:51 am

ServalCat wrote:Not sure if any of you folks have upgraded RAM, but it might also influence things. The solution, sadly, was to downgrade RAM back to 2 GB, then no problems.

Interesting. I have not upgraded my RAM and no issues with drive 3. I do not have a drive 4. That might also make a difference.
QTS 4.1.n/4.2.n/4.3.n MANUAL
Submit QNAP Support Ticket - QNAP Tutorials, FAQs, Downloads, Wiki - Product Support Status - Moogle's QNAP FAQ help V2
When you ask a question, please include the following
(Thanks to Toxic17 for the links)
QNAP md_checker nasreport (release 20180525)
===============================
Model: TS-869L -- RAM: 3G -- FW: QTS 4.1.4 Build 20150522 (used for data storage)
WD60EFRX-68L0BN1(x1)/68MYMN1(x7) Red HDDs -- RAID6: 8x6TB -- Cold spare: 1x6TB
Entware
===============================
Model: TS-451A -- RAM: 2G -- FW: QTS 4.2.3 Build 20170213 (used as a video server)
WL3000GSA6472(x3) White label NAS HDDs -- RAID5: 3x3TB
Entware -- MyKodi 17.3 (default is Kodi 16)
===============================
My 2017 Total Solar Eclipse Photos

P3R
Guru
Posts: 10981
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: "Disk Failed", then "Disk Unplugged" errors

Post by P3R » Tue Jul 18, 2017 6:32 pm

ServalCat wrote:The solution, sadly, was to downgrade RAM back to 2 GB, then no problems.
That's not a solution. It can at best be considered a work around to avoid a hardware fault or a huge software bug that in my opinion is unacceptable. If you even have Qnap RAM, you have a completely supported configuration that simply should work. I very much recommend that you open a case with Qnap support as I suggested previously.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

hscorpio15201
New here
Posts: 6
Joined: Sat Jul 15, 2017 10:39 am

Re: "Disk Failed", then "Disk Unplugged" errors

Post by hscorpio15201 » Tue Jul 18, 2017 9:21 pm

Yea I dont believe the ram was the issue too. I upgraded my memory too but the issue subsided when I reverted the firmware prior to the 7/6 update. I have a ticket open to see what they say but since I reverted, the NAS has been rock solid and all my disk passed their checks.

User avatar
Trexx
Experience counts
Posts: 4933
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota
Contact:

Re: "Disk Failed", then "Disk Unplugged" errors

Post by Trexx » Tue Jul 18, 2017 9:34 pm

From just reading the thread, the common denominator appears to be WD Red drives (varying capacities). Again as was recommended earlier, make sure you all have Helpdesk tickets open with QNAP. It could be a compatibility issue of some kind related to WD firmware.
Paul

Model: TS-877-1600 FW: 4.4.1.x
QTier (HDD): [RAID-5] 6 x 3TB HGST DeskStar NAS QTier (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
(SSD): [RAID-1] 2 x 500GB Evo 860
RAM: Kingston HyperX Fury 64GB DDR4-2666
GPU: EVGA GTX 1060, ACX 2.0(1 Fan), 6GB
UPS: CyberPower AVR1350 Ext. Backup: USB 3.0 Seagate 5TB
Media Boxes: Nvidia ShieldTV Pro, AppleTV 4k, Various Roku's

Model: TVS-673 32GB FW: 4.4.1.x Test/Backup Box
-----------------------------------------------------------------------------------------------------------------------------------------
NAS RAID Rebuild Times | Live QTS Videos | | QNAP NAS Guide | Information needed when you ask for HELP | QNAP Links, Tutorials, etc.
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq

User avatar
ServalCat
New here
Posts: 3
Joined: Mon Jul 17, 2017 5:14 pm

Re: "Disk Failed", then "Disk Unplugged" errors

Post by ServalCat » Wed Jul 19, 2017 4:06 am

Yep. I agree with that all. And learned that I should put quotes around the word "solution". ;-D
My TS451A is back to normal now and has survived for 24 h post error checking without the disk going down again. Like you all, I really want to work out if there is a common denominator to make sure my help ticket does not just get brushed off with the usual "factory reset" approach.
Yes, all my disks are WD reds 2TB in bays 1 and 2 and 3 TBs in bays 3 and 4.

What I did.
1) Rolled back QTS from 4.3.3.0238 build 20170703 to the previous build, without any noticeable effect (bay 3 and 4 still "unplugged").
2) Further rollback to build 20170606 and bay 3 and 4 now exist again but disk 3 is tagged for abnormal i/o activity.
3) Ran SMART diagnostic on 3. Rapid version worked showed no errors. Long version got stuck at 90% sometime after 3 hours (left it overnight).
4) cancelled SMART scan.
5) ran disk scan looking for bad blocks. That went fine and it cleared the bad i/o activity status.

I'm pretty new to these NAS boxes, so I am really grateful to see your thoughts on this.
I think the RAM issue is a red herring and probably also "disk bay 3". I'll put in my ticket with details of the disks and QTS versions etc.

Thanks
Duane
-----------------------------------------
Model: TS451A 8GB (single Qnap RAM board) Firmware version 4.3.3.0210 Build 20170606 (primarily as fileserver)
Disks: 2 + 2 + 3 + 3 TB as RAID 5 (will likely revert to a safer configuration at some point).
Disk models: (model firmware) 2x (WDC WD20EFRX-68EUZN0 82.00A82), 2x (WDC WD30EFRX-68EUZN0 82.00A82)
UPS: CPS Value800EIGP Network: 1Gb Ethernet from NAS port 1 to FRITZ!Box router.

Post Reply

Return to “System & Disk Volume Management”