"Disk Failed", then "Disk Unplugged" errors

[peelos]

So even the one most important (for a warranty claim) tell you the disk have failed big time.

Evidently.. however it is still not clear what caused the drive to fail suddenly without warning.. Hopefully it was just reading posts in this thread about faulty back panels and WD red issues with specific firmware that caused paranoia and the cause was just a faulty drive that didn't survive a reboot.

Why even consider that if it's under warranty? Even if it could be initialised, I wouldn't ever trust any of my data to it again.

Only to see if it is possible to initialise and then run WD tests, anyway - it was not.

If even their own software tell you the disk is dead, why wouldn't it be possible?

I have never tried to claim warranty from WD, and in my experience little in life is certain.. anyway, hopefully all goes smoothly..

In the meantime, 12.5hrs later rebuild is 88% running at 60 MB/s

[peelos]

Rebuild complete and everything seems to be working fine so far.

[FogDucker]

Add me to the list.

TVS-473 running 4.3.4.0486. 4x 3TB WD Reds in RAID5, 2x m.2 SSDs in RAID1 and 24GB of memory. Not even a year old.

Disk #3. Tried replacing it with 2 different brand new 4TB WD Reds.

One month later and I'm back in business.

QNAP support was able to remote in to make my 3 drive system readable, but slot #3 would still not work so I shipped in to the California repair depot who replaced parts. Replacing the slot with the 4TB WD I tried during the failure did not work - it reported "abnormal". Even though I was able to format and use the drive in a USB toaster, it still wouldn't work. Running a 'bad block' scan made the new drive usable and let the rebuild proceed.

Lessons learned? Regularly test backups! I didn't and got lucky. I rsync daily to an old Synology and think I'll use the Amazon Glacier app for a second, offsite backup.

The biggest pain with this whole incident wasn't that I was without data , it was not being able to run my virtual machines. That prevented me from book-keeping for my business and network monitoring for my clients, among other things. I may revisit this solution when it reaches end of life.

[seadevil81]

Hi,
I am glad I have come across this topic as I have been pulling my hair out with our QNAP TVS-463 and the Drive 3 unplugged error.

My company bought a number of TVS-463 QNAPS (40 in total) and 160 Western Digital 4TB Red Pro drives - WD4001FFSX drives.

We have had 8 TVS-463 devices fail (only 6 within the warranty period as they were not deployed until after a year of purchase). The 6 QNAP devices that I sent through to RMA from the vendor we purchased them from did not send them through to QNAP - they gave us a refund. I have a case open for 2 devices with drive 3 unplugged errors which are now out of warranty.

I have performed the following on the two faulty QNAP devices:
Hot swapped the drives in bay 3 and rebooted the devices multiple times.
All QNAP devices were originally running 4.2.2 firmware until I came across the article in regards to rebuild issues on Raid 5 and upgraded them all to 4.3.3.0238
Upgraded to latest firmware 4.3.4.0516 which has not resolved the issue.
Downgraded to the earliest firmware I could find - 4.2.0 which also did not resolve the issue.

I have not changed the RAM as they all came installed with 8GB as shipped in the box.

I am waiting for an update from QNAP but does anyone have anymore ideas or theories apart from the backplane?

I have 2 QNAP devices with the same fault and a whole heap of hard drives (all WD 4TB red Pro drives though).

[MrVideo]

Did you try replacing the drive 3 with a new drive (hot swapping it)?

[seadevil81]

Yes I have replaced the drives - hot swap, rebooted multiple times etc. As before on my previous post, tried different firmware versions etc.

Have verified the drives with Western Digital Utilities and they are working as normal.

It's rather sad to have spent a large amount of money on these devices, 21 are deployed at various sites and you need to drive close to 2 hours to some of the sites from our head office. Adding to the fact the remaining QNAP devices (32) are now out of warranty so any fixes to the backplane will be paid by the company I work for if we can get them repaired at all. Finally it seems like a waste of a perfectly good NAS if they need to be decomissioned due to this problem.

Very interested if anyone can prove or has evidence of problems with the backplane so I can raise this with QNAP in Asia Pacific support.

[FogDucker]

I could not get slot #3 to work at all. I tried 2 brand new WD reds. I tried the old one too. No dice. When slot #3 goes, it goes.

Slot #3 only worked after the backplane was replaced/repaired. It was a major inconvenience for me too as I also had to remove the extra RAM, m.2 SSDs and the disks. It also meant that I was without my virtual machines [Win10 & Linux] which impacted my business.

Having such a significant investment in QNAP hardware, I would hope that they could accommodate you somehow. Perhaps they can provide you with a replacement unit so that you can visit each site, swapping the chassis out so that you can return the defective unit without having to visit each site twice.

Good luck!

[seadevil81]

Hi Marc,
Thanks for the update, luckily we have 11 spare QNAPs at the moment (but that number is dropping as we swap our failing QNAP devices). We have 21 QNAPS out in our remote sites and so far we have not had any data loss to this point.

I will report back once QNAP sends through an update but yes its rather painful. I asked QNAP for any known issues with the batch of the TVS-463 with our listed serial numbers and support responded saying they did not know of any known batch issues. It sounds as though there is a rather big problem with not just the model of QNAPs we have but other 4 and 5 bay QNAP devices.

Thanks.

[seadevil81]

We have had 10 QNAPs accepted for RMA but we are now out of warranty for our remaining 30 QNAPS.

So we have 21 in production and 9 spare, it will be interesting in how many more fail. I tried to escalate the issue to QNAP and get the last 2 faulty devices sent to QNAP themselves rather than the reseller we bought them from but QNAP only allowed us to RMA the devices to the reseller. Looks like they know there is an issue but don't want to take responsibility for it.

[DafOwen]

Am getting something like this.

QNAP TS-853A-4G
Little under a year old ~ 10 months
FW: 4.3.4.0597
Disks : 5 x Toshiba 8Tb N300
(First 4 bought same time as NAS, 5th added later)

Disk 2 failed then unplugged a couple of nights ago.

Removed it, re-inserted - it started to rebuild the array - but it failed (+unplugged) before it completed.

Again - removed it + re-inserted.
Ran rapid SMART test within QNAP interface (while rebuild was happening) - showed Ok.
Again - disk showed as unplugged before the rebuild completed.

Rebooted - disk still showing as unplugged.
Now shut-down.

Have opened a support ticket with QNAP.

[gregort]

Qnap TS-563 2 of these drives Bays 4 and 5 have issues like everyone has talked about it.


Responce from Qnap
Hello again,

I have checked with our RMA team, unfortunately we don't have any part for this model any more, even Taiwan doesn't have the parts anymore.
and I need to inform you we are not able to perform any RMA for your unit since there is no part for this model any more.

They had me change out RAM Drives , check hard drives with WD software in another computer, which runs between 40 to 60 hours and found no issues.

TS-563 still being sold through B&H Photo , Amazon.

I can give more info if you have some contact that could help.

[gregort]

Qnap TS-563 2 of these drives Bays 4 and 5 have issues like everyone has talked about it.


Responce from Qnap
Hello again,

I have checked with our RMA team, unfortunately we don't have any part for this model any more, even Taiwan doesn't have the parts anymore.
and I need to inform you we are not able to perform any RMA for your unit since there is no part for this model any more.

They had me change out RAM Drives , check hard drives with WD software in another computer, which runs between 40 to 60 hours and found no issues.

TS-563 still being sold through B&H Photo , Amazon.

I can give more info if you have some contact that could help.

[DafOwen]

My QNAP response after sending them the Diagnostic Logs

--------------------------------
Disk-2 is very unreliable and its getting unplugged automatically or it shows failed.

We don't find any hardware errors in the logs. We would suggest you to replace the drive and check the rebuild completes ?
--------------------------------

The disk had past the quick Toshiba Diagnostic tool
Running the extended test - it had taken over 24hrs to reach 90% - not having used the tool before I wasn't sure how much longer it would take - so cancelled it.

Instead using Western Digital Data Lifeguard
Quick Test : Pass
Extended Test : Now running.

If it passes - I think I'll format the disk and re-insert it to see how the rebuild goes.
If it fails again - will try to get a warranty replacement.
Not sure what will happen if I try to return a disk with no obvious test fails.....

[P3R]

If it passes - I think I'll format the disk and re-insert it to see how the rebuild goes.
If it fails again - will try to get a warranty replacement.

That sound like good strategy.

But if the WD test completes without error, please be patient enough for the Toshiba test to complete. 8 TB is a very large amount of disk to check and the long tests are exactly that, LONG. If the test have used 24 hours for the first 90 % and then use a relatively much longer time for the remaining 10 %, that would be a sign of a disk problem even if the test finally completes without indicating an error.

This thread exists because of the isolated problem that appear to be present on some batches of specific NAS models. That doesn't mean that all disks that are failing/unplugged from a Qnap are suddenly okay. In most cases that still happens because the disks have actually malfunctioned. Mechanical hard drives are by far the most error prone component in almost all systems.

[DafOwen]

My QNAP response after sending them the Diagnostic Logs

If it passes - I think I'll format the disk and re-insert it to see how the rebuild goes.

This worked for me
Looks like DIsk 2 had got out of synch somehow during the fail but it was trying trying to use it for the Re-synch.