A lot of drives failing

[Wiking]

Hi everyone.
I have a s*it case with a costumer, and a LOT of drives failing.
This is the verry short version.

All NAS are rack mounted models, all drives are NAS drives. Power is verry stable.
There are older servers and QNAP NAS's at the locations, that never had any drive issues.

We have a costumer with 2 locations, both locations have UPS, and it is in Denmark = verry stable power.
The locations are in 2 cornors of Denmark.

2x QNAP 8 bay NAS, all WD Gold drives.
After 6 months, 3 drives failed on one, 4 on the other, data loss and all that exiting stuff.
All drives returned and QNAP's returned.

Then 2 new QNAP's 8 bay (New model number), Half Toshiba NAS drives, other half WD RED Pro.
So, after a couple of months, we had drives failing. But we have had drives on hand everytime, but they kept failing.

Now we are on NAS 5 and 6.
Last monday drive bay 3 had a failed drive, we changed the drive, but the new WD Red drive has failed.

Why are we seeing so many failed drives? We need input!
The next step is to go to Synology or some other storage solution.

[Moogle Stiltzkin]

i'm assuming these failed drives u pulled them out of the qnap and tested them on a desktop pc using the hdd manufacturer diagnostic tool to confirm the hard drives are at fault, and not simply letting it sit in the qnap and assume it's necessarily a qnap issue? Not saying it's not, but during troubleshoot this is the first thing to do, especially with this many failed drives.

i'm assuming these drives are not 2nd hand? meaning they were all factory sealed brand new when you first got them yes?


anyway are u sure the drives are bad, or do you mean they are dropping out of raid? what exactly is your qnap configuration? are you using raid? which one? What QNAP model exactly are you using? (the specific model name, not simply how many bays there are) you left all that info out.....

Also by any chance are you exposing your nas online?

What QTS/QUTS version are you using?


this many drives failing does sound alarming and unusual, but the fact remains not enough info to help troubleshoot. we don't know the condition of those drives, what configuration your qnap is setup as, and whether it's possibly there are other factors at play here.

if i had to guess (you are using raid5/6 ???) and your drives are dropping out of raid (which you then assume is bad drives, when maybe it's not. cause we don't know the condition of these drives or whether you properly tested them to find out before making your declaration).

For all we know could be a bad motherboard so the drives will drop out of raid? thats all i can think of (this is based on previous cases i may have read on this forum here for some past cases for a possible cause) ;_: but there a lot of things you should check first to find out the actual issue causing this.

If qnap helpdesk can't help you, can't you just bring it to some local IT repair and ask them to check it out? cause doesn't sound like you have the expertise to diagnose and troubleshoot the issue. You could try qnap helpdesk first if u want.
https://service.qnap.com/en-us

The next step is to go to Synology or some other storage solution.

no point telling us this. we are other qnap users like yourself. you should probably send an email to sales and threaten them there for hopefully better customer support

[Moogle Stiltzkin]

we changed the drive, but the new WD Red drive has failed.

ooh so u did buy brand new drives to replace, but u r saying those new drives are also still dropping out of raid? (u didn't mention using raid, but i assume thats what u meant when u said drives failing?) i see. which wd red exactly? there were some wd reds that were declared bad for raid because they were in fact SMR. do u know the exact model? WD60EFAX, WD60EFRX or something like that. can u tell us?

https://www.anandtech.com/show/15878/we ... s-hdd-mess

All NAS are rack mounted models, all drives are NAS drives. Power is verry stable.

Another thing you can do is go to qts/quts to check system logs to see if your nas went down unexpectantly and what happened prior to the any incident to help deduce what when wrong. this is another part of the troubleshoot process.


if it's a hardware issue, best ask an IT expert to check it out whether there is possibly a hardware fault at play here

[Wiking]

As i also wrote, this is the short version. We have done a lot of debugging, also with QNAP support.
They basickly said: "We don't know what is wrong, but we will open a case for a RMA"

All drives was pulled, and tested in a other computer, and a USB drive holder and in a other QNAP.
Some drives dident evens pin up, and other drives had bad sector 1 (Something like that). But yes, we tested the first batc

All drives are 100% brand new, directly from our reseller. All drives was exchanged after the frist batch of 16drives started failing.

No open ports to the "wild" Internet.
The drives was as i noted 100% new, there is UPS on both locations.
Temperatures are controlled by AC, so those ar OK.
RAID6 was used on all configs.

I have had some drives drop out of the RAID, then comming online, rebuilding and all god for a week, then the drive dies.
Then new drive, all good, and then fail again, compleatly dead.

It is no threat, i just don't have any good ideas.

All drives are on the positive list.
We had new WD Gold Drives (16 in the start), but now we tried 8 WD102KFBX drives, and some other Toshiba drives just to make sure we warent getting a bad batch.

[Moogle Stiltzkin]

As i also wrote, this is the short version. We have done a lot of debugging, also with QNAP support.
They basickly said: "We don't know what is wrong, but we will open a case for a RMA"

thx for the info. rma is usually a replacement, though if i'm not mistaken, the onus of paying for shipment cost will be on you. you best check those details out with qnap support

All drives was pulled, and tested in a other computer, and a USB drive holder and in a other QNAP.
Some drives didnt even spin up, and other drives had bad sector 1 (Something like that). But yes, we tested the first batc

All drives are 100% brand new, directly from our reseller. All drives was exchanged after the frist batch of 16drives started failing.

yeah test with another pc, and ideally use the hdd manufacturers own hdd diagnostic tool to confirm issue. considering they are brand new, and you using different hdd models and appropriate ones (basically nas hdds) i'm doubting the issue is 16 drives all failing at same time seems rather odd and unlikely how well packaged where these drives when you received them? ones i order myself are usually well packaged heavily in bubble wrap, and i always test them prior to using them on my nas to confirm their good condition prior to use.

No open ports to the "wild" Internet.
The drives was as i noted 100% new, there is UPS on both locations.
Temperatures are controlled by AC, so those ar OK.
RAID6 was used on all configs.

All drives are on the positive list.
We had new WD Gold Drives (16 in the start), but now we tried 8 WD102KFBX drives, and some other Toshiba drives just to make sure we warent getting a bad batch.

good info

I have had some drives drop out of the RAID, then coming online, rebuilding and all god for a week, then the drive dies.
Then new drive, all good, and then fail again, compleatly dead.

i understand about drives dropping out of raid, but what do you mean by completely dying? considering the brand new condition of the drives and how many of them you say were affected, i am more inclined to believe there is some sort of hardware issue (maybe the motherboard) which is causing the drives to drop out of raid.



in summary, all i can think of, is u should have someone check your qnap hardware to see if it's faulty. If the QNAP RMA is not your cup of tea, i guess the other option is to send it to an IT repair shop to get it checked and see if they can find whether maybe the qnap nas hardware is at fault. Cause i'm skeptical so many brand new drives are the issue here, i'm more likely to believe the qnap raid is failing probly hardware hence why raid keeps dropping drives.

A bad drive is usually when you pull out the drive and test on a pc using the HDD manufacturer own diagnostic tool, and it confirms the drive is faulty. That is what i consider a bad drive.

drives dropping from raid, may not necessarily mean the drive is bad. this is what i'm trying to say. so u can double check this



hopefully others can also chime in with their own ideas on this.


by the way what is your qnap nas model? and also what QTS/QUTS version and build are you using on it?

A while back, one of the qts builds was causing random sudden reboots which resulted in raid rebuilds. so in my own situation, the reason for constant raid failure was due to a bad firmware build. So if you want to rule that out, you best check if ur using an outdated firmware, who knows maybe that is the reason? But ur saying ur particular issue, it's the drives dropping out of raid is it? so just a thought just to check ur using a more recent stable firmware just to rule any of that out as a possibility

[PeterT1959]

It makes me think of an environmental issue. Heat / vibration

Sent from my Pixel 4a using Tapatalk

[Wiking]

@PeterT1959
We have 2 locations, 300km apart, with the same bug. Diffrent drives and on multiple QNAP NAS's.
Also we have servers and older QNAP's that dont have issues on the adresses.
There is no heavy traffic near by.

@Moogle Stiltzkin
Thank you for all the great inputs.
The first 2 NAS's was returned, and the costumer burght 2 new, other model NAS's (The original model was not in stock).
And all drives was exchanged.

The drives die, like they dont even spin up!
And others get so weird that the test software dosent work at all!

We tested in a PC, USB connection and a other QNAP. The drives are gone.
I checked the packaging for all drives (Even had pictures of all the orinal drives), and it was fine.
No bumbs on the boxes or things like that.

The drives was from 2 diffrent sellers, and sent to diffrent locations.

Regarding the hardware, we started with 2 QNAP's, and then returned those 2, and got 2 others and 16 new drives.
And when they started having the same issues, those was returned.
TS-873AU-RP, and TS8322PXU-RP are the 2 first models.

[Wiking]

Any good inputs?
I'm going on summer vacation for a couple of weeks, but i would still like some input.

[Wiking]

I'm still not closer to the reason for all the failed drives.
Nothing other, that we have had 6 NAS's with theese enormous bugs.

Any input are still welcome.

[Wiking]

So i just heard from a colleauge, that the "QNAP Destroys 2 drives, out of the blue" is a know issue they have had.

I havent heard about before, and sent me this link:
viewtopic.php?f=25&t=134212

But i just done beleve, that it is the hardware??

[qrusher]

try apply electroisolation wide scotch to place between hdd board and hdd tray frame

check voltage on a case
groud control actions

heat
wibration
esd
dont know what else can make hdd performance and life waste
if all other system components works properly in system