TS-419P II Drive 2 Failed - REPLACED HDD - HAPPENED AGAIN

[clickright]

Hi All,

Just setup a new TS-419PII Last week in RAID 1. (Firmware 3.6.1 Build 0302T)
Last night I received two emails from the unit at exactly 1:00AM.

Email 1:
IP Address: 192.168.1.4
Date/Time: 2012/06/01 01:00:48
Level: Warning
[Mirror Disk Volume: Drive 1 2] RAID device in degraded mode.

Email 2:
IP Address: 192.168.1.4
Date/Time: 2012/06/01 01:00:55
Level: Error
[Mirror Disk Volume: Drive 1 2] Drive 2 failed.
PS. The unit is not scheduled to run disk checks on this day!

Both drives are WD RE4 2TB WD2003FYYS Enterprise class drives (on the supported list).

I ran a SMART check and it passed. Out of desperation I rebooted the Unit and it is now Rebuilding @ 0%.

Is there anywhere I can check to see why this happened? What is going on -- this is a brand new unit with brand new RE4 drives!?
My old dlink RAID1 NAS with lasted 4 year with consumer drives without a peep.

Also I just received another email from the NAS:
The system was not shut down properly last time. (however I used the Web interface to restart it)


Update: June 1: 5:29PM
- As the drive suggested, I shutdown the unit and exchanged the drive with another brand new drive with my supplier.
I ran a complete SMART test in my PC and it passed, then installed it into the NAS.

Same drive as last time:

Western Digital RE4 WDC WD2003FYYS-02W0B01.0
Firmware Version: 01.01D02

I Installed the drive, and the raid rebuilded.
I ran a file system check: no problem found.

Then I ran a complete test on disk 2 and again the unit reports "Drive 2 Failed"

In status it says "Disk Read/Write Error"

The disk is still testing @ 20% so it has not even failed its SMART test yet.

I will let the test complete and post back with the results. I have attached my event log. Any ideas? Is there a better way I can investigate?

[clickright]

OK, so the new drive ran through the complete SMART test and PASSED. I also ran a bad block scan and it Passed. The bad block scan finished before the SMART scan had completed and the RAID had already begin rebuilding. Almost immediately it said that drive 2 had been unplugged and that there was a read/write error. I had to run the bad block scan again, and the drive showed as "Ready" and the RAID array is rebuilding again. It definitely seems that this issue is related to the complete SMART scan and QNAP and not the drive(s), can someone confirm?

Again these are WD RE4 Enterprise class 2GB drives.

[Scillonian]

Bare in mind when running the SMART tests that they mostly only test the drive internally and not its interface with the outside world. I've experienced this with the infamous Seagate STD2000DL003 drives where they pass the SMART tests without problem but do real work with them and they start randomly failing. It was probably not a good idea to be running the Bad Blocks Scan at the same time as the SMART test. They will both be trying to sequentially read the drive surface at the same.

It is possible that there is a problem with the drive connections in your Drive 2 bay which are causing the errors. As you only talk about having two drives I am assuming that bays 3 and 4 are vacant. Next time Drive 2 fails try removing it and inserting it into one of the empty bays. As an added step I would connect the drive to a PC and delete the partitions on the drive before reinserting it so the NAS is more likely to think it is a drive it has never seen before.

I am assuming that it is possible to have a RAID 1 across not contiguous drives but even if not it should still be possible to test the drive in the new location and get a better idea of whether it is a hardware fault with the NAS or with the drive.

For pre-installation testing of new drives I connect them to a spare PC and use GRC's SpinRite to exercise them. I use this specifically because it is the only utility I know of that will read and write to a raw (i.e. un-partitioned) drive.

[clickright]

Bare in mind when running the SMART tests that they mostly only test the drive internally and not its interface with the outside world. I've experienced this with the infamous Seagate STD2000DL003 drives where they pass the SMART tests without problem but do real work with them and they start randomly failing. It was probably not a good idea to be running the Bad Blocks Scan at the same time as the SMART test. They will both be trying to sequentially read the drive surface at the same.

It is possible that there is a problem with the drive connections in your Drive 2 bay which are causing the errors. As you only talk about having two drives I am assuming that bays 3 and 4 are vacant. Next time Drive 2 fails try removing it and inserting it into one of the empty bays. As an added step I would connect the drive to a PC and delete the partitions on the drive before reinserting it so the NAS is more likely to think it is a drive it has never seen before.

I am assuming that it is possible to have a RAID 1 across not contiguous drives but even if not it should still be possible to test the drive in the new location and get a better idea of whether it is a hardware fault with the NAS or with the drive.

For pre-installation testing of new drives I connect them to a spare PC and use GRC's SpinRite to exercise them. I use this specifically because it is the only utility I know of that will read and write to a raw (i.e. un-partitioned) drive.

Hi Scillonian,

Thank-you for your suggestions, however after doing extensive research in the forums and finding an absolutely unbelievable amount of people with the same problem (Random Drive lost even on a good drive)-- dating as far back as 2009, I have decided to cut my losses and return the unit to my supplier. I will setup my old NAS again this weekend and pick up a different NAS of another brand--this was my first and last QCRAP unit!

[Scillonian]

No problem. From my readings of the forums some models and disk combinations appear less reliable. The users of ARM based models with "green" drives seem to have more problems than users of Intel based models with "normal" 7200 rpm drives.

[clickright]

No problem. From my readings of the forums some models and disk combinations appear less reliable. The users of ARM based models with "green" drives seem to have more problems than users of Intel based models with "normal" 7200 rpm drives.

Definitely! I couldn't agree with you more, that's why I went with WD RE4 RAID drives for my RAID 1 array. Currently running WD Lifeguard tools just to verify that it's not the drive.

[pwilson]

OK, so the new drive ran through the complete SMART test and PASSED. I also ran a bad block scan and it Passed. The bad block scan finished before the SMART scan had completed and the RAID had already begin rebuilding. Almost immediately it said that drive 2 had been unplugged and that there was a read/write error. I had to run the bad block scan again, and the drive showed as "Ready" and the RAID array is rebuilding again. It definitely seems that this issue is related to the complete SMART scan and QNAP and not the drive(s), can someone confirm?

Again these are WD RE4 Enterprise class 2GB drives.

Did you upgrade the firmware on the RE4 Drives? Per: http://files.qnap.com/download/Forum/WD_New_Firmware_Release.pdf as recommended by Note4 on the QNAP HDD Compatibilty List

Note4 (WD RE4-GP WD2002FYPS)
For improved stability, you are suggested to contact Western Digital after Nov 15, 2009 for the new firmware upgrade. Please refer to http://files.qnap.com/download/Forum/WD ... elease.pdf

I can definitely appreciate your frustration. You deliberately avoided their Blue/Black/Green "Consumer" drives, (as recommended by QNAP), and purchased one of their allegedly better "Enterprise" drives, and still get burned. I haven't purchased anything from WDC in over 20 years.

[clickright]

Did you upgrade the firmware on the RE4 Drives? Per: http://files.qnap.com/download/Forum/WD_New_Firmware_Release.pdf as recommended by Note4 on the QNAP HDD Compatibilty List

No, They're RE4 WD2003FYYS Not RE4 GP drives. There is no firmware update available. It is listed on the compatibility list without a note.

Note4 (WD RE4-GP WD2002FYPS)
For improved stability, you are suggested to contact Western Digital after Nov 15, 2009 for the new firmware upgrade. Please refer to http://files.qnap.com/download/Forum/WD ... elease.pdf

I can definitely appreciate your frustration. You deliberately avoided their Blue/Black/Green "Consumer" drives, (as recommended by QNAP), and purchased one of their allegedly better "Enterprise" drives, and still get burned. I haven't purchased anything from WDC in over 20 years.[/quote]

That's funny, I won't purchase anything but WD and Hitachi! Too many issues with the other brands.

[clickright]

Just an update: Ran the WD Data Lifeguard diagnostics (Full) on Disk 2-- It took around 5 hours and passed without any errors. I am going to check disk 1 now and If I have time I will reinstall them and swap them in the NAS to see if I can replicate the issue.

[Scillonian]

Just an update: Ran the WD Data Lifeguard diagnostics (Full) on Disk 2-- It took around 5 hours and passed without any errors. I am going to check disk 1 now and If I have time I will reinstall them and swap them in the NAS to see if I can replicate the issue.

If you do have the time it will be interesting to know the results of this test -- could help find out if this is a drive or a NAS hardware problem.

That's funny, I won't purchase anything but WD and Hitachi! Too many issues with the other brands.

Agree with you here. I've had problems of one form or another with every Seagate drive I've had. The most reliable drives I've had have been Hitachi with WD close behind.

[clickright]

If you do have the time it will be interesting to know the results of this test -- could help find out if this is a drive or a NAS hardware problem.

That's my intention. I'm running WD Data Lifeguard diagnostics on disk 1 now. I have already installed disk 2 in disk 1 place in the QNAP and it's running the complete test through the unit (currently it's at 80%.) I should be able to re-build the array tonight and then I'll have an answer.

[clickright]

Update:

I swapped the drives in the QNAP and rebuilt the array. After starting the complete SMART test on disk 2 (now the original disk 1) it is again showing that Drive 2 has been removed and "Disk Read/Write Error" error shows on the volume management tam. Of course, the array is now degraded. This definitely confirms it is indeed the NAS unit.

Solution: Disabling the SMART check is not an option for me and looking on these forums there are many issues with this QNAP unit and others dropping disks whether running a SMART test or not so replacing the unit is also out of the question. This was a special order item and I hate putting my supplier through this (though I will make sure it never happens again) but I am returning the unit. Thankfully, my Synology DS412+ arrives on Tuesday

[pwilson]

Update:

I swapped the drives in the QNAP and rebuilt the array. After starting the complete SMART test on disk 2 (now the original disk 1) it is again showing that Drive 2 has been removed and "Disk Read/Write Error" error shows on the volume management tam. Of course, the array is now degraded. This definitely confirms it is indeed the NAS unit.

Solution: Disabling the SMART check is not an option for me and looking on these forums there are many issues with this QNAP unit and others dropping disks whether running a SMART test or not so replacing the unit is also out of the question. This was a special order item and I hate putting my supplier through this (though I will make sure it never happens again) but I am returning the unit. Thankfully, my Synology DS412+ arrives on Tuesday

Switching to a different Make/Model of NAS will be unlikely to work for you, if the issue was the drives. Surely you will want to eliminate the drives as the source of your issues before introducing different NAS hardware. If the drives themselves are the issue, you'll end up with two non-operational NAS units, and a collection of drives you can't use.

The symptoms you describe seem to suggest that you are using drives that don't like working in a RAID configuration. The error message: ""Disk Read/Write Error"" would suggest that the "Disk" experienced a "Read/Write Error", which strangely enough suggests issues with the Drives rather than the NAS. Symptoms and Error messages suggest "Drive failure", so I still don't understand your statement:

This definitely confirms it is indeed the NAS unit.

I wish you luck with your migration to Synology, but I suspect you will experience the same issue with the new NAS unit too, as I doubt the NAS hardware is the underlying issue here. Your new NAS will be here soon, so I'm sure you'll get an answer soon enough.

Good luck.

[clickright]

Switching to a different Make/Model of NAS will be unlikely to work for you, if the issue was the drives. Surely you will want to eliminate the drives as the source of your issues before introducing different NAS hardware. If the drives themselves are the issue, you'll end up with two non-operational NAS units, and a collection of drives you can't use.

The symptoms you describe seem to suggest that you are using drives that don't like working in a RAID configuration. The error message: ""Disk Read/Write Error"" would suggest that the "Disk" experienced a "Read/Write Error", which strangely enough suggests issues with the Drives rather than the NAS. Symptoms and Error messages suggest "Drive failure"

Normally I would agree with you, however it is the NAS unit reporting a Read/Write error in Volume management, not the disk itself or SMART -- both disks pass the complete SMART test each and every time & pass WDs diagnostics. I swapped disks and rebuilt the array and the issue did not follow the disks-- it followed the bay -- it was always "Disk 2" even when they were swapped.

I have been dealing with IT for over 10 years and have seen MANY failed drives -- everyone thinks I am overcautious with returning/exchanging drives that make even the weirdest noise or minor SMART errors -- I even went and exchanged the first drive as advised by the QNAP. The result: Same issue reproduceable every time on "Disk" 2.

Like you said I will find out soon. My supplier emailed me this AM -- my Synology will be here at 5:30PM PST (PS. I am in Vancouver, BC)

[sl1000]

Swapping the qnap for warrenty would also have been an option. disabeling power management on the drives might have been too.
Do me a favour, once the synology is in, check the power management settings it uses by default for me. on commandline do a hdparm -iI /dev/sda (small and capital i)
it might be the issue is powersaving related. i'm very interested in what acoustic (or apm) setting they are using.