Disk error after Data Scrubbing

Questions about SNMP, Power, System, Logs, disk, & RAID.
Locked
Pistoly
New here
Posts: 8
Joined: Mon Aug 18, 2014 2:10 am
Location: Vancouver B.C. Canada

Re: Disk error after Data Scrubbing

Post by Pistoly »

Hello,

I have recently updated to the latest firmware 4.3.3.0238 on my TS-451. I have four 3tb WD Red hard drives installed in a Raid 5 setup. Last night at midnight the system started a Scrubbing on my Raid Group 1.

I think something is wrong because it has been stuck on 84% for the past 6 hours. I have not tried rebooting or anything at this point other than come to this site to read up first. I have received no errors and the main unit has all four green lights for the hard drives blinking constantly, so it appears to be running but may be stuck or frozen?

All of my hard drives say that they are Good under the storage section with zero bad sectors.

Is there anything that can be done? I didn't want to just reboot without checking here first. Is there a way to manually stop the scrubbing?

Pete

**UPDATE**

I was too impatient to wait so I rebooted manually and now the system is doing a Raid Resync which also is now stuck at 84%. When it did reboot after the scrubbing, I did notice that it was now showing Disk #4 as have 2 Abnormal Sector Count.

I manually rebooted again and this time it is showing 11 Abnormal Sector Count on Disk #4 and it has restarted the Raid Resyncing. Is there any way to stop the process?
riahc3
Getting the hang of things
Posts: 84
Joined: Wed Jun 19, 2013 4:08 pm

Re: Disk error after Data Scrubbing

Post by riahc3 »

User avatar
Moogle Stiltzkin
Guru
Posts: 11445
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: Disk error after Data Scrubbing

Post by Moogle Stiltzkin »

Image


*update

oo it said that got fixed :) I asked expert they said raid scrubbing does not result in that issue linked. Yet neither does it avoid a problem as the result of what was described either. Eitherway that seems a different matter to raid scrub, you can make a different thread for it if you want but it was already fixed, so not much point. even more reason though to upgrade firmware.
QNAP running a version before 4.3.3.0154 20170413 or 4.2.5 20170413 then upgrade it immediately

- Wayne
Last edited by Moogle Stiltzkin on Sat Jul 22, 2017 11:05 am, edited 2 times in total.
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
User avatar
oyvindo
Experience counts
Posts: 1399
Joined: Tue May 19, 2009 2:08 am
Location: Norway, Oslo

Re: Disk error after Data Scrubbing

Post by oyvindo »

Thank you @riahc3 for pointing us to this article. :shock:
It's so sad to see how many users reported the exact same issue; When one drive in a RAID5 fails, it takes a second down with it during rebuild, and that's game over!
It is, after all, a relief to know that my problem had nothing to do with the fact that I used "incompatible" drives. It was a QNAP bug! That we know now.

I'm still a QNAP fan despite some "heavy weather". I know I have to take part of the blame for my own problems. But QNAP's customer service seems to be running into shallow water at high speed. That's never a good sign.
ImageImageImage
User avatar
Don
Guru
Posts: 12289
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: Disk error after Data Scrubbing

Post by Don »

Sorry, I must have missed the part in the article about "When one drive in a RAID5 fails, it takes a second down with it during rebuild, and that's game over!". The article states that when one drives fails you will have data corruption. Nothing about a second drive failing (unless I missed that part).
Use the forum search feature before posting.

Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
P3R
Guru
Posts: 13192
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Disk error after Data Scrubbing

Post by P3R »

oyvindo wrote:Thank you @riahc3 for pointing us to this article. :shock:
It's so sad to see how many users reported the exact same issue...
What!!! :-0
Did you even read the article?

When I read it it tells me that a successful disk rebuild could result in sporadic data corruption on a few data files. Actually the article says that it's normally so few errors that unless we do a big time data integrity check after the rebuild, we normally don't find the errors.

The only similarity I can find with this thread is that you said in here that you don't mind exactly the issues they felt was a huge problem (I by the way agree with them) and that you prefer to have those few data corruptions rather than have the data scrubbing process fail your RAID, that contained more faulty (and incompatible) disks than your choice of RAID level can cope with:
oyvindo wrote:A bad block here and there is of little or no harm either to a film or a song track...
By the way the article also clearly tell us the problems was fixed in April with QTS firmware 4.3.3.0154 while you report using QTS 4.3.3.0210, released in June... :roll:

The issues the article raises are totally different and in my opinion those issues are both interesting and disturbing. Therefore the article would have deserved to be discussed for what it really contains in an own thread in the forum instead of being grocely misinterpreted and posted off-topic here.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
User avatar
oyvindo
Experience counts
Posts: 1399
Joined: Tue May 19, 2009 2:08 am
Location: Norway, Oslo

Re: Disk error after Data Scrubbing

Post by oyvindo »

Well, maybe my imagination is better than my english, but that's how I read this: We had a number devices have drives fail in quick succession –
ImageImageImage
P3R
Guru
Posts: 13192
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Disk error after Data Scrubbing

Post by P3R »

oyvindo wrote:Well, maybe my imagination is better than my english, but that's how I read this: We had a number devices have drives fail in quick succession –
Yes individual disks (drives) fail. You of all should now that by now...
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
P3R
Guru
Posts: 13192
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Disk error after Data Scrubbing

Post by P3R »

Interesting article but unfortunately posted off-topic here.

Anyway, Qnap finally did the right thing and improved their communication.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
Bebopper
Starting out
Posts: 22
Joined: Fri Jul 30, 2010 3:11 am

Re: Disk error after Data Scrubbing

Post by Bebopper »

Anyone here have any idea how to turn off the beeping (at the beginning and the end) of the QNAP when it is performing the raid scrubbing? Checked the Global Settings of the Storage Manager but no luck there...
QNAP TS-451
P3R
Guru
Posts: 13192
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Disk error after Data Scrubbing

Post by P3R »

Bebopper wrote:Anyone here have any idea how to turn off the beeping (at the beginning and the end) of the QNAP when it is performing the raid scrubbing? Checked the Global Settings of the Storage Manager but no luck there...
Have you tried disabling the audio alerts (I would guess it's a System operation but I disable all them)? I don't for know for sure as I curently have no NAS that supports RAID scrubbing but I think it's worth a try.

Diasbling all beeping and instead enabling notifications are among the first few thing I do when installing a new NAS. The diagnostic beeping on boot up is the only one that as far as I know can't be disabled.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
direktorn
First post
Posts: 1
Joined: Sat Sep 02, 2017 5:53 pm

Re: Disk error after Data Scrubbing

Post by direktorn »

After upgraded to 4.3 on my TS-831X with 43 TB of disk I changed the scrubbing to occur once a month and forgot about it. Yesterday at work I realised that I couldn't receive emails anymore and knew I had some issues at home (where my email server sits)

At home I found out, after some troubleshooting that my nas was down on an iSCSI level, my vmware could not access it. However the nas was fine and had a data scrubbing job going. The job where at 13% after running for a whole 10 hours (!!) - After 24 hours it was up to 23% and after about 28 hours completely crashed my NAS (not responding to PING)

After a hard-reboot (Not something you want to do on a 43 TB nas) it started responding to PING requests once again, but died after 1-2 min. and came back after 3-4 min. Managed to login and had 10 critical events showing up and a "unable to mount" dialog box showing 3 out of 4 Volumes/LUNs as unmounted.. My forth Volume is running iSCSI and that volume is empty ... Thanks!!!

At the same time a job "Rebuilding Raid" was running, and will most likely run for a few DAYS leaving my nas unusable.

This could very well be a issue with the TS-831X running crappy Labs Alpine ARM 1.4Ghz CPU, but with 8GB of RAM. I have had various issues since i got it and this is my 4:th QNAP nas (still having my TS-853 Pro as a backup, but only with 16TB)

I'd avoid running Data Scrubbing if config is somewhat identical as the above. If i'm unlucky my NAS will be corrupted.
Locked

Return to “System & Disk Volume Management”