URE during RAID5 rebuilt

Questions about SNMP, Power, System, Logs, disk, & RAID.
Post Reply
janwer
Getting the hang of things
Posts: 72
Joined: Wed Dec 26, 2018 10:16 pm

URE during RAID5 rebuilt

Post by janwer » Mon Sep 16, 2019 5:02 am

What if I'll have URE during RAID5 rebuilt? 5x3TB HGST with 10^15 URE rate. What with 10^14 and 5x8TB?

How will QNAP behave? Stop rebulild like would some lazy programmers make it or continue and give nice report which files are corrupted?
Few URE in files area with degraded RAID5 is no problem in accessing the rest of data, right? I can still use NAS without rebuilding?
How many errors disc like HGST 3TB can tolerate and where in SMART should I find information?

Is it: Retired_Block_Count or Reallocated_Event_Count or Unrecorrectable_Sector_Count?

Why in QNAP I can't find: Realocated sector count, Reported uncorrect, UDMA CRC error when the same HDD installed in PC has this information readable?
TVS-951x 16GB RAM, 2x0,5 TB SSD RAID1, 3x8 TB RAID5
TS-431P2 8GB RAM, 4x 3 TB HGST RAID5
Unraid 8GB RAM, 10x 3TB HGST

dolbyman
Guru
Posts: 15201
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: URE during RAID5 rebuilt

Post by dolbyman » Mon Sep 16, 2019 5:16 am

the rebuild would fail on read error ..nothing qnap programs..just how linux md raid works .. qnal didn't invent anything here

I run Raid6 and swap my disks on early failure signs (relocated blocks etc)

janwer
Getting the hang of things
Posts: 72
Joined: Wed Dec 26, 2018 10:16 pm

Re: URE during RAID5 rebuilt

Post by janwer » Mon Sep 16, 2019 5:34 am

One error is enough to stop rebuild? Why?
I thought that was with hardware RAID and software can be smarter.
TVS-951x 16GB RAM, 2x0,5 TB SSD RAID1, 3x8 TB RAID5
TS-431P2 8GB RAM, 4x 3 TB HGST RAID5
Unraid 8GB RAM, 10x 3TB HGST

User avatar
Moogle Stiltzkin
Ask me anything
Posts: 7850
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: URE during RAID5 rebuilt

Post by Moogle Stiltzkin » Mon Sep 16, 2019 5:42 am

qts will just report that raid rebuild failed. full stop.

i've had many raid rebuilds over the years without issue. I think i did have a single raid rebuild that failed, but it happened ages ago that i'm not really sure tbh :S but raid is still not a backup, because there is a chance it can still fail.

If you are curious how URE fits into raid rebuild thats fine and all. But if you begin to think of it as your safety net, just don't.....

always have a backup. raid is not an excuse to not have one :X Unless of course your data is not important to you so the recovery is of least importance, then that would be understandable.


that said, i found some interesting articles for the subject you mentioned here
https://superuser.com/questions/700177/ ... 5-unusable

https://serverfault.com/questions/93754 ... -data-lost

https://standalone-sysadmin.com/recalcu ... 076bb31a37

*article posted 2016

HOW DOES RAID 5 REDUCE DATA LOSS?
RAID 5 takes your data and adds some parity data that makes it possible to reconstruct the original data if there is a drive failure (RAID 6 is similar, except it can reconstruct after two failures). So why would it stop working?

THE URE PROBLEM

RAID 5 works fine when there are no further failures or errors during data reconstruction. Back in 2007 though, almost all SATA drives, and many SCSI drives, were spec'd with one Unrecoverable Read Error (URE) at 10^14. That's one URE every 12.5TB.

One terabyte drives were coming into production then. If you had an 8 drive RAID 5 stripe, and one drive failed, the RAID controller would have to read 7TB of data to reconstruct the failed drive.

That meant a better than 50 percent chance that during the reconstruction a URE would scuttle the entire process. When that happens it would have been faster to use a backup to rebuild the data.

Of course, drives have only gotten bigger. Four terabyte drives are common and we now have 10TB drives.

WHY DOES RAID 5 STILL WORK?
Simple: drive vendors up'd the spec - for some drives - to one URE in 10^15 bits, or about 125TB. Of course, now that drive capacities have also increased by 10x, the problem of failure due to a URE during reconstruction is coming back.

Seagate's Barracuda Pro and WD's Gold datacenter drives are spec'd to less than 1 URE in 10^15 bits read.

However, many other large capacity drives aren't at the higher spec. If you use a low-spec drive in a RAID, there's a good chance the rebuild won't work.

It pays to look at spec sheets if you have critical applications or data. Or you can do what I do.

THE STORAGE BITS TAKE
I have a couple of 4 drive RAID 5 arrays. I don't worry about the URE problem because I have all the critical data backed up to the cloud.

In case of a drive failure your first action should be to copy all data from the array before replacing the failed drive. If you encounter a URE during copying, at least you've saved all the other data. Not all low-cost RAID controllers report read errors, so you might copy a corrupted file, but that would have happened anyway.

This reiterates the core premise of RAID: it provides data access after drive failures and is NOT a substitute for backup. Fortunately, hard drives are getting more reliable, so your chance of needing this advice is declining.

But as drive capacities continue to rise, vendors need to raise their URE spec. When will they do it?
https://www.zdnet.com/article/why-raid- ... s-usually/



these days i always recommend NAS hdds like either the wd red or seagate ironwolf which are made for NAS and expected to raid rebuild, operate 24/7 and be able to operate in close proximity with a bunch of other hdds in the same enclosure.
Last edited by Moogle Stiltzkin on Mon Sep 16, 2019 5:53 am, edited 8 times in total.
NAS
[Main Server] QNAP TS-877 w. 4tb [ 3x HGST Deskstar NAS (HDN724040ALE640) & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 + 16gb ddr4 Crucial + QWA-AC2600 wireless adapter.
[Backup] QNAP TS-653A w. 5x 2TB Samsung F3 (HD203WI) EXT4 Raid5
[^] QNAP TS-659 Pro II 1x 4TB HGST Deskstar NAS
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-228 w. 1x 1TB WD RE3 (WD1002FBYS)
[^] QNAP TS-128
Mobile NAS TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Asus AC68U Router|100dl/50ul MBPS FTTH Internet | Windows 10, WC PC-Intel i7 920 Ivy bridge desktop (1x 512gb Samsung 850 Pro SSD + 1x 4tb HGST Ultrastar 7K4000)


Guides & articles
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review

https://www.patreon.com/mooglestiltzkin

janwer
Getting the hang of things
Posts: 72
Joined: Wed Dec 26, 2018 10:16 pm

Re: URE during RAID5 rebuilt

Post by janwer » Mon Sep 16, 2019 5:43 am

I'm curious about how technically advanced is QNAP. Not too much...

Why not mark bad sector as bad and move on?
TVS-951x 16GB RAM, 2x0,5 TB SSD RAID1, 3x8 TB RAID5
TS-431P2 8GB RAM, 4x 3 TB HGST RAID5
Unraid 8GB RAM, 10x 3TB HGST

dolbyman
Guru
Posts: 15201
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: URE during RAID5 rebuilt

Post by dolbyman » Mon Sep 16, 2019 6:06 am

qnap uses software raid ...not sure why this is supposed to be bad

cpu's are much faster now than a couple of decades ago where you needed specialized instruction processors for parity calcs

Post Reply

Return to “System & Disk Volume Management”