ES1640DC v2 Scrub Massive Performance Hit

Post Reply
smccloud
Starting out
Posts: 26
Joined: Sat Aug 24, 2019 2:37 am

ES1640DC v2 Scrub Massive Performance Hit

Post by smccloud »

When the ES1640DC v2 I'm using as storage for our production vSphere hosts (16 drives in the equivalent of a RAID 10 array) does a scrub, the performance hit is so severe that all the VMs sitting on it go offline. I've thought about reconfiguring it to be two RAIDZ2 pools in a mirror (os RAID 60) but I don't know if that will help at all. Is a massive performance hit a known issue with data scrubs?
koshiro
New here
Posts: 4
Joined: Mon Feb 08, 2021 2:27 pm

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by koshiro »

Happens with resilvering too, and it has been reported few months ago by me, and they still didn’t fix it. I never buy QNAP QES again.
It means once your HDD crash and you have to replace it and rebuild your raid volume, you can’t serve anything for few days because resilvering IO takes 100% of the IO and changing ratio is just getting ignored.
Super lame thanks QNAP!
smccloud
Starting out
Posts: 26
Joined: Sat Aug 24, 2019 2:37 am

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by smccloud »

koshiro wrote: Fri Jul 02, 2021 8:42 pm Happens with resilvering too, and it has been reported few months ago by me, and they still didn’t fix it. I never buy QNAP QES again.
It means once your HDD crash and you have to replace it and rebuild your raid volume, you can’t serve anything for few days because resilvering IO takes 100% of the IO and changing ratio is just getting ignored.
Super lame thanks QNAP!
Well crap. For now, I will just disable monthly scrubs. Nice how they modified the default behavior for both a scrub and resilver......
koshiro
New here
Posts: 4
Joined: Mon Feb 08, 2021 2:27 pm

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by koshiro »

I think it’s a bug because support tried to change the ratio but nothing happened. But that’s almost half year ago and they still didn’t fix it. And I’m sure they will make a new bug in the fixed firmware in the future.
smccloud
Starting out
Posts: 26
Joined: Sat Aug 24, 2019 2:37 am

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by smccloud »

koshiro wrote: Fri Jul 02, 2021 9:00 pm I think it’s a bug because support tried to change the ratio but nothing happened. But that’s almost half year ago and they still didn’t fix it. And I’m sure they will make a new bug in the fixed firmware in the future.
Do you have a link to the bug for it?
koshiro
New here
Posts: 4
Joined: Mon Feb 08, 2021 2:27 pm

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by koshiro »

smccloud wrote: Fri Jul 02, 2021 10:02 pm
koshiro wrote: Fri Jul 02, 2021 9:00 pm I think it’s a bug because support tried to change the ratio but nothing happened. But that’s almost half year ago and they still didn’t fix it. And I’m sure they will make a new bug in the fixed firmware in the future.
Do you have a link to the bug for it?
I think it’s not public.
smccloud
Starting out
Posts: 26
Joined: Sat Aug 24, 2019 2:37 am

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by smccloud »

My ticket number is Q-202107-47579.

Latest from support "It looks like your issue is confirmed as related to the other known Scrubbing issue we have, and the team is still working on a solution for it. I'm still trying to find out more information, like an ETA for it, so I'll update you when I hear more."
Storever
Starting out
Posts: 15
Joined: Fri Mar 15, 2019 9:22 pm

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by Storever »

Facing the same issue on the ES1686dc. During a scrub, performance gets so poor that VMs are unable to use their disks residing on VMFS datastores connected on the QNAP via iSCSI. Support has suggested that we run the scrub monthly - which clearly indicates to me they have no understanding of how a virtualized environment utilizes storage. As though it would be acceptable to have VMs crash once a month!

They also offered to put in a "feature request" to be able to control the rate of scrubbing and thereby the amount of performance degradation - as though using this device for one of it's main intended purposes is an afterthought, and a 'feature' they'll consider adding.
Storever
Starting out
Posts: 15
Joined: Fri Mar 15, 2019 9:22 pm

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by Storever »

Looks like in the 9/16 update to QES they added throttling for RAID rebuild related performance. Still no word on the scrubbing issue yet.
P3R
Guru
Posts: 13183
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by P3R »

Storever wrote: Wed Oct 20, 2021 4:10 am Looks like in the 9/16 update to QES they added throttling for RAID rebuild related performance. Still no word on the scrubbing issue yet.
Doesn't it affect scrubbing as well even if not specifically spelled out? It does in QTS and I wouldn't have expected it to be different with ZFS.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
Storever
Starting out
Posts: 15
Joined: Fri Mar 15, 2019 9:22 pm

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by Storever »

Doesn't it affect scrubbing as well even if not specifically spelled out?
Who knows? Support hasn't updated my ticket on the issue in two months! And how could I tell aside from testing it by waiting to see if my VMs fail?
P3R
Guru
Posts: 13183
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by P3R »

Storever wrote: Thu Dec 09, 2021 11:03 pm Who knows? Support hasn't updated my ticket on the issue in two months!
That's really bad!
And how could I tell aside from testing it by waiting to see if my VMs fail?
I guess you can't. My comment was meant as an encouragement that I would expect it to affect scrubbing as well but I'm afraid that I don't know. :cry:
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
Post Reply

Return to “QES Operating System (QNAP Enterprise Storage OS)”