ES1640DC v2 Scrub Massive Performance Hit

Post Reply
smccloud
Starting out
Posts: 25
Joined: Sat Aug 24, 2019 2:37 am

ES1640DC v2 Scrub Massive Performance Hit

Post by smccloud » Fri Jul 02, 2021 8:28 pm

When the ES1640DC v2 I'm using as storage for our production vSphere hosts (16 drives in the equivalent of a RAID 10 array) does a scrub, the performance hit is so severe that all the VMs sitting on it go offline. I've thought about reconfiguring it to be two RAIDZ2 pools in a mirror (os RAID 60) but I don't know if that will help at all. Is a massive performance hit a known issue with data scrubs?

koshiro
New here
Posts: 4
Joined: Mon Feb 08, 2021 2:27 pm

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by koshiro » Fri Jul 02, 2021 8:42 pm

Happens with resilvering too, and it has been reported few months ago by me, and they still didn’t fix it. I never buy QNAP QES again.
It means once your HDD crash and you have to replace it and rebuild your raid volume, you can’t serve anything for few days because resilvering IO takes 100% of the IO and changing ratio is just getting ignored.
Super lame thanks QNAP!

smccloud
Starting out
Posts: 25
Joined: Sat Aug 24, 2019 2:37 am

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by smccloud » Fri Jul 02, 2021 8:45 pm

koshiro wrote:
Fri Jul 02, 2021 8:42 pm
Happens with resilvering too, and it has been reported few months ago by me, and they still didn’t fix it. I never buy QNAP QES again.
It means once your HDD crash and you have to replace it and rebuild your raid volume, you can’t serve anything for few days because resilvering IO takes 100% of the IO and changing ratio is just getting ignored.
Super lame thanks QNAP!
Well **. For now, I will just disable monthly scrubs. Nice how they modified the default behavior for both a scrub and resilver......

koshiro
New here
Posts: 4
Joined: Mon Feb 08, 2021 2:27 pm

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by koshiro » Fri Jul 02, 2021 9:00 pm

I think it’s a bug because support tried to change the ratio but nothing happened. But that’s almost half year ago and they still didn’t fix it. And I’m sure they will make a new bug in the fixed firmware in the future.

smccloud
Starting out
Posts: 25
Joined: Sat Aug 24, 2019 2:37 am

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by smccloud » Fri Jul 02, 2021 10:02 pm

koshiro wrote:
Fri Jul 02, 2021 9:00 pm
I think it’s a bug because support tried to change the ratio but nothing happened. But that’s almost half year ago and they still didn’t fix it. And I’m sure they will make a new bug in the fixed firmware in the future.
Do you have a link to the bug for it?

koshiro
New here
Posts: 4
Joined: Mon Feb 08, 2021 2:27 pm

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by koshiro » Sat Jul 03, 2021 6:58 pm

smccloud wrote:
Fri Jul 02, 2021 10:02 pm
koshiro wrote:
Fri Jul 02, 2021 9:00 pm
I think it’s a bug because support tried to change the ratio but nothing happened. But that’s almost half year ago and they still didn’t fix it. And I’m sure they will make a new bug in the fixed firmware in the future.
Do you have a link to the bug for it?
I think it’s not public.

smccloud
Starting out
Posts: 25
Joined: Sat Aug 24, 2019 2:37 am

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by smccloud » Wed Aug 04, 2021 2:33 am

My ticket number is Q-202107-47579.

Latest from support "It looks like your issue is confirmed as related to the other known Scrubbing issue we have, and the team is still working on a solution for it. I'm still trying to find out more information, like an ETA for it, so I'll update you when I hear more."

Storever
New here
Posts: 8
Joined: Fri Mar 15, 2019 9:22 pm

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by Storever » Fri Sep 03, 2021 8:12 pm

Facing the same issue on the ES1686dc. During a scrub, performance gets so poor that VMs are unable to use their disks residing on VMFS datastores connected on the QNAP via iSCSI. Support has suggested that we run the scrub monthly - which clearly indicates to me they have no understanding of how a virtualized environment utilizes storage. As though it would be acceptable to have VMs crash once a month!

They also offered to put in a "feature request" to be able to control the rate of scrubbing and thereby the amount of performance degradation - as though using this device for one of it's main intended purposes is an afterthought, and a 'feature' they'll consider adding.

Storever
New here
Posts: 8
Joined: Fri Mar 15, 2019 9:22 pm

Re: ES1640DC v2 Scrub Massive Performance Hit

Post by Storever » Wed Oct 20, 2021 4:10 am

Looks like in the 9/16 update to QES they added throttling for RAID rebuild related performance. Still no word on the scrubbing issue yet.

Post Reply

Return to “QES Operating System (QNAP Enterprise Storage OS)”