Page 1 of 1

Constant iSCSI disconnects caused by RAID scrubbing

Posted: Mon Sep 28, 2020 4:09 am
by phxazcraig
Title sums it up, this is the detail.
1. TS-451+ (two of them, each set up with a 5TB iSCSI target on interface #2, 1Gb connection.
2. Latest firmware, but also last one as I tried backrevving at one point. Firmware 4.4.3.1421
3. 16GB RAM, each 451 has four HDD's. One has 4 10TB drives, one has 4 4TB drives, both in RAID5
4. Three ESXi 6.7 servers making iSCSI connections to each TS-451+ and using datastores on each QNAP.
5. Dedicated iSCSI network for iSCSI, with two QNAPs and three ESXi connections and nothing else. I bought managed switches for both interfaces for testing to see if there was weird traffic on the subnet, but if anything there was less there than I thought. Nothing that looked like it wasn't normal traffic.
6. Symptom: In a word, dog slow. Anything coming off of those ESXi servers that needed to read some data from disk was very, very slow. For instance, I have a web site on a virtual linux box, and it was 10x slower at least serving up images once you got past the front page. Vcenter (virtual appliance here) very slow to do anything, almost unusable.
7. Event log on each ESXi server was filled with several hundred (over 300) iSCSI disconnect/reconnect messages. Last night they started at 4:45am, and ended at about 9:31am.
8. RAID scrubbing was scheduled to run last night (each Sunday) and start at 4am.
9. I noticed RAID scrubbing still had about an hour to go this morning around 8:30. And around 9:30 the system speed is fast again.

My conclusion is that RAID scrubbing is causing the system to drop iSCSI connections as long as it is running. This may be ok for a home user who can schedule some downtime while things happen, but it's not something you'd have in a business environment. I'm pretty sure it caused some corruption in several VM's last weekend when I had the QNAPs loaded down with a few more apps running at the same time.

What can I do to lessen the impact of RAID scrubbing WHILE IT RUNS?

Re: Constant isici disconnects caused by RAID scrubbing

Posted: Mon Sep 28, 2020 4:12 am
by phxazcraig
Argh, just to improve search engine results on this, let me spell the title correctly: Constant iSCSI disconnects caused by RAID scrubbing.

Re: Constant iSCSI disconnects caused by RAID scrubbing

Posted: Tue Apr 06, 2021 12:24 am
by dgreenwood
Really disappointed that no one followed up on your question, as I'm having the same issue. Scrubbing or syncing basically takes the QNAP offline and VMs that have an iSCSI connection to the device lock up until the process completes (about 27 hours in our case). Since I've found no way to disable or schedule the sync process, it takes down our backups on average at least once a month.

I have turned off RAID scrubbing, so that hasn't been an issue recently, but the syncing causes the same issue. It's not clear to me if this is expected behavior or if it's a problem that can be fixed.

Re: Constant iSCSI disconnects caused by RAID scrubbing

Posted: Tue Apr 06, 2021 12:56 am
by dolbyman
well if nobody knows an answer..what is supposed to be the follow up?

qnap doesn't come here..so if you want their take ..open a ticket