Constant iSCSI disconnects caused by RAID scrubbing

iSCSI related applications
Post Reply
phxazcraig
New here
Posts: 8
Joined: Sat Sep 12, 2020 6:09 am

Constant iSCSI disconnects caused by RAID scrubbing

Post by phxazcraig » Mon Sep 28, 2020 4:09 am

Title sums it up, this is the detail.
1. TS-451+ (two of them, each set up with a 5TB iSCSI target on interface #2, 1Gb connection.
2. Latest firmware, but also last one as I tried backrevving at one point. Firmware 4.4.3.1421
3. 16GB RAM, each 451 has four HDD's. One has 4 10TB drives, one has 4 4TB drives, both in RAID5
4. Three ESXi 6.7 servers making iSCSI connections to each TS-451+ and using datastores on each QNAP.
5. Dedicated iSCSI network for iSCSI, with two QNAPs and three ESXi connections and nothing else. I bought managed switches for both interfaces for testing to see if there was weird traffic on the subnet, but if anything there was less there than I thought. Nothing that looked like it wasn't normal traffic.
6. Symptom: In a word, dog slow. Anything coming off of those ESXi servers that needed to read some data from disk was very, very slow. For instance, I have a web site on a virtual linux box, and it was 10x slower at least serving up images once you got past the front page. Vcenter (virtual appliance here) very slow to do anything, almost unusable.
7. Event log on each ESXi server was filled with several hundred (over 300) iSCSI disconnect/reconnect messages. Last night they started at 4:45am, and ended at about 9:31am.
8. RAID scrubbing was scheduled to run last night (each Sunday) and start at 4am.
9. I noticed RAID scrubbing still had about an hour to go this morning around 8:30. And around 9:30 the system speed is fast again.

My conclusion is that RAID scrubbing is causing the system to drop iSCSI connections as long as it is running. This may be ok for a home user who can schedule some downtime while things happen, but it's not something you'd have in a business environment. I'm pretty sure it caused some corruption in several VM's last weekend when I had the QNAPs loaded down with a few more apps running at the same time.

What can I do to lessen the impact of RAID scrubbing WHILE IT RUNS?
Last edited by OneCD on Mon Sep 28, 2020 4:35 am, edited 1 time in total.

phxazcraig
New here
Posts: 8
Joined: Sat Sep 12, 2020 6:09 am

Re: Constant isici disconnects caused by RAID scrubbing

Post by phxazcraig » Mon Sep 28, 2020 4:12 am

Argh, just to improve search engine results on this, let me spell the title correctly: Constant iSCSI disconnects caused by RAID scrubbing.

dgreenwood
First post
Posts: 1
Joined: Tue Apr 06, 2021 12:12 am

Re: Constant iSCSI disconnects caused by RAID scrubbing

Post by dgreenwood » Tue Apr 06, 2021 12:24 am

Really disappointed that no one followed up on your question, as I'm having the same issue. Scrubbing or syncing basically takes the QNAP offline and VMs that have an iSCSI connection to the device lock up until the process completes (about 27 hours in our case). Since I've found no way to disable or schedule the sync process, it takes down our backups on average at least once a month.

I have turned off RAID scrubbing, so that hasn't been an issue recently, but the syncing causes the same issue. It's not clear to me if this is expected behavior or if it's a problem that can be fixed.

User avatar
dolbyman
Guru
Posts: 22334
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: Constant iSCSI disconnects caused by RAID scrubbing

Post by dolbyman » Tue Apr 06, 2021 12:56 am

well if nobody knows an answer..what is supposed to be the follow up?

qnap doesn't come here..so if you want their take ..open a ticket

Post Reply

Return to “iSCSI – Target & Virtual Disk”