Constant iSCSI disconnects caused by RAID scrubbing

iSCSI related applications
Post Reply
phxazcraig
Starting out
Posts: 10
Joined: Sat Sep 12, 2020 6:09 am

Constant iSCSI disconnects caused by RAID scrubbing

Post by phxazcraig »

Title sums it up, this is the detail.
1. TS-451+ (two of them, each set up with a 5TB iSCSI target on interface #2, 1Gb connection.
2. Latest firmware, but also last one as I tried backrevving at one point. Firmware 4.4.3.1421
3. 16GB RAM, each 451 has four HDD's. One has 4 10TB drives, one has 4 4TB drives, both in RAID5
4. Three ESXi 6.7 servers making iSCSI connections to each TS-451+ and using datastores on each QNAP.
5. Dedicated iSCSI network for iSCSI, with two QNAPs and three ESXi connections and nothing else. I bought managed switches for both interfaces for testing to see if there was weird traffic on the subnet, but if anything there was less there than I thought. Nothing that looked like it wasn't normal traffic.
6. Symptom: In a word, dog slow. Anything coming off of those ESXi servers that needed to read some data from disk was very, very slow. For instance, I have a web site on a virtual linux box, and it was 10x slower at least serving up images once you got past the front page. Vcenter (virtual appliance here) very slow to do anything, almost unusable.
7. Event log on each ESXi server was filled with several hundred (over 300) iSCSI disconnect/reconnect messages. Last night they started at 4:45am, and ended at about 9:31am.
8. RAID scrubbing was scheduled to run last night (each Sunday) and start at 4am.
9. I noticed RAID scrubbing still had about an hour to go this morning around 8:30. And around 9:30 the system speed is fast again.

My conclusion is that RAID scrubbing is causing the system to drop iSCSI connections as long as it is running. This may be ok for a home user who can schedule some downtime while things happen, but it's not something you'd have in a business environment. I'm pretty sure it caused some corruption in several VM's last weekend when I had the QNAPs loaded down with a few more apps running at the same time.

What can I do to lessen the impact of RAID scrubbing WHILE IT RUNS?
Last edited by OneCD on Mon Sep 28, 2020 4:35 am, edited 1 time in total.
phxazcraig
Starting out
Posts: 10
Joined: Sat Sep 12, 2020 6:09 am

Re: Constant isici disconnects caused by RAID scrubbing

Post by phxazcraig »

Argh, just to improve search engine results on this, let me spell the title correctly: Constant iSCSI disconnects caused by RAID scrubbing.
dgreenwood
First post
Posts: 1
Joined: Tue Apr 06, 2021 12:12 am

Re: Constant iSCSI disconnects caused by RAID scrubbing

Post by dgreenwood »

Really disappointed that no one followed up on your question, as I'm having the same issue. Scrubbing or syncing basically takes the QNAP offline and VMs that have an iSCSI connection to the device lock up until the process completes (about 27 hours in our case). Since I've found no way to disable or schedule the sync process, it takes down our backups on average at least once a month.

I have turned off RAID scrubbing, so that hasn't been an issue recently, but the syncing causes the same issue. It's not clear to me if this is expected behavior or if it's a problem that can be fixed.
User avatar
dolbyman
Guru
Posts: 34903
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: Constant iSCSI disconnects caused by RAID scrubbing

Post by dolbyman »

well if nobody knows an answer..what is supposed to be the follow up?

qnap doesn't come here..so if you want their take ..open a ticket
deathmage85
Starting out
Posts: 20
Joined: Thu May 06, 2021 12:24 pm

Re: Constant iSCSI disconnects caused by RAID scrubbing

Post by deathmage85 »

I personally use a pair of TS-832X's in my home-lab running a mix of 6.7 and 7.0 from VMware vExpert licensing.

I do have a few observations and questions for you, as I run VMware all the time and have for 8 years now over iSCSI in the lab and in production with QNAP arrays with zero instances of RAID scrubbing, my lab runs on 10G with QM2 R/W caching and Qtier enabled:

1. You say your using 1G on the connection, is jumbo frames enabled? - is jumbo frames enabled thru-n-thru?
2. Are you using a separate iSCSI switch in ESXi or is it using the same default vSS?
3. What is the I/O protocol your using in VMware for the iSCSI switch? - you should be using IP Hash
4. What kind of switch are you using for iSCSI?
5. When you turn off your ESXi lab, do you have a proper shutdown order? - i.e your NAS should be the last to power off - you should power off VM's, then ESXi hosts, and then you can safely turn off the NAS.
6. what is your RAID configuration for the NAS?
7. you said 5 TB's, what is the size of one drive in your RAID config?
8. Does your NAS have a QM2 card?
9. Are you using Qtier?

This is a good starting point.
Sr. System Administrator
Certifications: Microsoft Certified Solutions Expert x 3, Microsoft 365 Certified: Enterprise Administrator Expert, Microsoft Certified: Azure Administrator, Office 365 Messaging Administrator, Microsoft Certified: Azure Virtual Desktop Specialty, VMware VCP 5 & 6 DCV/NV, VMware vExpert 2015-2021, CompTIA: A+, Network+, Storage+, Server+, Security+, CySA+, CASP+
Blog: https://www.G15IT.com

Lab:
(1) TS-832X - (Hybrid) - (4) 8 TB WD Red Pro in RAID 10, (4) WD Blue 1 TB SSD's in RAID 5 - QM2 Dual M.2 SSD card

(1) TS-832X - (All-Flash) - (8) 1 TB WD Blue SSD's in RAID 5 - QM2 Dual M.2 SSD card

(2) TS-230 - (2) 6 TB's RAID 1 - Backup repository for Veeam BR 9.5

(1) TS-453D - (4) 6 TB's in RAID 6 - File Server with QM2 Dual M.2 SSD card
Post Reply

Return to “iSCSI – Target & Virtual Disk”