iSCSI Problems with QuTS Hero h5.2.2.2952 build 20241116

[ez12a]

Not sure where to put this since the TVS-h474 isn't considered a "business" NAS, so i'll put this here i guess.

Just FYI i had issues with h5.2.2.2952 build 20241116 and iSCSI. Issues would manifest as 1021 errors in Ubuntu 24.04 LTS. The XFS filesystem would offline, and i'd get several I/O errors occasionally throughout the day. This would happen at least once a day. It seemed to get progressively worse where I would restart the NUC a few times a day and the filesystem would drop out after a few hours, at which point i opened a QNAP support ticket. NFS mounts worked w/o issue.

Some pertinent log messages from Ubuntu if anyone needs:

Code: Select all

connection1:0: detected conn error (1021)
Kernel reported iSCSI connection 1:0 error (1021 - ISCSI_ERR_SCSI_EH_SESSION_RST: Session was dropped as a result of SCSI error recovery) state (3)
Kernel reported iSCSI connection 1:0 error (1021 - ISCSI_ERR_SCSI_EH_SESSION_RST: Session was dropped as a result of SCSI error recovery) state (2)
kernel: sd 2:0:0:1: Device offlined - not ready after error recovery
kernel: sd 2:0:0:1: rejecting I/O to offline device
kernel: XFS (sdb1): Filesystem has been shut down due to log error (0x2).

Based on some research it seems mostly an issue on the target (NAS) side. Replaced the network cables as a precaution to no avail. I unfortunately could not wait days for developers to remotely log in (i run home assistant off of the LUN and not having that automation was getting in the way) so I reverted back to the latest h5.1.9 to see if it would resolve the issues. The downgrade was successful and outside some icons in the Web UI being outdated (fixed by removing the old icon and replace with downgraded File Station app, as an example), everything else seems to work fine.

With h5.1.9 build it has been solid for over 24 hours now. No 1021 or offline filesystems.

High level overview of my setup:
1. Asus NUC 14 Pro running Ubuntu 24.04 LTS Server
2. Asus XT8 Router
3. QNAP TVS-h474

beware if you use iSCSI on this version!

[dolbyman]

open a ticket with QNAP please

[ez12a]

open a ticket with QNAP please

Yep i did, it's still open but i already reverted. This is just a heads up to anyone else having issues with iSCSI. Hopefully they have the ability to wait for support. I did not unfortunately.

[ez12a]

So issue popped up again after a 8 days on the rolled back version.

Went nearly scorched earth and tested a variety of quts hero versions and Ubuntu 22.04 as well. I noticed it got really bad when the scheduled scrubbing kicked off on new years. Like iscsi target would just stop responding and filesystems would drop.

Now I've landed on it being potentially cause by the Tailscale application I had installed early December. It's not clear exactly what in the configuration is causing it as I don't remember when I turned on/off certain features after installation of the app.

I did have the Tailscale on the qnap configured as an exit node along with the local subnet routed so I could use my pihole dns server on tailscale clients.

Since stopping tailscale in the qnap UI I'm seeing zero network related issues with iscsi on the NUC for the past 24 hr. I used see an occasional read or write I/O error (recoverable) before. Will keep monitoring but evidently may need at least a week to know for sure. Scrubbing continued without issue either.

[ez12a]

Update: looks like zfs scrubbing a cause of iscsi timeouts. Even at low priority.

I went as far as resetting the NAS, upgrading to the latest release, connecting a raspberry pi as well and was able to reproduce iscsi 1022 timeouts on it (though never dropped the filesystem). Since the nuc hw is relatively new I got a separate thunderbolt nic dongle and directly connected the nuc to the NAS 2nd nic, and still has timeouts. All the whole a low priority scrub was running.

Updated my support ticket with the findings. The performance impact of the scrub on iscsi is unacceptable. Iscsi unreliable for basically 24 hours for the scrub to complete.

[jonathanuwguy]

I ran into the same issue after upgrading to QTS 5.2.3.3006. My ubuntu failed to mount the target.
The error I received from iscsiadm is: iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
Tcpdump showing response with failure from qnap: Status: Initiator error (miscellaneous error) (0x0200)

I already make sure that only connected adapter is checked and that's not the issue. Doing some searching, the closest thing I found was mismatched config between the qnap array and what client sent. Because I had another ubuntu system that also connected to the same qnap server but to a different iscsi target and that one was working ok, I tried to use that system to mount the problematic iscsi target and I got the exact same issue. I did packet captures and compare each values sent from the client to the 2 iscsi targets and they're identical. That pointed to something messed up with that particular iscsi target after the upgrade.

What I did to fix my issue if going on to qnap, deleted the iscsi target (NOT THE LUN). You can do this by select the iscsi target and delete it. The LUN will become unmapped. I then recreated a new iscsi target and re-mapped that LUN to the new target and I was able to login successfully from my ubuntu box.

This may or may not work for you but it's just another solution that can be tried out for this problem. Hope Qnap has a fix for this soon. Cheers