iSCSI issue after (forced) 5.0.0 upgrade

iSCSI related applications
Post Reply
tribunal3117
Getting the hang of things
Posts: 89
Joined: Thu Jul 31, 2014 8:40 am

iSCSI issue after (forced) 5.0.0 upgrade

Post by tribunal3117 »

I'm one of the poor schmucks that had their NAS (TS-851) forced to upgrade to 5.0.0 this past week. I'm especially angry because my NAS is not exposed to the internet and is behind a firewall here on my home network. In any case, since the upgrade, I have yet to be able to bring down my NAS gracefully. I think it has something to do with the iSCSI system, and I will tell you why.

1) The night after the upgrade, my vmware host (ESXi 7.0u2) lost connection with the VMs at the time that the malware scanner was running. The scan completed successfully, but I noticed that the iSCSI screen under storage still had the ESXi host listed as "connected" even though that host had powered down. I tried to manually disable the iSCSI LUNs, but the screen just kept spinning. So, I turned off the NAS, and it would not come down clean; it had to be forced. When it came back up, I was able to reconnect all the vms, and everything seemed fine. Then the following evening, also during the malware scan, the entire host crashed and restarted. This time a RAID resync was triggered. I shut down the vmware host again (which had reconnected), and this time 1 iSCIS connection went away, but the other persisted. I allowed the resync to finish. This time I just shut down the NAS, and again it had to force a shutdown. When it came back up, yet another resync was triggered. I let that one go, and then brought up the vmware host yet again. That was last night, and after disabling the malware detector, it did not crash last night. Things are functional this morning, but I suspect if I tried to bring the system down, I would once again have to force it.

I use file-based LUNs with iSCSI.

So to recap.

The malware remover was running at the times when the system either crashed, or forced the ESXi host to lose iSCSI connections. iSCSI screen does not show connections as being stopped even when they are. I thnk that the iSCSI driver/process is being aggravated by the malware remover. I also think that the persistent iSCSI connections are what is causing the system to not come down clean.

Is there a way to restart the iSCSI process from the command line? If I could kill the process manually, I suspect I could bring the NAS down cleanly.

Everything was nice and stable on 4.5. I'm incredibly upset at QNAP over this forced upgrade.
tribunal3117
Getting the hang of things
Posts: 89
Joined: Thu Jul 31, 2014 8:40 am

Re: iSCSI issue after (forced) 5.0.0 upgrade

Post by tribunal3117 »

After disabling the malware remover, the NAS has stayed up. I suspect if I tried to shut it down, it still wouldn't come down clean. Hopefully this workaround will at least allow it to stay up.
FSC830
Experience counts
Posts: 2043
Joined: Thu Mar 03, 2016 1:11 am

Re: iSCSI issue after (forced) 5.0.0 upgrade

Post by FSC830 »

A forced update? Yes and no: viewtopic.php?f=45&t=164923#p809767

Regards
firetcv
First post
Posts: 1
Joined: Sat Mar 11, 2023 4:25 pm

Re: iSCSI issue after (forced) 5.0.0 upgrade

Post by firetcv »

Have the same problem that was killing my iscsi with vmware for the last months.
Thank you for connecting my problem to the malware remover.

Have disabled it now and will monitor. Qutshero version 5.0
tribunal3117
Getting the hang of things
Posts: 89
Joined: Thu Jul 31, 2014 8:40 am

Re: iSCSI issue after (forced) 5.0.0 upgrade

Post by tribunal3117 »

This is an old post, but it turned out not to be the malware remover. The real culprit was a bug introduced in the iSCSI code that was fixed in QTS in May 2022 following a support ticket I opened for this issue. It was also aggravated by a nightly job which updated the sizes in the GUI for shares on the share management screen. Since it had no impact outside that screen (for space alerting or anything really important), I disabled that job. This made it less likely to experience the disconnects, but it still occurred on rare occasions, and very heavy traffic would still precipitate the issue. Once I applied a firmware with the fix, no more disconnects.

All that said, I've since upgraded to a TVS-h874 running QuTS 5.0 firmware from January 2023, and have not experienced any iSCSI issues.

If you're running an older firmware, I highly suggest you upgrade as the root cause was an identified bug.
Post Reply

Return to “iSCSI – Target & Virtual Disk”