iSCSI problems after upgrade to 4.3.3

[shpokas]

Hi,
we have a QNAP Model:TS-870U-RP
Current firmware version:4.3.3.0210
Date:2017/06/06

It is connected via 10GBe to 2 ESXi 5.5 U3 servers.
All was fine for years until upgrade to QTS 4.3.3.

Issues:

1) high CPU usage, 15% when lightly loaded, goes up to 100%;

Screen Shot 2017-06-13 at 12.05.13.png

2) high disk latency (as in screenshots attached);

Screen Shot 2017-06-13 at 12.04.57.png

3) ESXi reports no VAAI UNMAP capability for storage;

# esxcli storage core device vaai status get|grep Delete
Delete Status: unsupported

4) VMFS heartbeat timeouts on regular intervals as in screenshots;

Screen Shot 2017-06-13 at 12.09.01.png

5) error and warning messages in iSCSI target log;
<4>[125056.785131] Detected MISCOMPARE for addr: ffff88008f176000 buf: ffff8800cda93200
<4>[125056.792847] Target/fileio: Send MISCOMPARE check condition and sense
<4>[125056.800768] ttyS0: 2 input overrun(s)
<4>[125141.604805] Detected MISCOMPARE for addr: ffff880020964000 buf: ffff8800b0300000
<4>[125141.612742] Target/fileio: Send MISCOMPARE check condition and sense
<4>[125141.619632] ttyS0: 3 input overrun(s)
<4>[125357.175734] ABORT_TASK: Found referenced iSCSI task_tag:0x3c490000, cmdsn:0x00004935
<6>[125357.183626] [iSCSI (block-based)][ABORT TASK] ip: 172.16.100.2, drop itt:0x3c490000, scsi op:0x12.
<4>[125357.192749] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag:0x3c490000. Aborted task: itt:0x3c490000, cmdsn:0x00004
935, scsi op:0x12
<4>[125357.206548] ttyS0: 2 input overrun(s)
<4>[125736.570963] ABORT_TASK: Found referenced iSCSI task_tag:0xf1b32300, cmdsn:0x0023afe7
<6>[125736.579470] [iSCSI (block-based)][ABORT TASK] ip: 172.16.1.2, drop itt:0xf1b32300, scsi op:0x12.
<4>[125736.588418] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag:0xf1b32300. Aborted task: itt:0xf1b32300, cmdsn:0x0023a
fe7, scsi op:0x12
<4>[125736.601847] ttyS0: 2 input overrun(s)
<4>[125916.992605] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 0xc68a1600
<4>[125917.000727] ttyS0: 3 input overrun(s)
<4>[126976.190926] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 0x3e2c0802
<4>[126976.199036] ttyS0: 2 input overrun(s)

There are no errors on network level, no packed drops or errors from both sides (ESXi, QNAP). We use MTU 9000 for all connections to QNAP via iSCSI.

As said, no issues for years before.
Any insight much appreciated. Thanks.

shpokas

[storageman]

1.Recheck Service Binding and ISCSI ports
2.Disconnect in ESXIi.
3.Delete and recreate targets on NAS.
4.Reconnect on ESXi.
5.If this fails can you connect to same target from another host (bare metal server).

[shpokas]

Done 1-4 with no improvement.

[storageman]

Did you try stop and start the ISCSI service?
Have you proved a new LUN/target works?
There are some pointers here that may help:
viewtopic.php?t=19112

[shpokas]

Hi,
did I say that LUN/target is not working? It is working, but ESXi servers lose access to VMFS volumes periodically.

I did a complete QNAP reset (keeping data) and later created iSCSI configuration again. Thanks for pointer, but I do not need to restore iSCSI configuration.
I need to get back to stable operation with ESXi servers working properly.

Br,
shpokas

[storageman]

I would definitely turn off jumbo frames for now and reboot and retest.
When the volumes disappear offline, are the ISCSI ports pinging OK from ESXi without latency.
Out of interest does an NFS datastore connect more reliably using the plugin?
https://www.qnap.com/en/how-to/tutorial ... ation-vaai

[shpokas]

Well, there's no a single error on any network interface that uses jumbo frames.
QNAP is connected directly to ESXi boxes (no switch).

On top of all this mess I deleted storage pool, thrashing backup data.
Not when I try to create same storage pool again, RAID0 with 3 disks, QNAP fails. kmesg log has errors regarding lvcreate command.
I suspect that 4.3.3 has botched this QNAP, but I need a way to recover without copying off 2 TB of data.

[storageman]

Now, I'm lost. How did you delete the pool and still be worried about copying off 2TB?
Clearly you know RAID 0 could be your problem here.

[shpokas]

I had two storage pools, RAID6 for data and RAID0 for backups. I still need to move RAID6 off QNAP.

[Henri04]

HI,

same here, downgrade to 4.2.5, than it will work again.
Problem started with 4.3, had to downgrade a couple boxes back to 4.2.5.
Need the QFIX for 4.2.5, how to get it?

Thanks

Henri

[shpokas]

Hi,
do you have similar issues?
With iSCSI? And ESXi?
Have you by any chance logged a case to QNAP? Maybe they will notice and fix this?
Did you mean a QFIX for Samba security problem?
Br,
shpokas

[shpokas]

I, too, downgraded to 4.2.5 and iSCSI issues disappeared.
1) No more errors and warnings in QNAP's /mnt/HDA_ROOT/.logs/kmsg log file
2) No more VMFS heartbeat timeouts for ESXi servers.

So 4.3.3 is probably still no go for iSCSI users. I have made a mistake and paid dearly.

[storageman]

Joke firmware releases again from Qnap - 4.3.3.
Their firmware release approach is appalling. They don't seem to have the slightest idea how to test properly.

[cw1972]

Interesting - all my iSCSI problems on my TS-853A after I upgraded to 4.3.3 - maybe I should look into downgrading, I'm having some real problems

[shpokas]

Latest QNAP firmware, 4.3.3.0262, 20170727 has resolved my iSCSI issues. It was suggested by QNAP support to install it and indeed, it has helped.
Now I can see performance improvements and lower CPU usage compared to 4.2.5.
Regards,
shpokas