Recent firmware updates broke iSCSI CHAP for me

iSCSI related applications
jpaalija
Starting out
Posts: 28
Joined: Tue Jul 12, 2016 12:54 am

Recent firmware updates broke iSCSI CHAP for me

Post by jpaalija »

I had initially firmware 4.5.3.1652(20210428) installed on my TS-853A. I can confirm iSCSI CHAP was working there just fine.

Couple of days ago QNAP installed without my permission firmware update 4.5.6.1670(20210515). QNAP rebooted automatically. My ESXi virtual machines crashed as they lost the iSCSI disks.

After a day I had time to investigate. I manually updated to 4.5.4.1715(20210630).

I discovered that now the iSCSI CHAP authentication is broken.

There are 3 options:

Update #1 broke it
Update #2 broke it
Automatic reboot of update #1 broke it

I have tried everything. Actually I spent my whole working day trying to fix this. I tried rebooting in different order, re-setting the same password to both QNAP and ESXi. Ping works, iSCSI WITHOUT the CHAP works. But I need working CHAP.

I sent this as ticket also.

I had planned to work on customer projects this day, but instead I have been spending the whole day trying to resolve this. I have absolutely no time to look into this anymore, I already lost 1000USD today because of the downtime :(

Anyone else had similar issues?
Last edited by jpaalija on Sat Jul 03, 2021 3:34 am, edited 1 time in total.
User avatar
dolbyman
Guru
Posts: 34903
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: Recent firmware updates broke iSCI CHAP for me

Post by dolbyman »

see here

viewtopic.php?f=25&t=161990

ticket is the only way to fix this .. as qnap does not come here
jpaalija
Starting out
Posts: 28
Joined: Tue Jul 12, 2016 12:54 am

Re: Recent firmware updates broke iSCI CHAP for me

Post by jpaalija »

dolbyman wrote: Sat Jul 03, 2021 12:59 am see here

viewtopic.php?f=25&t=161990

ticket is the only way to fix this .. as qnap does not come here
Good catch! I'm happy I'm not alone.
jpaalija
Starting out
Posts: 28
Joined: Tue Jul 12, 2016 12:54 am

Re: Recent firmware updates broke iSCI CHAP for me

Post by jpaalija »

Gotta drop this in regardless..

Logged in via SSH.

Code: Select all

[admin@NASFAD688 config]# ps aux | grep iscsi
 3259 admin           SW  [iscsi_ttx]
 3263 admin           SW  [iscsi_trx]
10987 admin           SW  [iscsi_np]
11006 admin      3452 S   /sbin/iscsi_logd
11024 admin           IW< [iscsi_eh]
11068 admin      1328 S   /sbin/iscsid --config=/etc/config/iscsi/sbin/iscsid.conf --initiatorname=/etc/iscsi/initiatorname.iscsi
11069 admin      2728 S < /sbin/iscsid --config=/etc/config/iscsi/sbin/iscsid.conf --initiatorname=/etc/iscsi/initiatorname.iscsi
20920 admin           SW  [iscsi_np]
24351 admin      1144 S   grep iscsi
Also

Code: Select all

[admin@NASFAD688 config]# cd /etc/config/
[admin@NASFAD688 config]# find | grep iscsi
./iscsi_trgt.conf.replaced
./iscsi_trgt.conf
./iscsi-init.conf
./iscsi_trgt.conf.last
[admin@NASFAD688 config]# pwd
/etc/config
Did we used to have /etc/config/iscsi/sbin/iscsid.conf ? Now we don't. Init scripts also need it to my understanding:

Code: Select all

#!/bin/sh
#
# chkconfig: 345 13 89
# description: Starts and stops the iSCSI initiator
#
# processname: /sbin/iscsid
# pidfile: /var/run/sbin/iscsid.pid
# config:  /etc/config/iscsi/sbin/iscsid.conf

# Source function library.

PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
INITNAME_FILE=/etc/iscsi/initiatorname.iscsi
CONF_FILE=/etc/config/iscsi/sbin/iscsid.conf
MAPPING_FILE=/etc/config/virtual_share_mapping
RETVAL=0
HERO_PROD=0
...
jpaalija
Starting out
Posts: 28
Joined: Tue Jul 12, 2016 12:54 am

Re: Recent firmware updates broke iSCI CHAP for me

Post by jpaalija »

Another poke:

Code: Select all

[admin@NASFAD688 config]# ll /etc/config/iscsi_trgt.conf
-rw-r--r-- 1 admin administrators 2.3K 2021-07-02 20:01 /etc/config/iscsi_trgt.conf
Contents:

Code: Select all

[target0Init1Info]
targetIndex = 0
initiatorIndex = 1
bCHAPEnable = FALSE
CHAPUserName =
CHAPPasswd =
bMutualCHAPEnable = FALSE
mutualCHAPUserName =
mutualCHAPPasswd =

[target0Init0Info]
targetIndex = 0
initiatorIndex = 0
bCHAPEnable = FALSE
CHAPUserName =
CHAPPasswd =
bMutualCHAPEnable = FALSE
mutualCHAPUserName =
mutualCHAPPasswd =
These are probably for QNAP UI. But the data needs to go to iscsid.conf in the correct format, I think.
jpaalija
Starting out
Posts: 28
Joined: Tue Jul 12, 2016 12:54 am

Re: Recent firmware updates broke iSCI CHAP for me

Post by jpaalija »

Changing on the Web User Interface the settings. They end up in/etc/config/iscsi_trgt.conf :

Code: Select all

[target0Init1Info]
targetIndex = 0
initiatorIndex = 1
bCHAPEnable = FALSE
CHAPUserName =
CHAPPasswd =
bMutualCHAPEnable = FALSE
mutualCHAPUserName =
mutualCHAPPasswd =

[target0Init0Info]
targetIndex = 0
initiatorIndex = 0
bCHAPEnable = TRUE
CHAPUserName = esxi
CHAPPasswd = esxi.12345678
bMutualCHAPEnable = FALSE
mutualCHAPUserName =
mutualCHAPPasswd =
How should iscsid get this data? Is it patched to read directly iscsi_trgt.conf ? I see also lines like

Code: Select all

LUNMetaPath = /share/CE_CACHEDEV2_DATA/.@iscsi.img/iSCSI-raidvol2lun-5ec8d40c:008
So maybe it reads directly the file but fails on the CHAP info?

I'll stop now for reals.
jpaalija
Starting out
Posts: 28
Joined: Tue Jul 12, 2016 12:54 am

Re: Recent firmware updates broke iSCSI CHAP for me

Post by jpaalija »

I was able to once connect to the service from VMware but after normal test reboots I could not do it anymore. I'm trying to get back to my data. Looks a bit grim right now. I'm trying everything, which probably is not a good thing.
jpaalija
Starting out
Posts: 28
Joined: Tue Jul 12, 2016 12:54 am

Re: Recent firmware updates broke iSCSI CHAP for me

Post by jpaalija »

Ok I have absolutely no idea what is going on anymore. I upgraded firmware back and forth, rebooted rebooted and rebooted, read strange kernel logs with info like

Code: Select all

[24124.921358] Unable to load target_core_user
[24124.981357] ------------[ cut here ]------------
[24124.986037] WARNING: CPU: 3 PID: 29816 at drivers/target/qnap/target_core_qtransport.c:470 qnap_transport_config_blkio_dev+0x10a/0x120 [target_core_mod]
[24124.999735] Modules linked in: target_core_pscsi rfcomm iscsi_tcp(O) libiscsi_tcp(O) libiscsi(O) scsi_transport_iscsi(O) iscsi_target_mod target_core_file target_core_iblock target_core_mod iscsi_target_qlog fbdisk(O) xt_nat dummy br_netfilter bridge stp bonding xt_connmark xt_TCPMSS xt_LOG xt_set ip_set_hash_netiface ip_set_hash_net ip_set ipt_MASQUERADE xt_REDIRECT nf_nat_redirect iptable_nat nf_nat_masquerade_ipv4 nf_nat_ipv4 nf_nat xt_policy xt_mark 8021q ipv6 uvcvideo videobuf2_v4l2 videobuf2_vmalloc videobuf2_memops videobuf2_core snd_usb_caiaq snd_usb_audio snd_usbmidi_lib snd_seq_midi snd_rawmidi fnotify(O) nfsd udf isofs sp5100_tco iTCO_wdt kcopy(PO) qtweak(PO) vfio_pci irqbypass vfio_virqfd vfio tbs_keys(O) ufsd(PO) jnl(O) cdc_acm pl2303 usbserial qm2_i2c(O) intel_ips drbd lru_cache flashcache(O)
[24125.071299]  dm_tier_hro_algo dm_thin_pool dm_bio_prison dm_persistent_data hal_netlink(O) k10temp coretemp r8152 usbnet mii igb e1000e(O) mv14xx(PO) scsi_transport_sas raid_class scsi_transport_fc uas usb_storage xhci_pci xhci_hcd usblp uhci_hcd ehci_pci ehci_hcd
[24125.094733] CPU: 3 PID: 29816 Comm: echo Tainted: P     U     O    4.14.24-qnap #1
[24125.102321] Hardware name: To be filled by O.E.M. To be filled by O.E.M./Aptio CRB, BIOS QX47AR12 10/01/2015
[24125.112177] task: ffff88023e9d2400 task.stack: ffffc90000154000
[24125.118143] RIP: 0010:qnap_transport_config_blkio_dev+0x10a/0x120 [target_core_mod]
[24125.125831] RSP: 0018:ffffc90000157d20 EFLAGS: 00010246
[24125.131080] RAX: 0000000000000000 RBX: ffff88022f3dc000 RCX: 0000000000000000
[24125.138242] RDX: 00000000ffffffed RSI: ffff88024cd75012 RDI: ffffffffa076907d
[24125.145415] RBP: 0000000000000000 R08: 0000000000000200 R09: 0000000000000000
[24125.152571] R10: 00000000000000fc R11: ffff8802341cf4f0 R12: ffff88024e490d00
[24125.159728] R13: ffff88022f3dd1d0 R14: 0000000000000083 R15: ffff88023d863500
[24125.166886] FS:  00007f19cf78f700(0000) GS:ffff88027fd80000(0000) knlGS:0000000000000000
[24125.175000] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[24125.180765] CR2: 00007ffce5c5f0d0 CR3: 000000023db0a000 CR4: 00000000001006e0
[24125.187922] Call Trace:
[24125.190399]  iblock_configure_device+0x1b6/0x450 [target_core_iblock]
[24125.196879]  target_configure_device+0x7f/0x6a0 [target_core_mod]
[24125.203050]  target_dev_enable_store+0x29/0x50 [target_core_mod]
[24125.209167]  configfs_write_file+0x9f/0xf0
[24125.213290]  __vfs_write+0x1e/0x130
[24125.216916]  ? __sb_start_write+0x51/0x90
[24125.220949]  vfs_write+0xf6/0x200
[24125.224355]  SyS_write+0x3d/0x90
[24125.227608]  do_syscall_64+0x83/0x2c0
[24125.231297]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[24125.236438] RIP: 0033:0x7f19cefb05d0
[24125.240032] RSP: 002b:00007fffebf12488 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[24125.247623] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f19cefb05d0
[24125.254777] RDX: 0000000000000002 RSI: 0000000001b6e010 RDI: 0000000000000001
[24125.261931] RBP: 0000000001b6e010 R08: 0000000000000000 R09: 0000000000000000
[24125.269085] R10: 000000000000085c R11: 0000000000000246 R12: 0000000000000002
[24125.276245] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000000
[24125.283402] Code: 98 10 00 00 04 00 00 00 48 c7 c6 80 6b 76 a0 48 c7 c7 b0 2c 77 a0 31 c0 bd ea ff ff ff e8 90 e2 96 e0 5b 89 e8 5d 41 5c 41 5d c3 <0f> 0b e9 4a ff ff ff bd ea ff ff ff eb e9 0f 1f 84 00 00 00 00
[24125.302335] ---[ end trace ca16e170925abf8a ]---
[24125.307055] IBLOCK: not support fast clone
At some point I backed up my iscsi config files
/etc/config/iscsi-init.conf /etc/config/iscsi_trgt.conf /etc/config/iscsi_trgt.conf.last /etc/config/iscsi_trgt.conf.replaced

At some point I also run storage util with iscsi_restore_cfg DONT TRY THIS AT HOME! I was lucky. This created a clean config file. VMware showed at some point the datastore as degraded and did not list anything.

I manually updated iscsi_trgt.conf then (with iscsi disabled). LUNSectorSize was 0. I set it back to 512. LUNInitBitmap00 were wrong? I set them from 00000001 to 00000003. Also some of the IQNs were 2003, I changed them to 2004 per what was in the backed up iscsi_trgt.conf

I rebooted, carefully unlocked my volumes , started vmware (without CHAP). And it worked. I was able to get my business critical data out. I will make a proper backupping solution before doing anything else.

PLEASE NOTE I MADE HUGE GAMBLES HERE AND BY SHEER LUCK IT WORKED. DONT BLAME ME IF YOU LOSE YOUR DATA WHILE FOLLOWING MY ACTIONS.
jpaalija
Starting out
Posts: 28
Joined: Tue Jul 12, 2016 12:54 am

Re: Recent firmware updates broke iSCSI CHAP for me

Post by jpaalija »

Some wild speculations while backups are running:

Maybe there is random % change of iSCSI functionality getting corrupted on every update

Maybe the problem was the automatic update which violently tore down the iSCSI from VMware

Anyways, I'm not doing anything until all backups have ran.
jpaalija
Starting out
Posts: 28
Joined: Tue Jul 12, 2016 12:54 am

Re: Recent firmware updates broke iSCSI CHAP for me

Post by jpaalija »

normal_degraded.png
If I let QNAP boot without much intervention, my ESXi gets the following condition about the iSCSI disk device being "Normal, Degraded"
You do not have the required permissions to view the files attached to this post.
jpaalija
Starting out
Posts: 28
Joined: Tue Jul 12, 2016 12:54 am

Re: Recent firmware updates broke iSCSI CHAP for me

Post by jpaalija »

Datastore is fine though?
datastore_fine.png
You do not have the required permissions to view the files attached to this post.
jpaalija
Starting out
Posts: 28
Joined: Tue Jul 12, 2016 12:54 am

Re: Recent firmware updates broke iSCSI CHAP for me

Post by jpaalija »

Okay, QNAP contacted my box and they said they will be fixing the CHAP issue in QTS 4.5.4.1728_20210713.
vinix
Starting out
Posts: 11
Joined: Tue Nov 28, 2017 12:59 pm

Re: Recent firmware updates broke iSCSI CHAP for me

Post by vinix »

I'm using QTS 4.5.4.1741 on TS-831X, NAS rebooted unexpectedly right after I enable CHAP. I've opened a ticket for another problem but talked about this. QNAP said it will be fixed in 4.5.4.1777, which is not released yet.
jpaalija
Starting out
Posts: 28
Joined: Tue Jul 12, 2016 12:54 am

Re: Recent firmware updates broke iSCSI CHAP for me

Post by jpaalija »

I lost my datastore again. I updated to the newest, whatever 5.xxx QTS. But I got the datastores back again.

I did:

Code: Select all

[root@ESXI:~] esxcfg-volume -l
Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).
VMFS UUID/label: 5ec91bb1-b662df65-2679-54b2031beace/QNAP
Can mount: Yes
Can resignature: Yes
Extent name: naa.6e843b6ff5661dfddb59d435cd9af5d2:1     range: 0 - 7464703 (MB)
Then

Code: Select all

[root@ESXI:~] esxcfg-volume -M 5ec91bb1-b662df65-2679-54b2031beace
Persistently mounting volume 5ec91bb1-b662df65-2679-54b2031beace
Now even CHAP is working and the changes persist after boot. But I think I should at some point maybe make export of data and reformat of the store or something. But this shall do for now. Phew.
jpaalija
Starting out
Posts: 28
Joined: Tue Jul 12, 2016 12:54 am

Re: Recent firmware updates broke iSCSI CHAP for me

Post by jpaalija »

I reinstalled everything and I'm testing how things work.

I have an idea that ESXi is extremely allergic to like any configuration changes after the iSCSI is working.

See: https://kb.vmware.com/s/article/1011387

"The snapshot LUNs issue occurs when the ESX host does not confirm the identity of the LUN with what it expects to see in the VMFS metadata. This issue occurs after replacing SAN hardware, firmware upgrades, SAN replication, DR tests, and some HBA firmware upgrades."

I sure hope upgrading QNAP firmware will not break stuff in the future.
Post Reply

Return to “iSCSI – Target & Virtual Disk”