iSCSI Freezes or Hangs

iSCSI related applications
ChrisPML
New here
Posts: 2
Joined: Fri Nov 15, 2013 3:20 pm

iSCSI Freezes or Hangs

Post by ChrisPML »

I have a TS-879U-RP system. We bought it to use as shared storage for a Server 2012 R2 failover cluster. We have two SSDs in RAID 1 and iSCSI setup for the CSV. We setup 2 VMs that are lightly used. Then yesterday we started to install SQL server on one of the VMs and the whole VM froze, at first we couldn't’t figure out what happened until we looked at the Failover Cluster Manger and saw the CSV Disks were offline and the ISCSI initiator could not reconnect. I logged into the Qnap web interface and everything seemed to be fine except the ISCSI Manager was running slow and was saying it still had connections. At this point I restarted it and waited 10 mins and logged back in to the web interface, only to find the ISCSI still wasn’t working and the system never restarted. I attempted another restart only to have it fail. I finally had to walk to the server room and manually hold down the power button to restart the system and everything started working again. Through all of this I got no warnings or errors (that would have been emailed to me).

I want to know if there is any commands, that I can use via the web interface or SSH, to see the state of the ISCSI Service, or restart it.

Thanks for any help.
Chris
akb123
First post
Posts: 1
Joined: Wed Jan 08, 2014 3:00 pm

Re: iSCSI Freezes or Hangs

Post by akb123 »

I have the exact same think happening to me!!!

I have a TS-879U-RP (firmware 4.0.5) with a Emulex OCe11102-NX NIC and SSDs. I am using it as a iSCSI target for a Windows 2012R2 Hyper-V cluster. The configuration passed the Windows Cluster verification and I thought all was good at first.

It ran for a month with out issue. Then last week we deployed an application that heavily uses file I/O. With some investigation I found that any sustained file writes to the virtual disk on the iSCSI LUN, like copying a 8GB file, will cause the LUN to crash. I thought it was a network problem, but then I saw you have the same issue so it must be the QNAP NAS. Did you solve your issue?
alankromer
First post
Posts: 1
Joined: Tue Aug 24, 2010 1:02 pm

Re: iSCSI Freezes or Hangs

Post by alankromer »

same here :evil:
100% reproducible

<4>[60374.840127] ------------[ cut here ]------------
<2>[60374.841019] kernel BUG at mm/slub.c:1363!
<4>[60374.841019] invalid opcode: 0000 [#1] SMP
<4>[60374.841019] Modules linked in: iscsi_tcp(O) libiscsi_tcp(O) libiscsi(O) scsi_transport_iscsi(O) iscsi_target_mod target_core_file target_core_iblock target_core_mod fbdisk(O) 8021q uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev v4l2_int_device snd_usb_caiaq snd_usb_audio hid_logitech snd_usbmidi_lib snd_hwdep snd_seq_midi snd_rawmidi fnotify(PO) xhci_hcd udf isofs thfsplus(O) tntfs(PO) usblp iTCO_wdt iTCO_vendor_support rtcmod(O) usbhid cryptodev(O) hal_netlink(O) usb_storage uhci_hcd ehci_hcd usbcore usb_common mv_sata(O)
<4>[60374.841019]
<4>[60374.841019] Pid: 9238, comm: QNAP_iblock Tainted: P O 3.4.6 #1 ICP / iEi ATOM NAS /ATOM NAS (QE05)
<4>[60374.841019] EIP: 0060:[<c049c610>] EFLAGS: 00010002 CPU: 1
<4>[60374.841019] EIP is at new_slab+0x16/0x18d
<4>[60374.841019] EAX: f6402480 EBX: 00000000 ECX: ffffffff EDX: 00008014
<4>[60374.841019] ESI: ffffffff EDI: f6401120 EBP: d3fbfd04 ESP: d3fbfcec
<4>[60374.841019] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
<4>[60374.841019] CR0: 8005003b CR2: b76176a0 CR3: 15289000 CR4: 000007c0
<4>[60374.841019] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
<4>[60374.841019] DR6: ffff0ff0 DR7: 00000400
<0>[60374.841019] Process QNAP_iblock (pid: 9238, ti=d3fbe000 task=d4a89360 task.ti=d3fbe000)
<0>[60374.841019] Stack:
<4>[60374.841019] f6402480 00000000 d3fbfd74 00000000 f6972558 f6401120 d3fbfd74 c049cf91
<4>[60374.841019] ffffffff 00008014 f6402480 00000296 00000000 00000010 f8bcae1d 00000000
<4>[60374.841019] f6402300 00000000 d3fbfdcc d3fbfd40 c0a62580 c0a6bdac 000082d0 000082d0
<0>[60374.841019] Call Trace:
<4>[60374.841019] [<c049cf91>] __slab_alloc+0x1de/0x299
<4>[60374.841019] [<f8bcae1d>] ? __tpc_do_rw+0x314/0x55d [target_core_mod]
<4>[60374.841019] [<c047ac52>] ? __alloc_pages_nodemask+0xdd/0x4d3
<4>[60374.841019] [<c043d333>] ? need_resched+0x14/0x1e
<4>[60374.841019] [<f8bca83b>] ? kzalloc+0xb/0xd [target_core_mod]
<4>[60374.841019] [<c049d0c3>] __kmalloc+0x77/0xe2
<4>[60374.841019] [<f8bca83b>] ? kzalloc+0xb/0xd [target_core_mod]
<4>[60374.841019] [<f8bca83b>] ? kzalloc+0xb/0xd [target_core_mod]
<4>[60374.841019] [<c0484801>] ? __mod_zone_page_state+0x3d/0x46
<4>[60374.841019] [<f8bca83b>] kzalloc+0xb/0xd [target_core_mod]
<4>[60374.841019] [<f8bcce9a>] __generic_alloc_sg_list+0xa5/0x277 [target_core_mod]
<4>[60374.841019] [<f8bcbc65>] ? __tpc_get_rod_loc_by_rod_off+0x99/0x112 [target_core_mod]
<4>[60374.841019] [<f8bca147>] ? __tpc_is_token_invalid+0x71/0x94 [target_core_mod]
<4>[60374.841019] [<f8bc8677>] __do_write_by_token+0x3e3/0x952 [target_core_mod]
<4>[60374.841019] [<f8bc8bf8>] iblock_do_write_by_token+0x8/0xa [target_core_mod]
<4>[60374.841019] [<f8bc7f18>] tpc_write_by_token+0x31d/0x336 [target_core_mod]
<4>[60374.841019] [<f8bbac36>] __transport_execute_tasks+0xa8/0x108 [target_core_mod]
<4>[60374.841019] [<f8bbadb4>] transport_execute_tasks+0x11e/0x127 [target_core_mod]
<4>[60374.841019] [<f8bbadd9>] transport_generic_process_write+0x1c/0x1f [target_core_mod]
<4>[60374.841019] [<f8bbd7ff>] transport_processing_thread+0x14c/0x334 [target_core_mod]
<4>[60374.841019] [<c0437655>] ? wake_up_bit+0x1b/0x1b
<4>[60374.841019] [<c0437271>] kthread+0x5b/0x60
<4>[60374.841019] [<f8bbd6b3>] ? transport_kmap_data_sg+0xb1/0xb1 [target_core_mod]
<4>[60374.841019] [<c0437216>] ? kthread_bind+0x52/0x52
<4>[60374.841019] [<c087cebe>] kernel_thread_helper+0x6/0xd
<0>[60374.841019] Code: c8 b9 40 2f a6 c0 c1 e8 10 89 c2 89 d8 e8 db ff ff ff 5b 5d c3 55 89 e5 57 56 89 ce 53 83 ec 0c f7 c2 06 00 00 fe 89 45 e8 74 04 <0f> 0b eb fe 8b 45 e8 8b 40 1c 89 45 f0 a1 b4 bd a6 c0 25 f0 1e
<0>[60374.841019] EIP: [<c049c610>] new_slab+0x16/0x18d SS:ESP 0068:d3fbfcec
<4>[60374.841019] ---[ end trace e82880bdc69e3fbd ]---
User avatar
doktornotor
Ask me anything
Posts: 7472
Joined: Tue Apr 24, 2012 5:44 am

Re: iSCSI Freezes or Hangs

Post by doktornotor »

Contact QNAP support. Noone's gonna fix buggy kernel here.
I'm gone from this forum till QNAP stop wasting volunteers' time. Get help from QNAP helpdesk instead.
Warning: offensive signature and materials damaging QNAP reputation follow:
QNAP's FW security issues
QNAP's hardware compatibility list madness
QNAP's new logo competition
Dear QNAP, kindly fire your clueless incompetent forum "admin" And while at it, don't forget the webmaster!
User avatar
tpro
Starting out
Posts: 19
Joined: Tue Oct 26, 2010 11:15 pm
Location: Northern California

Re: iSCSI Freezes or Hangs

Post by tpro »

I've got the same exact problem too on a TS-559 Pro+ (4.0.3). I don't know it's the same thing detailed in the kernel related post, but the symptoms are the same as far as runs fine until heavy iSCSI traffic then either the iSCSI service stops responding and/or a reboot out of the blue.

I think the initial question would be a valuable one to answer as far as are there any tools/logs that can be consulted as far as the iSCSI service. There is just nothing to view status, control the service, etc. through the web interface and iSCSI is the core of our NAS usage.

Thanks!
User avatar
doktornotor
Ask me anything
Posts: 7472
Joined: Tue Apr 24, 2012 5:44 am

Re: iSCSI Freezes or Hangs

Post by doktornotor »

Once again: the only way to get support from community on kernel issues is to install Debian. Otherwise, head straight to QNAP helpdesk
I'm gone from this forum till QNAP stop wasting volunteers' time. Get help from QNAP helpdesk instead.
Warning: offensive signature and materials damaging QNAP reputation follow:
QNAP's FW security issues
QNAP's hardware compatibility list madness
QNAP's new logo competition
Dear QNAP, kindly fire your clueless incompetent forum "admin" And while at it, don't forget the webmaster!
User avatar
tpro
Starting out
Posts: 19
Joined: Tue Oct 26, 2010 11:15 pm
Location: Northern California

Re: iSCSI Freezes or Hangs

Post by tpro »

doktornotor wrote:Once again: the only way to get support from community on kernel issues is to install Debian. Otherwise, head straight to QNAP helpdesk
Once again, already did. My response was for the sake of other people out there that are researching what may be going on with their units (you know, the "community".)

Since no one has really provided and answer to the initial question still, I'll help also by sharing what I found in my little quest, actually being *helpful* (since this is a community forum, right?) Although, still no answer is really provided as far as the initial query of tools and or other troubleshooting methods.

What I did was SSH into the box then reviewed the kmsg logs residing in /mnt/HDA_ROOT/.logs to match my information to the other post *to even determine if it was a kernel issue*.

I've got several of these boxes and the only thing they are used for is iSCSI in a business environment. Unfortunately it's about the only thing that you can get no information on. Coming from a EMC and Dell Enterprise Storage education and background, while these "fit the budget" for the smaller company I now choose to work for, they are utterly lacking in insight into problems when using them in business environment compared to the more established vendors.

Oh, and sorry for letting others know they are not alone, I'll be sure to do it again.
User avatar
Jeddi
Starting out
Posts: 11
Joined: Thu Feb 13, 2014 3:08 pm

Re: iSCSI Freezes or Hangs

Post by Jeddi »

I'm seeing a similar problem. I'm on a 569L, running 4.0.5 firmware.

I had already disabled / uninstalled pretty much all the widgets and applets, and the appliance is exclusively serving iSCSI - 4 LUNs under 1 target, all used by a single Debian VM on an ESXi host. The connection between those two bits of hardware is a single and dedicated GbE CAT5. Just by copying data to the iSCSI LUNs, after usually between 5 or 15 minutes the QNAP reboots itself.

I experimented with Delayed Ack (ie disabling it) on ESXi. No change.

I reduced my MTU size back from 9000 to 1500, but no change from that, either.

I have set up NFS shares from the QNAP, to confirm that they are fine (and they are - I've just copied 150GB of data into the box over NFSv3 with no problem). This is particularly frustrating because one of the reasons I'm using iSCSI is to work around the QNAP's inability to enforce different permissions (RO .v. RW) for an NFS share.

I've raised a ticket with QNAP support, but given others are seeing similar things (it's helpful to have that visibility, isn't it?) and that it's looking like a firmware patch will be required to resolve it, that it could be 'some time' before I can safely use iSCSI on this device.
User avatar
tpro
Starting out
Posts: 19
Joined: Tue Oct 26, 2010 11:15 pm
Location: Northern California

Re: iSCSI Freezes or Hangs

Post by tpro »

I put in a ticket on this around the beginning of the month with no action whatsoever. Rather disappointed. Our work-around? "Don't move so much data" :( Makes me really miss the EMC stuff I worked with in the past.
User avatar
Jeddi
Starting out
Posts: 11
Joined: Thu Feb 13, 2014 3:08 pm

Re: iSCSI Freezes or Hangs

Post by Jeddi »

tpro wrote:"Don't move so much data"
Yeah, I started to think about ways I could throttle this at the iSCSI initiator end, but then realised that a) it'd be a lot of work, b) I really shouldn't have to do it, and c) even less frequent reboots would render the system unusable as I need to enter a decryption password on my VM's at boot.
:( Makes me really miss the EMC stuff I worked with in the past.
I suspect quite a few of us come from a background of working with medium to high end gear.

Anyway, for what it's worth, after tracking down a few other threads where people had indicated some cautious assessments of success by going to a newer firmware release, I also went down that path today - jumping from 4.0.5 up to 4.1.0. This is definitely beta software - I'm running the 2013-12-20 release.

So far I've banged on about 10 times as much data through my ESXi virtual machine talking to 4 x iSCSI LUNs on the QNAP as would previously have ensured the QNAP would just reboot. I'm now in the process of throwing 800GB over to it.
User avatar
Jeddi
Starting out
Posts: 11
Joined: Thu Feb 13, 2014 3:08 pm

Re: iSCSI Freezes or Hangs

Post by Jeddi »

An update on this - I've now copied 400+GB at 30MB/s, without incident, with the beta firmware.

Even though this compares to the previous two decades of expectations from my IT gear (that copying data won't result in random reboots) this can probably be considered a Great Success compared to my experiences over the past couple of weeks.

It's been 36 hours since I received the promise of a 24-hour response from the QNAP support team. No response has been received, natch.
User avatar
Jeddi
Starting out
Posts: 11
Joined: Thu Feb 13, 2014 3:08 pm

Re: iSCSI Freezes or Hangs

Post by Jeddi »

FYI - one month of QNAP 569L uptime. I've no idea if that beta firmware has other challenges relating to applets, but it's stable with very heavy iSCSI loads.
User avatar
tpro
Starting out
Posts: 19
Joined: Tue Oct 26, 2010 11:15 pm
Location: Northern California

Re: iSCSI Freezes or Hangs

Post by tpro »

So while they've never responded to anyone, perhaps they had resolved it in firmware. Thanks for the update.
Colindean2000
New here
Posts: 5
Joined: Sat Mar 15, 2014 5:47 am

Re: iSCSI Freezes or Hangs

Post by Colindean2000 »

We've got both a SS-EC2479U-SAS-RP and a TS-1079 Pro and been the victim of seemingly random reboots of both NAS boxes. The SS-EC2479U-SAS-RP is used in a 2012 R2 failover cluster and the 1079 Pro is used as storage for a hyper v replication server all connected to the same domain over 10gig Ethernet.

Both are running firmware 4.0.5 and the issue seems to present itself when there is a lot of replication data being received on either device so definitely matches the other cases here! I had no idea high IO could reboot a NAS!

Anyway because the SS-EC2479U-SAS-RP is less than a month old we are having it replaced as faulty but when the new unit arrives it's getting 4.1 which I am pretty unhappy about as it's only beta, I'd be much happier if it were the full release...

Thanks Jeddi you have given me the answer I was looking for, that 4.1 seems to be stable under high IO load.

Qnap seem to be totally unaware of this issue...
User avatar
Jeddi
Starting out
Posts: 11
Joined: Thu Feb 13, 2014 3:08 pm

Re: iSCSI Freezes or Hangs

Post by Jeddi »

tpro - did you file a bug with their support desk? I did, and I've been having email exchanges with a guy who's putting some effort into resolving the problem. My trouble is that I now have a functional appliance, so I can't really offer much - I've sent off system dumps / log bundles using their trouble-shooting app, but (perhaps not surprisingly) there isn't much in there that's insightful. Everyone with GNU/Linux sys-admin experience knows the joy of looking at logs that show the system was happy and healthy right up until the out of place [rsyslog]...start line.

Colindean2000 -- I'd recommend you file a bug. It'd be great if these guys could work with you to resolve the problem, and since you've got a failing box ... you're a good candidate. :) Actually the box you sent back, you could identify that s/n as I'd hope it'd be returned back to the manufacturer, not just the vendor for a re-issue.

My tracking number is #RAR-703-40121 - I'd suggest you refer to it and let them confirm that it's the same issue.

To be candid, I am really happy that someone's looking into this, though I'm worried that a) the bug got through QA in the first place, b) the bug is not present in the new (beta) firmware yet there's no clear indication, evidently, in their internal issue tracker or changelogs of what might have been the fix (ie. the problem was fixed accidentally), c) there's several people on the forums that have been hit by this bug but it sounds like these are not being investigated as the same issue within QNAP, and d) that they're unable to replicate this bug in their own labs, given that it evidently isn't constrained to a single model, or to a particular breed of iSCSI initiator (MS Windows, Debian, etc).

The big concern is that absent an understanding of the causal factor(s) by their team, there's no guarantee that the bug won't reappear in any future firmware version. It's not a good place to be - not wanting to upgrade your rc/beta box to a stable rev of the firmware in case it doesn't work.
Post Reply

Return to “iSCSI – Target & Virtual Disk”