Deleting many files from iSCSI drive hangs and blocks writes...

[kuhndj]

Having an odd problem and I've done everything I can think to do on the WSS side:

System -
Storage: TVS1282-T3 (firmware 4.3.2.0141 B20170326), core i7-7700, 64GB with 32TB of Samsung SSD (8x4TB) storage (no cache assigned), one iSCSI LUN ~25TB.
Server: Windows Storage Server 2016 Std (HPE, Xeon Silver 4110, 32GB Ram)

Symptom - Deleting folders with about 1000 files (Tif, 15MB each) experiences frequent pauses or hangs... If I've not deleted a folder in some time then the folders will delete quickly (few seconds), but if I have just deleted one folder then often the second will get partway through the delete and then stop for some arbitrary amount of time (from 10-20 seconds up to 5 minutes). That by itself is just a bit annoying... but the problem is that while that folder delete is hung up all other write or delete operations also hang. (read operations continue normally). after the delay the delete will continue for another couple hundred files than hang again and again until finished.
While the delete is hung up, CPU utilization by the system process is locked at 7% (about one core) and the task using most of that time is IOGetIOPriorityHint.

Writing thousands of files at once doesn't seem to cause this issue... just deleting them.

I have a screen vid of the issue but your attachments are too size limited to attach (10mb).

[kuhndj]

Also... the volume currently has about 6TB free (25% or so).

The only thing I see on the QNAP performance metrics that looks odd is that latency often gets very long (over 500ms). Processor and memory utilization are both low...

I'm attaching two images showing the File Controller status while this issue is happening...

in the first image you can see activity on two 10gbe interfaces (the top one is iSCSI, bottom is a 10GBe production network). I added a red line where a delete operation started (the delete operation doesn't use enough bandwidth to be visible) and immediately blocks all write activity on the iscsi interface... you can see on the lower interface that there's a ~60MB/s write operation that continues but those writes are not making it to the QNAP.

The second image shows things about 20-30 seconds later when the 60MB/s write completes and some time later the iSCSI interface becomes responsive again and the buffered up data is burst written to the QNAP.

FCDiag1.jpg

FCDiag2.jpg

[dolbyman]

best to open a ticket with qnap..don't think we can do much about it here

[storageman]

What kind of LUN is it?
Thick or thin?
For performance always use thick (instant allocation).