TVS-h1288X performance problem

[docchris]

My impression is that yes I have to delete the pool and create it from scratch again

That will be fun ... moving a total of 700TB of data ...

[KurganZ]

I have no idea of what you guys are doing wrong. With a 10G connection from a Mac (with a QNA-T310G1T) or a PC (with a QXG-10G1T) I am getting amazing performance over 10G with
either the TVS-h1288X or the TVS-h1688X. I am not using any M.2 Caching - or any caching at all. This has nothing to do with QuTS 4.5.2. QuTS installed on two SSD drives as storage pool 1 -
all drives (either 8 drives or 12 drives) thin provisioned RAID 6 - and I get close to 1000 MB/sec. I have no idea of what you guys are doing wrong. It's so easy. I have done at least 10 of each of these
systems ( as well as a TS-h1683XU-RP) without issue.
I don't work for QNAP, nor do I sell any QNAP products. I just set these systems up for people.
Bob Zelin

So, you have one vdev per pool? I didn't go that way because I wanted the ability to upgrade only 6 of the 12 drives and still get an increase in size. Could be the issue, but that is based on conjecture, not so easy to test that theory.

[docchris]

I have one vdev per pool as I am not planning on upgrading for quite some time - and I still have the issue

[KurganZ]

after more remote access - the network part of the equation is a red herring.

the issue is purely that after a period of use the RAID6 array (8 HDD with NVME cache/ZIL) just slows to a virtual halt. this as harder to identify since the filestation GUI misreports the speed of copies anyway

(I am currently copying a file at 263 BYTES/sec - if it gets any slower it would be faster to copy the data by hand!!!)

support can see the array has ground to a halt (with 20TB of free space), and have confirmed that, but cannot tell why. they have now escalated it to a developer.

How did you get support to look at your ticket? Mine is about 24 hours old...with no updates. I'm thinking at this point I'll lose the weekend in waiting. SO frustrating. At this point, 20 minutes wasn't long enough for a controlled shut down. Read speeds are around that same 263 BYTES/sec you mention, like timeouts from most operations off the box. Even filestation copies to NVME storage will go nowhere for more than a minute or two for something trivial like a 250 MB file.

[docchris]

sounds EXACTLY like mine

yeah - mine wouldn't shutdown either, i had to ssh into it and manually kill the processes because it was just waiting for them to close

it took 48-72 hours for them to first look into it.

its amazing they managed to ship such an expensive system with such a big problem

[KurganZ]

True docchris. How could this happen? It can't be "size of storage pool" as it related to RAM because mine has to be about as small as anyones (~30TB of usable space in the largest pool (3 of 3))...and probably much more memory. The other two pools are a RAID of SSDs of 0.5 TB (Pool 1 for the OS), a 0.5 TB one disk NVME (pool 2 specifically for for VMs), and a single 1 TB SSD for caching. Like, if I'm seeing this speed, how can anyone have 10 Gb network saturation? It must be something more basic than the ratio of RAM to pool size, that just doesn't pass the sniff test. Now...IF it is number of FILES rather than space, THAT could potentially be true, but even then....I'm doubting it to be that absent some evidence. It did seem to work much better before the pool was 2/3 full (e.g. 20 TB), like the rsync job would have NEVER in a YEAR finished at the current speeds.

[docchris]

i think its probably just a memory leak somewhere (its certainly better now i have 80GB of RAM... but thats ridiculous when i had 16GB and it was 75% free...)

but it does seem odd that they are asking us to recreate the pool to fix it

i just wish they would hurry up and release it.

[docchris]

it will also be very interesting to see how this issue is described in the release notes

if it is not affecting everyone i wonder if there is some specific relatively minor difference that is triggering this

[KurganZ]

OK, so I MIGHT have figured out what causes performance to tank. It appears that if I disable Linux Station (LS) where I have Ubuntu 20.04 (? whatever the latest one is as an option) and reboot the NAS, performance can be > 1000 MB/s (read and write) from a Virtual Machine (Windows 10) to an SMB share (that is not SSD/NVME). If I leave Aja running and then allow LS to start, performance will drop to < 300 MB/s write and < 1 MB/s read (literally in just a few seconds). Shutting down LS at that point does nothing. You have to disable it, then restart, and you are back to the speeds I was hoping for originally. I have reproduced this 3 times, I have updated my ticket with some screenshots, and will work with them on Tuesday. I'm hoping to leave the machine up and see if performance stays a this level, or if LS is simply speeding up the inevitable. I hope that helps!

[KurganZ]

So, I can now reproduce this by just starting one Windows 10 VM, performance from off the box will plummet to <60 MB/s write and usually 0 MB/s read. CPU processes on the NAS skyrocket to 50+% for no discernible reason. The VM isn't directly causing this spike. You can see lots of (like 20 or more) 5-8% CPU usage pids (each) called things like "ddt_handle_sync" just soaking the NAS. The ~100 MB/sec read/write will resume once this VM is shut down. BTW, the VM only has 6 cores allocated of the 12 available, so the NAS should in theory have tons of headroom.

[rinthos]

So, I can now reproduce this by just starting one Windows 10 VM, performance from off the box will plummet to <60 MB/s write and usually 0 MB/s read. CPU processes on the NAS skyrocket to 50+% for no discernible reason. The VM isn't directly causing this spike. You can see lots of (like 20 or more) 5-8% CPU usage pids (each) called things like "ddt_handle_sync" just soaking the NAS. The ~100 MB/sec read/write will resume once this VM is shut down. BTW, the VM only has 6 cores allocated of the 12 available, so the NAS should in theory have tons of headroom.

@KurganZ ,

Just wondering if you were able to make any headroom w/ QNAP support on the cause and fix? I have a new TS-h886 and going to build it out, but if there's going to be a requirement to re-create the storage pools after deploying I'd rather hold off. Currently only seeing QTS Hero 4.5.1 versions (no 4.5.2 released yet apparently for Hero edition which sounds like it may have some sort of fix for this...?"

[docchris]

@KurganZ ,

Just wondering if you were able to make any headroom w/ QNAP support on the cause and fix? I have a new TS-h886 and going to build it out, but if there's going to be a requirement to re-create the storage pools after deploying I'd rather hold off. Currently only seeing QTS Hero 4.5.1 versions (no 4.5.2 released yet apparently for Hero edition which sounds like it may have some sort of fix for this...?"

Earlier this week I asked support when Hero 4.5.2 would be released, they said

I do not have any ETA for h4.5.2. I believe it should not be long, but i can't be sure.

They also confirmed that the storage pools would would need to be recreated to avoid this issue

[rinthos]

@KurganZ ,

Just wondering if you were able to make any headroom w/ QNAP support on the cause and fix? I have a new TS-h886 and going to build it out, but if there's going to be a requirement to re-create the storage pools after deploying I'd rather hold off. Currently only seeing QTS Hero 4.5.1 versions (no 4.5.2 released yet apparently for Hero edition which sounds like it may have some sort of fix for this...?"

Earlier this week I asked support when Hero 4.5.2 would be released, they said

I do not have any ETA for h4.5.2. I believe it should not be long, but i can't be sure.

They also confirmed that the storage pools would would need to be recreated to avoid this issue

Thanks docchris for sharing. Definitely ubfortunate but I guess I will postpone a deployment at this point. Not worth having to setup and risk this, and then having to recreate pools and move back data. Thanks again for the information. Would be nice if QNAP would update known issues for these sort of things vs.having to traverse the forums, but glad I stumbled upon this thread.

[KurganZ]

Soon I intend on testing out the "performance profile" that you declare when creating a shared folder. I have some evidence that this make a large difference in performance but I haven't as of yet made this a true apples to apples comparison. I need to put a copy of a set of files in shared folders of different sizes and see what (if any) difference that makes. I don't see a way to look at the decision I made when setting up the share(s), if anyone knows how to check, I would appreciate the tip!

[KurganZ]

I found where in the GUI it tells you...you have to look in "storage snapshots", this is not a setting (like compression) that you can change after the fact. I believe that 4K MAY be a contributing factor here...more to come.