Slow transfer speed due to SSD cache acceleration

[dolbyman]

@galosu82

there was a more elaborate thread..but I cannot find it right now

see this for some pointers
viewtopic.php?f=25&t=148414&start=15#p724949

AfroDieter also made some comments about the inner workings earlier in this topic
viewtopic.php?t=124852&start=225#p727574

[P3R]

Thunderbolt 3/4 connection + SSD cache are good options for video editing environments.

That's an odd statement to make after running a bunch of test programs but without doing any video editing...

Unfortunately, Thunderbolt connection doesn't always perform the same (on Macs it is slower than 10 GbE)...

So the recommendation to run 10 GbE on your Macs was correct.

...apparently, SSD cache performance improvement becomes less noticeable when the SSDs are full.

Maybe this is also why real world experience of video editing have shown better results without SSD caching??! But since you only have six spinning disks maybe a very frequently flushed cache to some extent can compensate for that perfomance bottleneck?

[galosu82]

Thunderbolt 3/4 connection + SSD cache are good options for video editing environments.

That's an odd statement to make after running a bunch of test programs but without doing any video editing...

Unfortunately, Thunderbolt connection doesn't always perform the same (on Macs it is slower than 10 GbE)...

So the recommendation to run 10 GbE on your Macs was correct.

...apparently, SSD cache performance improvement becomes less noticeable when the SSDs are full.

Maybe this is also why real world experience of video editing have shown better results without SSD caching??! But since you only have six spinning disks maybe a very frequently flushed cache to some extent can compensate for that perfomance bottleneck?

Hello P3R, what other tests I could do other than the tests that someone (Blackmagic / AJA) designed for video making environment? I've no idea if there is any other reliable test that could make sense to do.

[P3R]

Hello P3R, what other tests I could do other than the tests that someone (Blackmagic / AJA) designed for video making environment?

I see no point with more test applications. Performance tests will only give you indications and part of the answer and this apply to all usages. The only way to know for sure how a system will perform for a certain usage is to get your feet wet by having actual users working on it and evaluate that. Luckily the cache is easy to switch on and off so to do your own comparable tests for exactly your specific usage should be easy to do.

[Furyous]

After running for a few hours, the SSD cache is impairing performance again. I ended up turning it off.

Cache turned on:

With Cache.PNG

Cache turned off:

Without Cache.PNG

[galosu82]

After running for a few hours, the SSD cache is impairing performance again. I ended up turning it off.

Cache turned on:
With Cache.PNG

Cache turned off:
Without Cache.PNG

Hello Furyous, have you tried using the settings proposed by cryptochrome (viewtopic.php?p=800080#p800080)? What speed were you getting when the cache was empty? How are you connected to the NAS?

[HiTach]

Any differences seen with cache behavior after using QTS 5.0?

[Furyous]

Hello Furyous, have you tried using the settings proposed by cryptochrome (viewtopic.php?p=800080#p800080)?

I did, my original configuration matched that.

What speed were you getting when the cache was empty?

It worked well when I first made the cache but when I tested later (I assumed the cache filled) it was erratic throughput

How are you connected to the NAS?

NAS and Computer are both connected to the same switch.

[holger_kuehn]

How are you connected to the NAS?

NAS and Computer are both connected to the same switch.

So it is via SMB, but is it 1GbE, 2,5 GbE or 10 GbE? Or something different at all ....

[Furyous]

How are you connected to the NAS?

NAS and Computer are both connected to the same switch.

So it is via SMB, but is it 1GbE, 2,5 GbE or 10 GbE? Or something different at all ....

I am not sure why it matters as I get full, consistent speed with cache turned off (it has been turned off for weeks now and no issues) but here it is:

Port Config.PNG

Port Stats.PNG

[Furyous]

BTW I have tested extensively without port channel including direct connect before I found this thread and fixed the issue by turning off cache. I was ready to change to another NAS manufacturer out of frustration.

[s_clarke]

Hi All - some really good info on this thread which has helped me get to the bottom of my issue. I've been struggling to find the problem, QNAP support have been helpful and have been trying to look into the issues - but SSD Caching is certainly where my issues lie.

The strange thing is that I didn't experience these issues with my pre-RMA device (TVS-672n) and older firmware (prior 5.0). So for me this has only been felt on the TVS-672x and 5.0 and above.

One result stood out for me when running an internal file copy on the NAS via this command dd if=/dev/zero of=/share/CACHEDEV1_DATA/Public/test10GB bs=1M count=10240

SSD Cache enabled:

10737418240 bytes (10.0GB) copied, 284.18582 seconds, 30.2MB/s

SSD Cache disabled:

10737418240 bytes (10.0GB) copied, 60.167352 seconds, 170.2MB/s

That is such a huge difference and really does illustrate the issues the caching is causing. I've fed all of this back to QNAP in the hope that they can finally discover a flaw in how it's implemented, like others I'm also seeing the allocated cache stick at 99% and never flush (which is probably the problem).

As a workaround, which I'm happy with, I've removed SSD Caching and have created a single RAID Volume for my SSD's which I'm using for my live data dumps - in essence it's giving me a fast cache for my live data, with a little manual process run later to copy it to HDD. I'm happy with that but hopefully QNAP eventually acknowledge the flawed implementation of SSD Caching on their devices.

[cryptochrome]

Nice find! Don't get your hopes up, though. QNAP support has been fed proper evidence of the issue in countless tickets over the years, from many customers. I doubt this will change anything.

[drooplewx]

Hi Guys,
I was thinking of purchasing a QM2 PCIe extension card and 2 1TB M.2 SSD for my TS-253d, I think I can save the money for something else now.

[cryptochrome]

On a rather small NAS like that, you could look into Qtier, where the NAS automatically moves the most used data to the fast SSD and the lesser used data to slow HDDs. I don't think Qtier has the same problems as the cache. In fact, accessing SSD pools over the network is freaking fast. It's just the caching that's borked.