Slow transfer speed due to SSD cache acceleration

[scotland21]

I have logged my own ticket, Qnap need to address this issue as they make big claims as to it's performance gains.
Using the SSD for read only is not an option, I have spent money to enable me to use read\write, this runs into hundreds of pounds as I have 4 compatible units all setup with SSD cache.

[amigoccs]

Hi,

Is there anything to do with trim?

I have studied using SSD in NAS but give up eventually in How To Get wiper.sh Running On QNAP Storage Devices To TRIM SSDs?. Just cannot recompile the kernel to enable trim.

Which kind of SSD are you using now? TLC, MLC, or SLC? Does trim enabled in SSD cache?

[pokrakam]

I had my suspicions there too and also mentioned it to QNAP, but the behaviour is not consistent with something related to trim. I won't rule it out completely though.

Mine is a Crucial MX300, TLC, flashed to latest firmware, trim scheduled at 4am daily. But a few people with different SSD makes are seeing the issue.

Even after a secure erase, the issue comes up well before it hits even 50% full. Also, the performance recovers again within an hour, yet TRIM is scheduled once a day.

And when it is 100% full and should thus be hitting write amplification straight away, performance is still pretty good.

Also, the slowdown can be very dramatic. I could live with it if it dropped to HDD speeds but it drops far lower than that in some instances.
Even if it hits theoretical maximum write amplification, the SSD should at least maintain a HDD-like level of performance without trim.
However, to give one real world example: I started a tar archive job, it took 36 minutes to archive the first 42GB of data. I cancelled it, switched off the cache, restarted. It ran through 230GB in 38 minutes. Them SSDs are a very expensive handbrake.

On the other hand, with SSDs the performance is quite decent for running a VM stored on the NAS. Completely unusable without cache.

[scotland21]

I have confirmation of my support ticket which has been passed to Qnap's second line.
I know there are those on here who have little faith, however another ticket they have to add to all the others telling them one of their key features is seriously broken.

[amigoccs]

Hi pokrakam,

> Even after a secure erase, the issue comes up well before it hits even 50% full.

Maybe it's already full due to write amplifier?

http://www.anandtech.com/show/2829/4

> Also, the performance recovers again within an hour, yet TRIM is scheduled once a day.

Maybe it's the Pseudo-SLC flushing to TLC? One it's empty, Pseudo-SLC are ready to write again.

> And when it is 100% full and should thus be hitting write amplification straight away, performance is still pretty good.

Very strange. Can we verify SSD with any other tools? I am not sure if we should trust these graphical monitors. Because I used to notice the "Resource Monitor" in dashboard gadget is showing differently to "Resource Monitor" from "control Panel"

> Even if it hits theoretical maximum write amplification, the SSD should at least maintain a HDD-like level of performance without trim.

Yes, I think the SSD chipset will handle and CPU doesn't need to wait.

> However, to give one real world example: I started a tar archive job, it took 36 minutes to archive the first 42GB of data. I cancelled it, switched off the cache, restarted.
> It ran through 230GB in 38 minutes. Them SSDs are a very expensive handbrake.

Very useful information. I never expect SSD actually slow down disk I/O.

> On the other hand, with SSDs the performance is quite decent for running a VM stored on the NAS. Completely unusable without cache.

That's what I need: use SSD cache to accelerate Virtualization Station. I use another NAS for storage.

Thanks for your sharing!

[pokrakam]

Maybe it's already full due to write amplifier?

No, that was the point behind a secure erase. In SSDs this will perform the erase cycle, giving you an equivalent of a brand new SSD ready to be written to. I get great performance for maybe the first 50-100GB of sustained load (500GB SSD), then it gradually caves in.

Maybe it's the Pseudo-SLC flushing to TLC? One it's empty, Pseudo-SLC are ready to write again.

That was also some of my thinking, but it takes an awful long time to do this. I would expect minutes at most, but I really have to leave it to rest for an hour or more to get back to normal.

Very strange. Can we verify SSD with any other tools? I am not sure if we should trust these graphical monitors. Because I used to notice the "Resource Monitor" in dashboard gadget is showing differently to "Resource Monitor" from "control Panel"

I don't use resource monitor for my tests, I use the linux time command to monitor a tar operation between volumes on the same NAS (no network).

[okenny]

was just about to order some new SSDs - glad I read this thread... thanks all.
So read cache is fine? Problems only occur with a write cache? Isn't a read cache already pretty useful?

Would QTier be a reasonable backup option for SSDs? It shoud still improve peformance somewhat, right?

[scotland21]

QTier contains regularly accessed data so for everyday use it does help performance, however this is stored data which it moves to the SSD and plays no part with the transfer of data.
The write cache issue is a biggie, If you're transferring small amounts of data MB's then it's not a major issue.
When you start transferring GB's of data it hits the skids and slows down the entire system.
To enable the write cache you need 2 SSD's so the investment financially has been made to enable what should be a big performance gain for data transfer.
It's like putting a V8 engine in a tiny car and trying to drive with the brakes on.

I've not heard from Qnap since they confirmed my ticket has been logged.

[hakannordell]

.....So read cache is fine? Problems only occur with a write cache? .....

My picture from Wed May 03, is with only read cache enabled.

[RobbieG]

I will add my name to the ring of seeing this problem (TVS-873, Cache acceleration with two MX300 (ATA Crucial_CT525MX300SSD4), M.2 SATA, RAID 1, Read/Write)
I saw this performance issue when I set it to sequential.
Changing to random with a 1 MB bypass block I do not see this issue (although, maybe I am essentially "disabling" cache with these settings for large transfers).
I too am wondering if I wasted money on purchasing drives for cache "acceleration".

[cryptochrome]

It's true, the cache problem is most apparent when disabling cache block size (e.g. set it to sequential). The more reduce the cache block size, the less of a problem you have. However, the problem does not go away. Wait until you have some IO heavy and data heavy transfers and the cache gets to flush state (e.g. it starts dumping data to hard drives). This usually starts to happen at around the cache fill mark of 30-40%. Once the cache starts flushing while you still have data going into the cache (above mentioned IO or heavy data operations), your system comes to a grinding halt, regardless of caching block size.

[scotland21]

This is my reply received for my support ticket:
Hi John,

What size of file do you test ?
Because if you test with files which size is bigger than 10MB, it won't be faster, which ssd is good at read/write with smaller files.
And the ssd cache function has a learning process, it won't bring into efficacy right away.

I haven't seen anything in any promotional or support documentation to explain any limitations or to suggest how the cache must be setup.
I'm dismayed and disappointing with this response as I use cache because I transfer large files.

[cryptochrome]

This is the same first level "we read default questions to customers" response I received when I opened my ticket months ago. I tried to explain to them but they were unable to even remotely comprehend the issue at hand. Their entire level 1 support staff is a joke, and a waste of time.

Of course the cache will have the most benefit on small blocks and random read/write operations, that doesn't mean though that large files cannot be cached. And it definitely does not mean that caching them causes the whole disk system to run into twenty-thousand millisecond latencies. I mean seriously, the cache is waiting 20 freaking seconds before it allows the next read/write operation. And QNAP support have failed for months now to get this through their empty little heads.

This thread? Do they even read this? I am massively fed up with this BS.

And at the same time they keep introdicing new feature after new feature, all of it never leaves beta, and instead of finishing any of them or fixing bugs like this that have been known for a year now they introduce yet another feature nobody wants.

F U QNAP

[RobbieG]

It annoys me too, and I hope they can at least intelligently and directly address the issue.

My knowledge of all of this is pretty minimal, but is what you're hoping the cache will do, what it is actually supposed to do (I originally thought this would speed up transfers to and from the NAS)?

Or is it not reasonable to expect it to speed up transfers, and it's real purpose is for supporting operations within the NAS (and would this be a lesser benefit to just getting more RAM)?

[cryptochrome]

It should speed up all disk operations, regardless of where the data is coming from. Of course it can't do much for a 1 gbit ethernet connection, but if you are working through Thunderbolt or 10 GbE it will (should) give you massive speedups even over the network. And that's how they actually advertise it. And still do. On all the models, they talks about SSD cache everywhere. A smart lawyer could probably construct a case of false advertising out of this whole mess.