Slow transfer speed due to SSD cache acceleration

Questions about SNMP, Power, System, Logs, disk, & RAID.
Post Reply
andrewket
New here
Posts: 9
Joined: Sun Sep 05, 2010 5:08 am

Re: Slow transfer speed due to SSD cache acceleration

Post by andrewket » Sun Mar 05, 2017 9:09 am

Bumping this thread. Have any of you with open tickets found a solution other than disabling the cache?

pokrakam
Starting out
Posts: 43
Joined: Wed Sep 17, 2008 5:27 am

Re: Slow transfer speed due to SSD cache acceleration

Post by pokrakam » Sun Mar 05, 2017 6:09 pm

In my case they have either had a change of heart or decided my setup is one of the more suitable for testing, as they have been on it numerous times.

Still ongoing, and from what I've seen it's a tricky one. The cache performs well until it becomes tired (yes, that's my technical term for it!). It has become clear that deterioration happens with extended usage.

So my temporary solution is that, even though part of the intended usage is for larger data files, I've set it to only cache a small block size. The cache stays at under 50% utilised during normal usage, and it speeds up VM performance to a decent level. So am getting a partial benefit out of it for now.

cryptochrome
Been there, done that
Posts: 545
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome » Sun Mar 05, 2017 6:45 pm

I can confirm these observations. The system comes to a grind when the write cache is full (or heavily utilized) and it starts flushing contents to disk. During that time it becomes extremely slow. Unfortunately this only happens when you write more data to the cache. So if you copy a huge amount of new data to the NAS and the cache starts flushing out, latency for the cache increases to more than 20 seconds (!) sometimes.

Making the cache block size smaller helps tremendously, but unfortunately not always. Prime example: Torrents. Say you add 3-4 large torrents that are well seeded and you download with high speeds (150 mb/sec+), it will produce a massive amount of small blocks being written to disk, and that will bring the cache to its knees as well.

@pokaram: What are they doing in your case? Did they acknowledge it's a problem and they will work on it?

Thank you!

-hitman-
Starting out
Posts: 13
Joined: Fri Dec 30, 2016 6:14 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by -hitman- » Sun Mar 05, 2017 10:04 pm

Qnap support told me that SSD cache is only beneficial in certain disk usage scenario's and I kind of agree, I did try some settings and found using a small block size (/other than default) as mentioned helped a lot and transfer speed - client to NAS became max HW speed, however turning off cache gave the same results as noted earlier in thread.

I have since ditched the cache drives in favour of being able to add more storage into the bays instead of Cached SSD's and enjoy full speed without - my personal NAS usage doesn't need cache.

pokrakam
Starting out
Posts: 43
Joined: Wed Sep 17, 2008 5:27 am

Re: Slow transfer speed due to SSD cache acceleration

Post by pokrakam » Mon Mar 06, 2017 4:54 am

They have been on the system several times over the last six weeks, have acknowledged that it's not performing as it should, but no statement on whether it's a 'support' issue (i.e. QNAP fault) or not. To be fair to them, this is not all that clear.

I am speculating that they picked up my case again because the issue may also happen with other 'supported' SSDs/systems, and/or possibly my setup may be easy to work with as I have a reliably reproducible test case that is local to the NAS.

The last session was the previous Friday, just over a week ago, and they are looking at the results. We had a session with a newly formatted cache and it flew throughout the hour or so they were on there. Then they were back on the system a week later when the cache was over 70% full and we witnessed the crawl.

I could live with it if it would slow down to HDD speeds, but it goes way slower than that. My test run takes on average 1:20 uncached. Cached I'm looking at 30-40s. Once it gets tired it can take 2:30 or more.

@hitman - yes, it is very much usage dependent and won't help everyone. We have quite a mixed bag of use cases: databases and virtual machines are very much suited to small block sizes as we're accessing small portions of a multi-GB file. Working with uncompressed video footage uses anything from 5-20MB file for each frame, so that's a lot of large block accesses. In both cases the SSD cache helps a lot. With more than one user simultaneously generating load, latency also becomes a bigger factor - which is where an SSD really comes into it's own.
I saw the least benefits as a single user doing regular file storage type activities such as reading and copying stuff over the net (98%-112% of uncached performance on my simple tests over GB Ethernet).

cryptochrome
Been there, done that
Posts: 545
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome » Mon Mar 06, 2017 5:11 pm

I can certainly agree that a cache is not beneficial to every use case, but come on, bringing the whole system to a grinding halt because the latency for disk access increases to 20 freaking seconds (not milliseconds) is just not right and obviously a bug. And it's been going since the release of the feature. I am going to be blunt here, they should get their act together for a change.

User avatar
storageman
Ask me anything
Posts: 5505
Joined: Thu Sep 22, 2011 10:57 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by storageman » Mon Mar 06, 2017 6:23 pm

I would like to know if there are still issues using SAS or DC type SSDs on the SAS based Qnaps.
Part of this issue also relates to the performance of desktop based SSDs.

scotland21
Know my way around
Posts: 103
Joined: Sun Feb 28, 2010 10:21 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by scotland21 » Sat Apr 15, 2017 6:32 am

Is there any further news on this, I am currently experiencing the same type issues when backing up data between my various nas, all qnap devices sitting at 33 MB\S transfer rate.
Turn the cache off now hitting 65-107 MB\S.
TS-509pro 4.2.4, TS-809pro 4.2.4, TS-459II 4.2.4,TS-559II 4.2.4
TVS-663 4.3.3.0095, TS-853pro 4.3.3.0095, TVS-871 4.3.3.0095
Scotland21 8)

cryptochrome
Been there, done that
Posts: 545
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome » Sat Apr 15, 2017 5:37 pm

No news, because QNAP never acknowledged the bug (said they can't reproduce it), e.g. no one from them is even looking at it.

scotland21
Know my way around
Posts: 103
Joined: Sun Feb 28, 2010 10:21 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by scotland21 » Sat Apr 15, 2017 6:43 pm

pokrakam posts suggest that Qnap are looking at this?
If they aren't I wouldn't have asked for an update.
Confused :?
TS-509pro 4.2.4, TS-809pro 4.2.4, TS-459II 4.2.4,TS-559II 4.2.4
TVS-663 4.3.3.0095, TS-853pro 4.3.3.0095, TVS-871 4.3.3.0095
Scotland21 8)

pokrakam
Starting out
Posts: 43
Joined: Wed Sep 17, 2008 5:27 am

Re: Slow transfer speed due to SSD cache acceleration

Post by pokrakam » Sun Apr 16, 2017 12:12 am

Qnap are still looking at my setup, just did some more tests last week.

scotland21
Know my way around
Posts: 103
Joined: Sun Feb 28, 2010 10:21 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by scotland21 » Sun Apr 16, 2017 12:57 am

Thanks pokrakam for confirming this is still being investigated by Qnap, the cost of SSD's to effectively slow down the nas is really frustrating so I hope they fix it soon.
TS-509pro 4.2.4, TS-809pro 4.2.4, TS-459II 4.2.4,TS-559II 4.2.4
TVS-663 4.3.3.0095, TS-853pro 4.3.3.0095, TVS-871 4.3.3.0095
Scotland21 8)

peter247
Getting the hang of things
Posts: 60
Joined: Sun Apr 16, 2017 5:55 am

Re: Slow transfer speed due to SSD cache acceleration

Post by peter247 » Sun Apr 16, 2017 11:19 pm

I did a migration from raid 1 to raid 5 ( with a 1tb ssd cache drive ) and started filling my nas up and found my nas came to a total stop .
Looking at system status the latency was about 600-800,000ms mark and only 20-30 iops , disabled and removed the cache and it went to a latency of 20ms and 300 iops.
So the ssd cache didn`t work for me.

pokrakam
Starting out
Posts: 43
Joined: Wed Sep 17, 2008 5:27 am

Re: Slow transfer speed due to SSD cache acceleration

Post by pokrakam » Mon Apr 17, 2017 12:55 am

Matches my observed behaviour. However it is only under sustained heavy load that it crumbles. Migrating a lot of data is exactly that type of scenario. Some of my observations:
- If I run Virtual Machines off the NAS (sporadic random access) and little other usage, it performs very well.
- If the system has had very light load for a few days, I can run a good 50-100GB worth of heavy load testing before it slows down
- When it's tired it can slow down after only 10GB of sustained load
This would still perform well for the average file server or VM application.

I also see increased latency when it struggles, typically in the 2000-5000ms region, and <200 IOPS. Normal performance when the SSD cache is working well is closer to 20-30ms latency and >400 IOPS.

scotland21
Know my way around
Posts: 103
Joined: Sun Feb 28, 2010 10:21 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by scotland21 » Mon Apr 17, 2017 4:49 am

I have just carried out a similar test with a 60GB file and concur with both your findings, copying to my pc is fine but copying back to the nas it starts off ok but then drops significantly.
Latency hits 20830 with a iops of approx. 72, turn off the cache mid transfer and I see the latency drop to 2200 and the iops go up to 240.
SSD's are not cheap and to have them bring a nas to it's knees when using the feature to if's full potential is a major flaw.
I think Qnap should acknowledge on this forum that they are aware of the issue and are working on it as clearly people on this forum aren't fully aware.
However I suspect they won't as it's not a good advert for their firmware and certainly won't win them any sales.
TS-509pro 4.2.4, TS-809pro 4.2.4, TS-459II 4.2.4,TS-559II 4.2.4
TVS-663 4.3.3.0095, TS-853pro 4.3.3.0095, TVS-871 4.3.3.0095
Scotland21 8)

Post Reply

Return to “System & Disk Volume Management”