Page 1 of 25

Slow transfer speed due to SSD cache acceleration

Posted: Sun Jul 31, 2016 1:04 am
by Paul Hermans
I hope this is the right forum for my problem.
I have a problem with transferring large files and more specific in one direction. When sending such a file from a PC via SMB to the NAS, I have transfer speeds between 110 and 120 MB/s. On the other hand when transferring the same file from the NAS to the PC the transfer speeds are between 15 and 30 MB/S. The transfer speed fluctuated also a lot. In the beginning I thought it was a network problem but after doing all possible tests I came to the conclusion that the problem was with the NAS. So I started experimenting with all kinds of settings and finally I disabled the SSD cache acceleration. The reason for this was that the disk performance was very poor at around 400 IOPS. Suddenly the transfer speed when copying from the NAS to the PC went up to between 110 and 120 MB/S. The same speed than in the other direction. I saw the same result with the disk performance which went up to over 4000 IOPS.

My question now is why the SSD cache acceleration slows the transfer down so much and what is the right way to use the cache acceleration? There is also a setting: Bypass Block Size. What does that do?

The files being transferred are between 20 an 50 GB.

My configuration is:

Host: TVS-EC1080+ with 10 x WD Gold 6TB disks (RAID6) + 2 x 128 GB SSD (RAID0)
Expansion unit: REXP-1000 Pro with 10 x WD Gold 6TB disks (RAID6)
Firmware: 4.2.1 Build 20160601
Network: Dynamic Link Aggregation on 4 interfaces (IEEE 802.3ad)
SMB version: 2.1 (because there still a dedicated WIndows 7 PC's on the network)
Switch: 2 x Cisco SG500X-24 Stackable Managed switch
Router: Cisco 2951
Cables: Cat 6
Workstation: HP Envy Phoenix with i7-3770K 3.5GHz quad core, 32GB RAM and Qualcomm Atheros AR8161 NIC (Other PC's are in the same order)
OS: Windows 10 Pro

Re: Slow transfer speed due to SSD cache acceleration

Posted: Mon Aug 01, 2016 6:49 pm
by storageman
That's not good
If RAID 0 your cache was read only. I assume setting it read/write makes no difference.
The cache mainly benefits random read/writes not sequential, your files are especially big.
Still such negative effect on read performance is bad.
Not sure about Bypass Block size but for big files you want a big block size.

Re: Slow transfer speed due to SSD cache acceleration

Posted: Wed Aug 24, 2016 9:43 am
by cryptochrome
I have just come across the same problem. In my case, I was transferring huge amounts of data from one NAS to another. I had SSD cache enabled with the Bypass Block Size disabled (e.g., cache everything, even sequential operations on large files). The performance was miserable, data was only moving at around 30 MB/sec over a 10 Gig Ethernet connection. Latency in the Disk Performance dashboard reached peaks of 40.000 (!) ms and was averaging between 10.000 and 20.000 ms (thats 10-20 seconds of latency for IOPS). Crazy!

The second I changed the Bypass Block Size down to 4 MB: Boom. Latency dropped to 200-400 ms and transfer speeds went up 5-fold.

I am not really sure yet what exactly is going on. My bet is there is an issue with the SSD caching policy. Either it's bugged, or the caching of sequential (large size) operations simply doesn't work. Although the manuals for dm-cache (lvmcache, used by QNAP) say it's perfectly fine to cache large blocks (e.g. disable the cache bypass).

I opened a ticket with QNAP support. Not expecting much there, but hey...

Re: RE: Re: Slow transfer speed due to SSD cache acceleration

Posted: Sun Aug 28, 2016 9:55 pm
by peelos
cryptochrome wrote:I have just come across the same problem. In my case, I was transferring huge amounts of data from one NAS to another. I had SSD cache enabled with the Bypass Block Size disabled (e.g., cache everything, even sequential operations on large files). The performance was miserable, data was only moving at around 30 MB/sec over a 10 Gig Ethernet connection. Latency in the Disk Performance dashboard reached peaks of 40.000 (!) ms and was averaging between 10.000 and 20.000 ms (thats 10-20 seconds of latency for IOPS). Crazy!

The second I changed the Bypass Block Size down to 4 MB: Boom. Latency dropped to 200-400 ms and transfer speeds went up 5-fold.

I am not really sure yet what exactly is going on. My bet is there is an issue with the SSD caching policy. Either it's bugged, or the caching of sequential (large size) operations simply doesn't work. Although the manuals for dm-cache (lvmcache, used by QNAP) say it's perfectly fine to cache large blocks (e.g. disable the cache bypass).

I opened a ticket with QNAP support. Not expecting much there, but hey...
Did you have any luck troubleshooting this further crypto? With 1gb Ethernet I have miserable 20mbps transfer speeds even if bypassing SSD cache block size of 32mb..

Re: Slow transfer speed due to SSD cache acceleration

Posted: Sun Aug 28, 2016 10:42 pm
by cryptochrome
Not really, peelos. 32 MB is too much. I get much better results with 4 MB. Slightly worse with 8 MB and anything above that is just terrible. The default (if I remember correctly) is 1 MB.

Not sure why this is happening, as dm-cache (that's what QNAP use in QTS) should handle even large sequential writes with much bigger chunk sizes perfectly well. Whether it makes sense to do that is another topic. My guess is this could be a bug in either dm-cache or the implementation in QTS. I have a ticket open with QNAP, and except for an initial response where they seemed to have completely misunderstood the problem, I haven't heard back in 7 or 8 days now.

Re: Slow transfer speed due to SSD cache acceleration

Posted: Mon Aug 29, 2016 12:13 am
by peelos
4mb chunks seem to have helped get to 100mbps transfers using AFP.. Not sure the ssd cache helps at all with 1Gbps ethernet, will have to test it properly when I sort everything else out with the NAS

Re: Slow transfer speed due to SSD cache acceleration

Posted: Mon Aug 29, 2016 3:23 am
by cryptochrome
I think the cache is primarily designed to cache local actions and not so much to act as a network buffer when receiving data. That's my conclusion of my observations so far. What's interesting is that someone in the large TVS-x82 thread mentioned a few weeks back that he saw a special network cache feature in the 4.2.2 beta, but I have yet to find that in the released code.

It's just too bad that QNAP don't document their features properly. All they have is "click here to enabled this and that" but no real technical background information. If you try to get such information from their support, they usually fail to deliver.

Anyway. I will keep digging into this. I am reading up alot about dm-cache and how it works.

Re: Slow transfer speed due to SSD cache acceleration

Posted: Tue Nov 29, 2016 11:14 pm
by nikrampi
Hi all, sorry to resume old thread but I find myself in same situation as described by cryptochrome and peelos and I've reached this thread searching solution for SSD cache config.
cryptochrome wrote:Not really, peelos. 32 MB is too much. I get much better results with 4 MB. Slightly worse with 8 MB and anything above that is just terrible. The default (if I remember correctly) is 1 MB.

Not sure why this is happening, as dm-cache (that's what QNAP use in QTS) should handle even large sequential writes with much bigger chunk sizes perfectly well. Whether it makes sense to do that is another topic. My guess is this could be a bug in either dm-cache or the implementation in QTS. I have a ticket open with QNAP, and except for an initial response where they seemed to have completely misunderstood the problem, I haven't heard back in 7 or 8 days now.
Can I ask if you've got any insight from the support? I'm wondering if I should open a ticket too on this same issue.
cryptochrome wrote:What's interesting is that someone in the large TVS-x82 thread mentioned a few weeks back that he saw a special network cache feature in the 4.2.2 beta, but I have yet to find that in the released code.
I could not find any reference on this point on the changelog now that the firmware is out, I'll probably give a try in the next few weeks just to see if there are improvements on this issue.
Has anybody tried to tinker with SSD caching settings on latest firmware?

Thanks again for any support. I'll share my improvements if I have any luck in making SSD caching large chunks of data.

Re: Slow transfer speed due to SSD cache acceleration

Posted: Tue Nov 29, 2016 11:21 pm
by cryptochrome
Unfortunately QNAP support had zero clue about what I was talking about and after weeks of going back and forth with them, they basically said "we can't reproduce the issue, so there is no issue".

I have since reduced the cache block size to 1 MB which greatly reduces the issue but does not entirely remove it. I would love to crank this up so that larger file operations (which are frequent in my use case) could benefit, and I hope that soon to be released 4.3 improves on this (maybe through update dm-cache/lvm components) but I don't have high hopes.

Re: Slow transfer speed due to SSD cache acceleration

Posted: Thu Dec 01, 2016 6:01 am
by sqparnell
I had same issues - over Thunderbolt, was getting really slow and sporadic speeds, averaging around 70 MB/s read and write. Turned off SSD Cache and instantly saw steady performance - 800 MB/s download and 530 MB/s upload. Thanks!

Re: Slow transfer speed due to SSD cache acceleration

Posted: Thu Dec 01, 2016 6:26 am
by Don
If more people open tickets then they will see it is not an isolated issue.

Re: Slow transfer speed due to SSD cache acceleration

Posted: Thu Dec 01, 2016 6:49 pm
by cryptochrome
Oh I pointed them to another thread where some people reported they had the issue (the large TVSx82 thread), but that didn't stop them from saying the can't reproduce it. But yes, please, everyone open tickets. The more, the better.

I have invested a boatload of money into SSDs just for caching and it just doesn't work.

Re: Slow transfer speed due to SSD cache acceleration

Posted: Tue Dec 13, 2016 9:08 am
by vitamind
It's been driving me crazy for the last couple of months. I upgraded to Windows 10 on both PCs and figured that was the reason behind the now slow speeds. I've tried everything on so many forums and never even thought it could be the server. Sure enough I turned Cache Acceleration off and instantly my speeds increased to what they use to be. I'll start a ticket on this issue as well. Thanks.

TVS-EC1080
4.2.2 Build 20161102

Re: Slow transfer speed due to SSD cache acceleration

Posted: Tue Dec 13, 2016 5:51 pm
by cryptochrome
Please report your ticket number here and point them to this thread, vitamind. Thank you :)

Re: Slow transfer speed due to SSD cache acceleration

Posted: Wed Dec 14, 2016 2:41 pm
by xrstokes
Has this been solved?? About to invest a pile of money and might have to go to sinology instead.