Slow transfer speed due to SSD cache acceleration

Questions about SNMP, Power, System, Logs, disk, & RAID.
Post Reply
Paul Hermans
Starting out
Posts: 11
Joined: Fri Jan 29, 2016 8:24 pm

Slow transfer speed due to SSD cache acceleration

Post by Paul Hermans »

I hope this is the right forum for my problem.
I have a problem with transferring large files and more specific in one direction. When sending such a file from a PC via SMB to the NAS, I have transfer speeds between 110 and 120 MB/s. On the other hand when transferring the same file from the NAS to the PC the transfer speeds are between 15 and 30 MB/S. The transfer speed fluctuated also a lot. In the beginning I thought it was a network problem but after doing all possible tests I came to the conclusion that the problem was with the NAS. So I started experimenting with all kinds of settings and finally I disabled the SSD cache acceleration. The reason for this was that the disk performance was very poor at around 400 IOPS. Suddenly the transfer speed when copying from the NAS to the PC went up to between 110 and 120 MB/S. The same speed than in the other direction. I saw the same result with the disk performance which went up to over 4000 IOPS.

My question now is why the SSD cache acceleration slows the transfer down so much and what is the right way to use the cache acceleration? There is also a setting: Bypass Block Size. What does that do?

The files being transferred are between 20 an 50 GB.

My configuration is:

Host: TVS-EC1080+ with 10 x WD Gold 6TB disks (RAID6) + 2 x 128 GB SSD (RAID0)
Expansion unit: REXP-1000 Pro with 10 x WD Gold 6TB disks (RAID6)
Firmware: 4.2.1 Build 20160601
Network: Dynamic Link Aggregation on 4 interfaces (IEEE 802.3ad)
SMB version: 2.1 (because there still a dedicated WIndows 7 PC's on the network)
Switch: 2 x Cisco SG500X-24 Stackable Managed switch
Router: Cisco 2951
Cables: Cat 6
Workstation: HP Envy Phoenix with i7-3770K 3.5GHz quad core, 32GB RAM and Qualcomm Atheros AR8161 NIC (Other PC's are in the same order)
OS: Windows 10 Pro
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by storageman »

That's not good
If RAID 0 your cache was read only. I assume setting it read/write makes no difference.
The cache mainly benefits random read/writes not sequential, your files are especially big.
Still such negative effect on read performance is bad.
Not sure about Bypass Block size but for big files you want a big block size.
cryptochrome
Been there, done that
Posts: 577
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome »

I have just come across the same problem. In my case, I was transferring huge amounts of data from one NAS to another. I had SSD cache enabled with the Bypass Block Size disabled (e.g., cache everything, even sequential operations on large files). The performance was miserable, data was only moving at around 30 MB/sec over a 10 Gig Ethernet connection. Latency in the Disk Performance dashboard reached peaks of 40.000 (!) ms and was averaging between 10.000 and 20.000 ms (thats 10-20 seconds of latency for IOPS). Crazy!

The second I changed the Bypass Block Size down to 4 MB: Boom. Latency dropped to 200-400 ms and transfer speeds went up 5-fold.

I am not really sure yet what exactly is going on. My bet is there is an issue with the SSD caching policy. Either it's bugged, or the caching of sequential (large size) operations simply doesn't work. Although the manuals for dm-cache (lvmcache, used by QNAP) say it's perfectly fine to cache large blocks (e.g. disable the cache bypass).

I opened a ticket with QNAP support. Not expecting much there, but hey...
User avatar
peelos
Been there, done that
Posts: 580
Joined: Sun Jun 26, 2016 9:28 pm

Re: RE: Re: Slow transfer speed due to SSD cache acceleration

Post by peelos »

cryptochrome wrote:I have just come across the same problem. In my case, I was transferring huge amounts of data from one NAS to another. I had SSD cache enabled with the Bypass Block Size disabled (e.g., cache everything, even sequential operations on large files). The performance was miserable, data was only moving at around 30 MB/sec over a 10 Gig Ethernet connection. Latency in the Disk Performance dashboard reached peaks of 40.000 (!) ms and was averaging between 10.000 and 20.000 ms (thats 10-20 seconds of latency for IOPS). Crazy!

The second I changed the Bypass Block Size down to 4 MB: Boom. Latency dropped to 200-400 ms and transfer speeds went up 5-fold.

I am not really sure yet what exactly is going on. My bet is there is an issue with the SSD caching policy. Either it's bugged, or the caching of sequential (large size) operations simply doesn't work. Although the manuals for dm-cache (lvmcache, used by QNAP) say it's perfectly fine to cache large blocks (e.g. disable the cache bypass).

I opened a ticket with QNAP support. Not expecting much there, but hey...
Did you have any luck troubleshooting this further crypto? With 1gb Ethernet I have miserable 20mbps transfer speeds even if bypassing SSD cache block size of 32mb..
NAS: TVS-1282-i7-7700-40G / 4 x 500GB SSD 2.5" RAID 10 / 2 x 500GB M.2 SSD / 8 x 12TB WD Whites 3.5" RAID 6 / Noctua L9x65 / 3 x 80mm PWM Noctua fans / Corsair 600W PSU / Asus Turbo GTX 1060 6GB GPU
Software: Plex Media Server / Transmission / Sonarr / Radarr / Bazarr / Jackett / Tautulli / Home Assistant / Resilio Sync / Python / NetData / SortMyQPKGs
pfSense Firewall / OpenVPN Server: QOTOM Fanless Mini PC / Core i5 / 8GB RAM / 128GB SSD / 4 Gigabit NICs / AES-NI
Wireless Routers: 2 x Netgear AC1900 R7000 Nighthawk / 1 x Netgear AC3200 R8000 Nighthawk / FreshTomato Firmware
cryptochrome
Been there, done that
Posts: 577
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome »

Not really, peelos. 32 MB is too much. I get much better results with 4 MB. Slightly worse with 8 MB and anything above that is just terrible. The default (if I remember correctly) is 1 MB.

Not sure why this is happening, as dm-cache (that's what QNAP use in QTS) should handle even large sequential writes with much bigger chunk sizes perfectly well. Whether it makes sense to do that is another topic. My guess is this could be a bug in either dm-cache or the implementation in QTS. I have a ticket open with QNAP, and except for an initial response where they seemed to have completely misunderstood the problem, I haven't heard back in 7 or 8 days now.
User avatar
peelos
Been there, done that
Posts: 580
Joined: Sun Jun 26, 2016 9:28 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by peelos »

4mb chunks seem to have helped get to 100mbps transfers using AFP.. Not sure the ssd cache helps at all with 1Gbps ethernet, will have to test it properly when I sort everything else out with the NAS
NAS: TVS-1282-i7-7700-40G / 4 x 500GB SSD 2.5" RAID 10 / 2 x 500GB M.2 SSD / 8 x 12TB WD Whites 3.5" RAID 6 / Noctua L9x65 / 3 x 80mm PWM Noctua fans / Corsair 600W PSU / Asus Turbo GTX 1060 6GB GPU
Software: Plex Media Server / Transmission / Sonarr / Radarr / Bazarr / Jackett / Tautulli / Home Assistant / Resilio Sync / Python / NetData / SortMyQPKGs
pfSense Firewall / OpenVPN Server: QOTOM Fanless Mini PC / Core i5 / 8GB RAM / 128GB SSD / 4 Gigabit NICs / AES-NI
Wireless Routers: 2 x Netgear AC1900 R7000 Nighthawk / 1 x Netgear AC3200 R8000 Nighthawk / FreshTomato Firmware
cryptochrome
Been there, done that
Posts: 577
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome »

I think the cache is primarily designed to cache local actions and not so much to act as a network buffer when receiving data. That's my conclusion of my observations so far. What's interesting is that someone in the large TVS-x82 thread mentioned a few weeks back that he saw a special network cache feature in the 4.2.2 beta, but I have yet to find that in the released code.

It's just too bad that QNAP don't document their features properly. All they have is "click here to enabled this and that" but no real technical background information. If you try to get such information from their support, they usually fail to deliver.

Anyway. I will keep digging into this. I am reading up alot about dm-cache and how it works.
nikrampi
First post
Posts: 1
Joined: Tue Nov 29, 2016 10:45 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by nikrampi »

Hi all, sorry to resume old thread but I find myself in same situation as described by cryptochrome and peelos and I've reached this thread searching solution for SSD cache config.
cryptochrome wrote:Not really, peelos. 32 MB is too much. I get much better results with 4 MB. Slightly worse with 8 MB and anything above that is just terrible. The default (if I remember correctly) is 1 MB.

Not sure why this is happening, as dm-cache (that's what QNAP use in QTS) should handle even large sequential writes with much bigger chunk sizes perfectly well. Whether it makes sense to do that is another topic. My guess is this could be a bug in either dm-cache or the implementation in QTS. I have a ticket open with QNAP, and except for an initial response where they seemed to have completely misunderstood the problem, I haven't heard back in 7 or 8 days now.
Can I ask if you've got any insight from the support? I'm wondering if I should open a ticket too on this same issue.
cryptochrome wrote:What's interesting is that someone in the large TVS-x82 thread mentioned a few weeks back that he saw a special network cache feature in the 4.2.2 beta, but I have yet to find that in the released code.
I could not find any reference on this point on the changelog now that the firmware is out, I'll probably give a try in the next few weeks just to see if there are improvements on this issue.
Has anybody tried to tinker with SSD caching settings on latest firmware?

Thanks again for any support. I'll share my improvements if I have any luck in making SSD caching large chunks of data.
cryptochrome
Been there, done that
Posts: 577
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome »

Unfortunately QNAP support had zero clue about what I was talking about and after weeks of going back and forth with them, they basically said "we can't reproduce the issue, so there is no issue".

I have since reduced the cache block size to 1 MB which greatly reduces the issue but does not entirely remove it. I would love to crank this up so that larger file operations (which are frequent in my use case) could benefit, and I hope that soon to be released 4.3 improves on this (maybe through update dm-cache/lvm components) but I don't have high hopes.
sqparnell
New here
Posts: 3
Joined: Thu Dec 01, 2016 5:33 am

Re: Slow transfer speed due to SSD cache acceleration

Post by sqparnell »

I had same issues - over Thunderbolt, was getting really slow and sporadic speeds, averaging around 70 MB/s read and write. Turned off SSD Cache and instantly saw steady performance - 800 MB/s download and 530 MB/s upload. Thanks!
User avatar
Don
Guru
Posts: 12289
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York

Re: Slow transfer speed due to SSD cache acceleration

Post by Don »

If more people open tickets then they will see it is not an isolated issue.
Use the forum search feature before posting.

Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
cryptochrome
Been there, done that
Posts: 577
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome »

Oh I pointed them to another thread where some people reported they had the issue (the large TVSx82 thread), but that didn't stop them from saying the can't reproduce it. But yes, please, everyone open tickets. The more, the better.

I have invested a boatload of money into SSDs just for caching and it just doesn't work.
vitamind
Starting out
Posts: 14
Joined: Wed Dec 30, 2015 1:29 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by vitamind »

It's been driving me crazy for the last couple of months. I upgraded to Windows 10 on both PCs and figured that was the reason behind the now slow speeds. I've tried everything on so many forums and never even thought it could be the server. Sure enough I turned Cache Acceleration off and instantly my speeds increased to what they use to be. I'll start a ticket on this issue as well. Thanks.

TVS-EC1080
4.2.2 Build 20161102
cryptochrome
Been there, done that
Posts: 577
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome »

Please report your ticket number here and point them to this thread, vitamind. Thank you :)
xrstokes
New here
Posts: 2
Joined: Wed Dec 14, 2016 2:30 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by xrstokes »

Has this been solved?? About to invest a pile of money and might have to go to sinology instead.
Post Reply

Return to “System & Disk Volume Management”