Slow transfer speed due to SSD cache acceleration

Questions about SNMP, Power, System, Logs, disk, & RAID.
Post Reply
mman3222

Re: Slow transfer speed due to SSD cache acceleration

Post by mman3222 »

AfroDieter wrote: Mon Sep 23, 2019 7:46 am OK guys, I'm sorry this post is going to be so long, but if you actually give a ** about getting this cache performance thing solved, you may want to plow through it anway.

TL;DR: I think I'm coming to an understanding of why the performance starts out great with the cache and then hits a brick wall after some time, and I think I may be on the right path for finding a solution that avoids this performance crash. Read on for the gory details.

@joshuacant, my personal belief right now is as follows, and I'm doing research and investigation (and learning) to confirm or deny it, so here goes:

I believe that the QNAP device is not properly flushing the cache of dirty blocks. Over time the number of dirty blocks grows and grows until the entire cache is full of dirty blocks, upon which all subsequent writes will force the dirty blocks to first be written out to disk, then the blocks are free to be overwritten with new data. This is probably what people are seeing when they say that upon filling the cache performance plummets.

I'm new to all of this, so I've been doing some intense Googling, reading, and poking around on my TS-963X.

...........
Hi AfroDieter, thank you for your detailed posts and brining this to QNAP's attention. I also have noticed that QNAP is not flushing my cache acceleration properly. I have ticket open, and they've escalated the ticket on Thursday, and I'm waiting for HQ to connect to my QNAP remotely to try and help resolve the issue. However, I bought a new QNAP TVS-672XT to handle Plex and storage (I wanted a new one anyway) and I STILL can't get the cache acceleration to flush on it's own. On top of that, when I visit the cache acceleration page the hit rate is always super high. 100% for hours and hours, and there is NO activity going on the NAS. The first tech support person wanted me to to change cache acceleration to Random I/O with 32MB sectors, but that defeats the purpose of me being able to use this as a file sharing storage and copy large files at 1GB/sec with my 10Gbe connection. The speed dropped down to 200MB/sec.

At this point, I don't know what to do. I've been living with the issue for about a year, and I have to manually login to the NAS and flush the cache acceleration weekly. It's really annoying, and I wish they would solve the issue.

If QNAP doesn't help, or they can't solve the problem, here is going to be my work around:
1. Disable cache acceleration
2. Create a new mount/drive with my 1TB Samsung 970 EVO
3. After I'm done downloading large files, copy them to the Samsung SSD for the day. (Add these folders to Plex.)
4. Every night I will have "SyncBackSE" move the files from the Samsung SSD to my regular QNAP storage.
5. Plex will see the file(s) moved from the SSD to the regular storage and update the file locations for my video files.
MaStr
New here
Posts: 2
Joined: Mon Aug 26, 2019 1:50 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by MaStr »

I only want to dig that thread out of the graveyard... because I hit that SSD cache wall yesterday, too. (As soon as the cache hits 99%, it never decreases and a full cache drops the speed and waits to long for a dirty background I/O )

I also tried to mitigate the dirty_block sync problem with running a shell-script, which monitors the amount of dirty_blocks and triggers a do_sync.
But I can't find a way to bring the flashcache modules to flush out old unused cache pages. Based on the documentation, the LRU feature should make sure the drop not necessary pages out of the cache. It seems that for some reason the background kernel tasks aren't scheduled/started to sort out unused cache pages.

About the fact, that QNAP is using an old unmaintained version of flashcache. This is not 100% true, because QNAP added a bunch of code. I am guessing they, at least try, to make it working well and keep working on it.
I downloaded the GPL Sourcecode of 4.4.1 and made a compare against the latest Facebook version. At least they try to improve constantly the code.

I'm running 4.4.3.1400 firmware on a TVS-873, and my system also publishes more information about the kernel flashcache-module then the corresponding version of 4.4.1 .

At least a recent GPL code package for 4.4.3 would be useful to get closer to that problem of non freed cache.
mman3222

Re: Slow transfer speed due to SSD cache acceleration

Post by mman3222 »

I opened a ticket back on August 27th, because this issue still has not been resolved. I've been dealing with it for over 1 year now! I have to login to the QNAP, check the Cache Acceleration, and when I see it getting close to 100%, I have to manually flush the cache and wait a long time for it to complete.

QNAP escalated the ticket to HQ for me. I was working with Daniel through support. He was great and understood exactly how the feature is supposed to work. So after Daniel being the middleman between me and HQ, they did not give me good news today. Their response was that the "feature" is working as intended! LOL! Here is their official response:

So according to the team, it looks like your NAS currently has normal behavior in terms of the flushing.
They said the flushing is dependent on "dirty blocks", not just the overall usage amoun, so even if the overall usage is at 99% it might not flush all the way.


They connected to my QNAP remotely TWICE to investigate.

Daniel offered to submit this as a feature request. Here was my response back:

That's really disappointing to hear. To me, that is not how flushing the cache should work. Everything transfers super fast at 1GB/sec over 10Gbe, but as soon as the cache acceleration is full, my transfer speeds drop to normal speeds. So yes, it does affect my NAS' performance. I have to go into my NAS and manually flush the cache acceleration and pay attention to when it gets close to 100% full. Not only that, but it takes FOREVER to flush when you have nearly 1TB full. That's why it would be great to have the cache acceleration flush nightly, etc.

According to your team's description, this 100% seems like false advertising in how the function is supposed to work. I don't see anywhere on QNAP's website and support documents that the cache acceleration is depenedent on flushing dirty blocks.

In fact, on QNAP's support documentation located here: https://docs.qnap.com/nas/QTS4.4.1/ENG/ ... 610F2.html

QNAP even says "QTS writes incoming data to the SSD cache first, then flushes the data to regular storage LATER. Read access to the new data is also accelerated while it is in the cache."

Later is not happening.

If HQ can't offer me a solution, yes, please submit a feature request. As mentioned before, I bought a brand new QNAP for $1,700 thinking it was just my old QNAP not working, and the new one would work as intended. This is really disappointing to see QNAP saying it's working as intended.


So unless you all start submitting tickets, QNAP is going to think this is not a problem. We ALL need to submit a ticket to get this fixed. I encourage you to submit a ticket!
mman3222

Re: Slow transfer speed due to SSD cache acceleration

Post by mman3222 »

klaasdj wrote: Fri Sep 11, 2020 1:38 pm The only solution so far is (and I was triggered by mman3222's contribution) use SSD as a separate volume, write all daily data to this volume, which is really fast, and over night copy the data from SSD to the HDD based volume. I also happen the use SyncBack but the Pro version for as long as it exists.
This works for me.
Thanks!

Same here. Works for me. Still crappy the feature doesn't work as intended.

Since I have Plex running directly off my TVS-672XT, Plex picks up the new location of the video files after Syncback is done moving the video files from the NVMe to my regular storage.

It's a workaround, and NOT how cache is supposed to work, but it works for me :?
P3R
Guru
Posts: 13183
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Slow transfer speed due to SSD cache acceleration

Post by P3R »

I see this constant mentioning of Syncback being used for NAS-internal synchronizations and wonder why? While I love Syncback for doing Windows backups I don't understand why it is used for this when the Qnap app Hybrid Backup Sync can do the synchronizations directly in the NAS.

Using Syncback on a client computer means copying all data out of the NAS and then back in so that all data will need to be transported over the network twice in addition to that the client have to be switched on nightly.

Using Syncback in a VM at least mean not all data have to be copied over the physical network and that clients need to run all night but the virtualization add overhead and native applications are easier to manage.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
mman3222

Re: Slow transfer speed due to SSD cache acceleration

Post by mman3222 »

I'm not familiar with Hybrid backup Sync. I guess I never really looked into the QNAP apps before.

Lets say I have file "123.mkv" in NVMeDrive\Movies. Will it copy to my regular store or MOVE/cut it to regular storage? When Syncback runs at night, it moves the files so then my cache drive is 100% empty again.
P3R
Guru
Posts: 13183
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Slow transfer speed due to SSD cache acceleration

Post by P3R »

mman3222 wrote: Fri Sep 11, 2020 3:01 pm Lets say I have file "123.mkv" in NVMeDrive\Movies. Will it copy to my regular store or MOVE/cut it to regular storage? When Syncback runs at night, it moves the files so then my cache drive is 100% empty again.
Ah, okay. It's not really synchronizations, it's moving data.

HBS can't do moves, only backups and syncs. That's very easy to work around though as piggy-backed to the sync job you could have a job that sync an empty folder to the SSD folder and thereby delete all data on it. In addition, that would be a completely safe transfer of the data. I don't know if Syncback actually do a sync plus delete behind the scenes or if it really use the more risky approach of moving? If the latter, that would be another disadvantage with Syncback.

If you're not familiar with HBS, how do you backup the data on the NAS? Do you use Synback for that as well?
Last edited by P3R on Fri Sep 11, 2020 3:45 pm, edited 1 time in total.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
P3R
Guru
Posts: 13183
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Slow transfer speed due to SSD cache acceleration

Post by P3R »

klaasdj wrote: Fri Sep 11, 2020 3:08 pm P3R knows it better AGAIN!
Please present your opposing arguments on the subject instead of attacking the person.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
MaStr
New here
Posts: 2
Joined: Mon Aug 26, 2019 1:50 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by MaStr »

mman3222 wrote: Fri Sep 11, 2020 4:40 am I opened a ticket back on August 27th, because this issue still has not been resolved. I've been dealing with it for over 1 year now! I have to login to the QNAP, check the Cache Acceleration, and when I see it getting close to 100%, I have to manually flush the cache and wait a long time for it to complete.

QNAP escalated the ticket to HQ for me. I was working with Daniel through support. He was great and understood exactly how the feature is supposed to work. So after Daniel being the middleman between me and HQ, they did not give me good news today. Their response was that the "feature" is working as intended! LOL!

[ .. ]

So unless you all start submitting tickets, QNAP is going to think this is not a problem. We ALL need to submit a ticket to get this fixed. I encourage you to submit a ticket!
Thank you for reporting back your investigation and results with the QNAP team.

:-(
mman3222

Re: Slow transfer speed due to SSD cache acceleration

Post by mman3222 »

P3R wrote: Fri Sep 11, 2020 3:37 pm
mman3222 wrote: Fri Sep 11, 2020 3:01 pm Lets say I have file "123.mkv" in NVMeDrive\Movies. Will it copy to my regular store or MOVE/cut it to regular storage? When Syncback runs at night, it moves the files so then my cache drive is 100% empty again.
Ah, okay. It's not really synchronizations, it's moving data.

HBS can't do moves, only backups and syncs. That's very easy to work around though as piggy-backed to the sync job you could have a job that sync an empty folder to the SSD folder and thereby delete all data on it. In addition, that would be a completely safe transfer of the data. I don't know if Syncback actually do a sync plus delete behind the scenes or if it really use the more risky approach of moving? If the latter, that would be another disadvantage with Syncback.

If you're not familiar with HBS, how do you backup the data on the NAS? Do you use Synback for that as well?
It's just easier for me to stick with Syncback then. I only have 0 - 600GB MAX in a night it needs to move. However, the average is probably 50 - 100GB.

I use Google Drive with Stablebit to backup my data. And you're correct. I use Syncback for that too. Once a day it backs up. Stablebit tricks Google Drive to making it a "local" hard drive on my computer. It's encrypted, and Syncback backs all my stuff up flawlessly. I've been using this method for around 2 years now.
P3R
Guru
Posts: 13183
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Slow transfer speed due to SSD cache acceleration

Post by P3R »

mman3222 wrote: Fri Sep 11, 2020 11:02 pm I use Google Drive with Stablebit to backup my data. And you're correct. I use Syncback for that too. Once a day it backs up. Stablebit tricks Google Drive to making it a "local" hard drive on my computer. It's encrypted, and Syncback backs all my stuff up flawlessly. I've been using this method for around 2 years now.
HBS can backup to Google drive as well, including doing client-side encryption, but if what you have work and you're happy with it I understand that you don't want to change as HBS will no doubt require some learning time.

I just wanted to inform that a good NAS-integrated backup/sync solution existed for those that wasn't aware. I do agree that Syncback is a nice Windows backup software. :)
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
pokrakam
Know my way around
Posts: 101
Joined: Wed Sep 17, 2008 5:27 am

Re: Slow transfer speed due to SSD cache acceleration

Post by pokrakam »

FWIW, I have had numerous issues with HBS as it does not play nicely with S3-compatible storage. So I use Arq backup on a Mac to backup my NAS.
HiTach
Starting out
Posts: 38
Joined: Fri Feb 15, 2019 6:03 am

Re: Slow transfer speed due to SSD cache acceleration

Post by HiTach »

So today I checked my W/R speeds and was very surprised. :S :-0 :DD
This 872XT system has been up for 12 days. I have not flushed the cache since.
See images for surprising speeds - 1202MB/s Write 1862MB/s Read
Repeatable over several tests +/- 100 MB/s.
Most system specs in signature, using Mellanox MCX313A single port 40GB/s card on host pc but still MCX314A dual port 40GB/s card on NAS.
I use the second NAS 40GB/s port with adapters to provide 10Gbe on 10-BaseT.

I also confirmed by transfer multiple 30GB movie files at 1.3GB/s to 1.9GB/s with Windows file copy.

I have no idea what, but for months I have been lucky to see 700MB/s W/R and sometimes as low as 500 or even 400 MB/s.

I'll keep track over the coming days and hope it's sustained. I am up to date on QTS 4.4.3.1421.
You do not have the required permissions to view the files attached to this post.
Workstation DAS/NAS: TVS-872XT i5 16GB QuTS Hero 5.x + 2x NB 12-P fans
2x1TB Samsung 970 EVO plus RAID1 w/EK WB M.2 heatsinks
8x20TB Seagate Exos X22 RAID6 112TB Thin volume
1x20Gbe Qnap Thunderbolt 3 card, 1x40Gbe Mellanox DAS network connection, 10Gbe local virtual switch
Backup NAS: TVS-872XT i5 16GB QTS 5.x + 2x NB 12-P fans
8x10TB WD Gold RAID6 54TB Thick volume w/snapshots
1x40Gbe Qnap Thunderbolt 3 card, 1x40Gbe + SFP+/RJ45 for local 10gbe virtual switch
LAN switch 10Gbe(2)/1Gbe(8) NETGEAR Nighthawk SX10 (GS810EMX)
mman3222

Re: Slow transfer speed due to SSD cache acceleration

Post by mman3222 »

Those speeds are normal for me with my TS-672XT. I have a Samsung 1TB M.2 970 EVO SSD that I use for cache and I get 1.15GB/sec over a mixture of cat 5e / cat 6 cables.

I'm surprised you were getting such slow speeds prior to this.
dnavas
Getting the hang of things
Posts: 58
Joined: Wed Feb 24, 2010 5:37 am
Location: San Jose

Re: Slow transfer speed due to SSD cache acceleration

Post by dnavas »

Hmm, technically I think QNAP is right?? -- eviction from a cache typically occurs on cache-full. It could very well be that the cache is being written through to the drive (although, if you're spending a lot of time waiting for a flush, it appears not, and that DOES appear to be counter to their docs), but that doesn't imply it should be evicted from the cache. The cache will definitely fill up and absent some kind of timeout mechanism....
What most of us actually want, I think, is a large write buffer (not a cache) and a large read cache with asynchronous fill (don't block reads to populate the cache).
Does that sound right, or am I misunderstanding this thread?
Post Reply

Return to “System & Disk Volume Management”