Slow transfer speed due to SSD cache acceleration

Questions about SNMP, Power, System, Logs, disk, & RAID.
JRaymond
Starting out
Posts: 21
Joined: Mon Oct 27, 2014 4:11 am

Re: Slow transfer speed due to SSD cache acceleration

Post by JRaymond » Sun Aug 02, 2020 6:16 am

I started a thread on my issues before finding this one...located here viewtopic.php?f=25&t=156214

I'm seeing 35 MB/s for 10 NAS configured the same as my master. I need 5x that to make this project work in a timely fashion. I'm seeing 14 hours to transfer 2.5TB of data out from my master to the clients.

We just started this project...so I figured there would be SOME tweaking but FFS, I've got a 10GB pipe out of the NAS and can only get 35MB/s for 10 systems?

hardmack
New here
Posts: 5
Joined: Sun Jul 19, 2020 6:44 am

Re: Slow transfer speed due to SSD cache acceleration

Post by hardmack » Sun Aug 02, 2020 11:05 am

JRaymond wrote:
Sun Aug 02, 2020 6:16 am
I started a thread on my issues before finding this one...located here viewtopic.php?f=25&t=156214

I'm seeing 35 MB/s for 10 NAS configured the same as my master. I need 5x that to make this project work in a timely fashion. I'm seeing 14 hours to transfer 2.5TB of data out from my master to the clients.

We just started this project...so I figured there would be SOME tweaking but FFS, I've got a 10GB pipe out of the NAS and can only get 35MB/s for 10 systems?
I saw the same transfer speeds, I had to disable ssd caching all together and instantly once disabled full network speed

JRaymond
Starting out
Posts: 21
Joined: Mon Oct 27, 2014 4:11 am

Re: Slow transfer speed due to SSD cache acceleration

Post by JRaymond » Mon Aug 03, 2020 7:21 am

hardmack wrote:
Sun Aug 02, 2020 11:05 am
JRaymond wrote:
Sun Aug 02, 2020 6:16 am
I started a thread on my issues before finding this one...located here viewtopic.php?f=25&t=156214

I'm seeing 35 MB/s for 10 NAS configured the same as my master. I need 5x that to make this project work in a timely fashion. I'm seeing 14 hours to transfer 2.5TB of data out from my master to the clients.

We just started this project...so I figured there would be SOME tweaking but FFS, I've got a 10GB pipe out of the NAS and can only get 35MB/s for 10 systems?
I saw the same transfer speeds, I had to disable ssd caching all together and instantly once disabled full network speed
I "have" no SSD caching...at all. There are no SSD's installed in the system.

mman3222
New here
Posts: 6
Joined: Sun Aug 23, 2020 11:34 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by mman3222 » Sun Aug 23, 2020 11:46 pm

AfroDieter wrote:
Mon Sep 23, 2019 7:46 am
OK guys, I'm sorry this post is going to be so long, but if you actually give a ** about getting this cache performance thing solved, you may want to plow through it anway.

TL;DR: I think I'm coming to an understanding of why the performance starts out great with the cache and then hits a brick wall after some time, and I think I may be on the right path for finding a solution that avoids this performance crash. Read on for the gory details.

@joshuacant, my personal belief right now is as follows, and I'm doing research and investigation (and learning) to confirm or deny it, so here goes:

I believe that the QNAP device is not properly flushing the cache of dirty blocks. Over time the number of dirty blocks grows and grows until the entire cache is full of dirty blocks, upon which all subsequent writes will force the dirty blocks to first be written out to disk, then the blocks are free to be overwritten with new data. This is probably what people are seeing when they say that upon filling the cache performance plummets.

I'm new to all of this, so I've been doing some intense Googling, reading, and poking around on my TS-963X.

...........
Hi AfroDieter, thank you for your detailed posts and brining this to QNAP's attention. I also have noticed that QNAP is not flushing my cache acceleration properly. I have ticket open, and they've escalated the ticket on Thursday, and I'm waiting for HQ to connect to my QNAP remotely to try and help resolve the issue. However, I bought a new QNAP TVS-672XT to handle Plex and storage (I wanted a new one anyway) and I STILL can't get the cache acceleration to flush on it's own. On top of that, when I visit the cache acceleration page the hit rate is always super high. 100% for hours and hours, and there is NO activity going on the NAS. The first tech support person wanted me to to change cache acceleration to Random I/O with 32MB sectors, but that defeats the purpose of me being able to use this as a file sharing storage and copy large files at 1GB/sec with my 10Gbe connection. The speed dropped down to 200MB/sec.

At this point, I don't know what to do. I've been living with the issue for about a year, and I have to manually login to the NAS and flush the cache acceleration weekly. It's really annoying, and I wish they would solve the issue.

If QNAP doesn't help, or they can't solve the problem, here is going to be my work around:
1. Disable cache acceleration
2. Create a new mount/drive with my 1TB Samsung 970 EVO
3. After I'm done downloading large files, copy them to the Samsung SSD for the day. (Add these folders to Plex.)
4. Every night I will have "SyncBackSE" move the files from the Samsung SSD to my regular QNAP storage.
5. Plex will see the file(s) moved from the SSD to the regular storage and update the file locations for my video files.

MaStr
New here
Posts: 2
Joined: Mon Aug 26, 2019 1:50 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by MaStr » Fri Sep 11, 2020 2:54 am

I only want to dig that thread out of the graveyard... because I hit that SSD cache wall yesterday, too. (As soon as the cache hits 99%, it never decreases and a full cache drops the speed and waits to long for a dirty background I/O )

I also tried to mitigate the dirty_block sync problem with running a shell-script, which monitors the amount of dirty_blocks and triggers a do_sync.
But I can't find a way to bring the flashcache modules to flush out old unused cache pages. Based on the documentation, the LRU feature should make sure the drop not necessary pages out of the cache. It seems that for some reason the background kernel tasks aren't scheduled/started to sort out unused cache pages.

About the fact, that QNAP is using an old unmaintained version of flashcache. This is not 100% true, because QNAP added a bunch of code. I am guessing they, at least try, to make it working well and keep working on it.
I downloaded the GPL Sourcecode of 4.4.1 and made a compare against the latest Facebook version. At least they try to improve constantly the code.

I'm running 4.4.3.1400 firmware on a TVS-873, and my system also publishes more information about the kernel flashcache-module then the corresponding version of 4.4.1 .

At least a recent GPL code package for 4.4.3 would be useful to get closer to that problem of non freed cache.

mman3222
New here
Posts: 6
Joined: Sun Aug 23, 2020 11:34 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by mman3222 » Fri Sep 11, 2020 4:40 am

I opened a ticket back on August 27th, because this issue still has not been resolved. I've been dealing with it for over 1 year now! I have to login to the QNAP, check the Cache Acceleration, and when I see it getting close to 100%, I have to manually flush the cache and wait a long time for it to complete.

QNAP escalated the ticket to HQ for me. I was working with Daniel through support. He was great and understood exactly how the feature is supposed to work. So after Daniel being the middleman between me and HQ, they did not give me good news today. Their response was that the "feature" is working as intended! LOL! Here is their official response:

So according to the team, it looks like your NAS currently has normal behavior in terms of the flushing.
They said the flushing is dependent on "dirty blocks", not just the overall usage amoun, so even if the overall usage is at 99% it might not flush all the way.


They connected to my QNAP remotely TWICE to investigate.

Daniel offered to submit this as a feature request. Here was my response back:

That's really disappointing to hear. To me, that is not how flushing the cache should work. Everything transfers super fast at 1GB/sec over 10Gbe, but as soon as the cache acceleration is full, my transfer speeds drop to normal speeds. So yes, it does affect my NAS' performance. I have to go into my NAS and manually flush the cache acceleration and pay attention to when it gets close to 100% full. Not only that, but it takes FOREVER to flush when you have nearly 1TB full. That's why it would be great to have the cache acceleration flush nightly, etc.

According to your team's description, this 100% seems like false advertising in how the function is supposed to work. I don't see anywhere on QNAP's website and support documents that the cache acceleration is depenedent on flushing dirty blocks.

In fact, on QNAP's support documentation located here: https://docs.qnap.com/nas/QTS4.4.1/ENG/ ... 610F2.html

QNAP even says "QTS writes incoming data to the SSD cache first, then flushes the data to regular storage LATER. Read access to the new data is also accelerated while it is in the cache."

Later is not happening.

If HQ can't offer me a solution, yes, please submit a feature request. As mentioned before, I bought a brand new QNAP for $1,700 thinking it was just my old QNAP not working, and the new one would work as intended. This is really disappointing to see QNAP saying it's working as intended.


So unless you all start submitting tickets, QNAP is going to think this is not a problem. We ALL need to submit a ticket to get this fixed. I encourage you to submit a ticket!

User avatar
klaasdj
New here
Posts: 9
Joined: Sat Apr 18, 2009 1:03 pm
Location: Maassluis

Re: Slow transfer speed due to SSD cache acceleration

Post by klaasdj » Fri Sep 11, 2020 1:38 pm

So finally we all come the conclusion that QNAP does not understand the algorithm of SSD caching and Qtier which they proudly advertise. Both do not function the way it should. They (QNAP that is) can of course claim that it does work on the assupmption that their code is right, and that they described it the right way but if it does at the end not function the way it should it is a "paper tiger".
After my first "contribution" (July 17th) I have tried various configurations of caching and Qtier on my TS-453D. I even bought a QM2-2P10G1TA card (Dual M.2 PCI NVMe card with 10GB NBase-T support) and put 2 Intel 660 512GB NVMe drives in it. Result is 0.0 for both Qtier and SSD caching. It starts quit promising, but after a few days they both **. (SSD caching AND Qtier)

Also got my "old" TVS-951X up and running again. Next to 2 x 8TB WD Gold drives I put in 2 Samsung EVO Pro SSD drives of 1TB each. Tested it with caching and Qtier. Activated Qtier before making folders activated it after making folders but the net result is 0.0 and caching has the samne problem.

Just for the fun of this I also bought an second hand TVS-463, put my earlier purchased 10GB Aquantia card in the PCIe slot and filled 2 drive bays with 6TB WD RED Pro and 2 drive bays with the SSD's mentioned above. Nada, result is 0.0 for Qtier and SSD caching

The only solution so far is (and I was triggered by mman3222's contribution) use SSD as a separate volume, write all daily data to this volume, which is really fast, and over night copy the data from SSD to the HDD based volume. I also happen the use SyncBack but the Pro version for as long as it exists.
This works for me.

May be some day QNAP developers will see the light and understand how Qtier should work ( (just Automatic tiering as many other brands implemented it) and they also should work on the algorithm of SSD caching. And then maybe this thread can be closed.
TS-453D with QM2-2P10G1TA PCIe card with 2 Intel 660 NVMe M.2 flash modules, 2 x 8TB WD Gold HDD's and 32GB Kingston RAM
TVS-951X with 2 x 6TB WD RED and 2 1TB Samsung EVO PRO SSD and 16B RAM
QSW-M408-4C 10GbE Managed Switch, with 4-Port 10GbE SFP+/RJ45 Combo and 8-Port Gigabit

mman3222
New here
Posts: 6
Joined: Sun Aug 23, 2020 11:34 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by mman3222 » Fri Sep 11, 2020 1:43 pm

klaasdj wrote:
Fri Sep 11, 2020 1:38 pm
The only solution so far is (and I was triggered by mman3222's contribution) use SSD as a separate volume, write all daily data to this volume, which is really fast, and over night copy the data from SSD to the HDD based volume. I also happen the use SyncBack but the Pro version for as long as it exists.
This works for me.
Thanks!

Same here. Works for me. Still crappy the feature doesn't work as intended.

Since I have Plex running directly off my TVS-672XT, Plex picks up the new location of the video files after Syncback is done moving the video files from the NVMe to my regular storage.

It's a workaround, and NOT how cache is supposed to work, but it works for me :?

P3R
Guru
Posts: 12452
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Slow transfer speed due to SSD cache acceleration

Post by P3R » Fri Sep 11, 2020 2:56 pm

I see this constant mentioning of Syncback being used for NAS-internal synchronizations and wonder why? While I love Syncback for doing Windows backups I don't understand why it is used for this when the Qnap app Hybrid Backup Sync can do the synchronizations directly in the NAS.

Using Syncback on a client computer means copying all data out of the NAS and then back in so that all data will need to be transported over the network twice in addition to that the client have to be switched on nightly.

Using Syncback in a VM at least mean not all data have to be copied over the physical network and that clients need to run all night but the virtualization add overhead and native applications are easier to manage.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

mman3222
New here
Posts: 6
Joined: Sun Aug 23, 2020 11:34 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by mman3222 » Fri Sep 11, 2020 3:01 pm

I'm not familiar with Hybrid backup Sync. I guess I never really looked into the QNAP apps before.

Lets say I have file "123.mkv" in NVMeDrive\Movies. Will it copy to my regular store or MOVE/cut it to regular storage? When Syncback runs at night, it moves the files so then my cache drive is 100% empty again.

User avatar
klaasdj
New here
Posts: 9
Joined: Sat Apr 18, 2009 1:03 pm
Location: Maassluis

Re: Slow transfer speed due to SSD cache acceleration

Post by klaasdj » Fri Sep 11, 2020 3:08 pm

Quod erat expectata

P3R knows it better AGAIN!
TS-453D with QM2-2P10G1TA PCIe card with 2 Intel 660 NVMe M.2 flash modules, 2 x 8TB WD Gold HDD's and 32GB Kingston RAM
TVS-951X with 2 x 6TB WD RED and 2 1TB Samsung EVO PRO SSD and 16B RAM
QSW-M408-4C 10GbE Managed Switch, with 4-Port 10GbE SFP+/RJ45 Combo and 8-Port Gigabit

P3R
Guru
Posts: 12452
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Slow transfer speed due to SSD cache acceleration

Post by P3R » Fri Sep 11, 2020 3:37 pm

mman3222 wrote:
Fri Sep 11, 2020 3:01 pm
Lets say I have file "123.mkv" in NVMeDrive\Movies. Will it copy to my regular store or MOVE/cut it to regular storage? When Syncback runs at night, it moves the files so then my cache drive is 100% empty again.
Ah, okay. It's not really synchronizations, it's moving data.

HBS can't do moves, only backups and syncs. That's very easy to work around though as piggy-backed to the sync job you could have a job that sync an empty folder to the SSD folder and thereby delete all data on it. In addition, that would be a completely safe transfer of the data. I don't know if Syncback actually do a sync plus delete behind the scenes or if it really use the more risky approach of moving? If the latter, that would be another disadvantage with Syncback.

If you're not familiar with HBS, how do you backup the data on the NAS? Do you use Synback for that as well?
Last edited by P3R on Fri Sep 11, 2020 3:45 pm, edited 1 time in total.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

P3R
Guru
Posts: 12452
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Slow transfer speed due to SSD cache acceleration

Post by P3R » Fri Sep 11, 2020 3:44 pm

klaasdj wrote:
Fri Sep 11, 2020 3:08 pm
P3R knows it better AGAIN!
Please present your opposing arguments on the subject instead of attacking the person.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

User avatar
klaasdj
New here
Posts: 9
Joined: Sat Apr 18, 2009 1:03 pm
Location: Maassluis

Re: Slow transfer speed due to SSD cache acceleration

Post by klaasdj » Fri Sep 11, 2020 4:20 pm

P3R, We have had that discussion before I think.

Getting involved and commenting on everything (not even knowing the details) is not what a real Guru ought to do in my opinion.
End of the discussion ..............
TS-453D with QM2-2P10G1TA PCIe card with 2 Intel 660 NVMe M.2 flash modules, 2 x 8TB WD Gold HDD's and 32GB Kingston RAM
TVS-951X with 2 x 6TB WD RED and 2 1TB Samsung EVO PRO SSD and 16B RAM
QSW-M408-4C 10GbE Managed Switch, with 4-Port 10GbE SFP+/RJ45 Combo and 8-Port Gigabit

MaStr
New here
Posts: 2
Joined: Mon Aug 26, 2019 1:50 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by MaStr » Fri Sep 11, 2020 6:42 pm

mman3222 wrote:
Fri Sep 11, 2020 4:40 am
I opened a ticket back on August 27th, because this issue still has not been resolved. I've been dealing with it for over 1 year now! I have to login to the QNAP, check the Cache Acceleration, and when I see it getting close to 100%, I have to manually flush the cache and wait a long time for it to complete.

QNAP escalated the ticket to HQ for me. I was working with Daniel through support. He was great and understood exactly how the feature is supposed to work. So after Daniel being the middleman between me and HQ, they did not give me good news today. Their response was that the "feature" is working as intended! LOL!

[ .. ]

So unless you all start submitting tickets, QNAP is going to think this is not a problem. We ALL need to submit a ticket to get this fixed. I encourage you to submit a ticket!
Thank you for reporting back your investigation and results with the QNAP team.

:-(

Post Reply

Return to “System & Disk Volume Management”