Slow transfer speed due to SSD cache acceleration

Questions about SNMP, Power, System, Logs, disk, & RAID.
Post Reply
User avatar
jds580s
Know my way around
Posts: 206
Joined: Tue Dec 08, 2009 4:52 am

Re: Slow transfer speed due to SSD cache acceleration

Post by jds580s »

cryptochrome wrote:
P3R wrote:Since it seems that not everyone experience these problems
I have yet to see a single working SSD cache implementation on any QNAP model. This thread was started over a year ago. Countless support tickets were opened with QNAP by many people. Nothing ever came out of it. They don't even acknowledge there is a problem because allegedly they can't reproduce it. Read this thread from start to finish.
If you watch the one of the recent videos on the Qnap YouTube page they discuss Qtier 2.0 which they say is intended to replace the current SSD cache by rolling it’s intended benefits into Qtier.

I don’t recall for certain but I think it’s supposed to come out with 4.3.4

I am hoping that this new implementation is their solution to the current SSD cache issues.

Model: TVS-1282-i5-16G

[list]
[*]Firmware: QTS 4.5.3.1652 build 20210428
[*]Network: 10GbE ASUS XG-C100C card, MTU 9k
[*]RAID 1: [System] 2x WD Blue M.2 SSD 250GB
[*]RAID 6: [DATA] 5x HGST HDN728080ALE604 8TB
[list]
[*] Qtier RAID 1: 2x Samsung SSD 850 EVO 500GB + 2x Samsung SSD 860 EVO 500GB
[*] Cache RAID 1: 2x Samsung SSD 960 EVO 500GB NVMe M.2 in two NGFF PCIe 3.0 x4 adapter cards[/list][/list]
Model: TS-459 Pro
[list]
[*]Firmware: QTS 4.2.6 build 20210327
[*]RAID 6: 2x HGST HDN724030ALE640 3TB, x2 Seagate ST3000VN000 3TB
[*]External: 4TB HGST eSATA Drive, UPS
[*]Network: 1 Gbps, MTU 1500[/list]
[/color]
okenny
Easy as a breeze
Posts: 292
Joined: Tue Dec 29, 2015 6:19 am
Location: Germany

Re: Slow transfer speed due to SSD cache acceleration

Post by okenny »

QTier 2 sounds promising, any idea when 4.3.4 could be available?
QNAP TVS-1282-i7-7700K-40G with Corsiar H5 SF watercooling, 3x 80mm PWM Noctua fans, 1x 60mm PWM Noctua fan, Corsair SF450 PSU
QNAP TVS-872XT i9-9900k 64GB, Noctua Cooler
TS-653B 16GB
User avatar
jds580s
Know my way around
Posts: 206
Joined: Tue Dec 08, 2009 4:52 am

Re: Slow transfer speed due to SSD cache acceleration

Post by jds580s »

okenny wrote:QTier 2 sounds promising, any idea when 4.3.4 could be available?
I’m not exactly sure, but Qnap showed it as “coming soon” at the end of August.

The release notes for the current Beta version are here.
https://www.qnap.com/en/releasenotes/in ... t_choose=5

In addition to the Qtier changes they also say that
The SSD cache flush algorithms have been updated for better caching performance when under high load.
Perhaps that will also help for those that don’t want to move to Qtier.

They have scheduled a live broadcast about the 4.3.4 Beta On Nov 1st
https://www.qnap.com/event/2017/qts434-sneakpreview/en/
Perhaps some more details or target release date will come out then.

Model: TVS-1282-i5-16G

[list]
[*]Firmware: QTS 4.5.3.1652 build 20210428
[*]Network: 10GbE ASUS XG-C100C card, MTU 9k
[*]RAID 1: [System] 2x WD Blue M.2 SSD 250GB
[*]RAID 6: [DATA] 5x HGST HDN728080ALE604 8TB
[list]
[*] Qtier RAID 1: 2x Samsung SSD 850 EVO 500GB + 2x Samsung SSD 860 EVO 500GB
[*] Cache RAID 1: 2x Samsung SSD 960 EVO 500GB NVMe M.2 in two NGFF PCIe 3.0 x4 adapter cards[/list][/list]
Model: TS-459 Pro
[list]
[*]Firmware: QTS 4.2.6 build 20210327
[*]RAID 6: 2x HGST HDN724030ALE640 3TB, x2 Seagate ST3000VN000 3TB
[*]External: 4TB HGST eSATA Drive, UPS
[*]Network: 1 Gbps, MTU 1500[/list]
[/color]
vikino
Know my way around
Posts: 131
Joined: Sun Jul 10, 2016 3:06 am

Re: Slow transfer speed due to SSD cache acceleration

Post by vikino »

Well...that it is how does it work...
Yesterday i installed QTS 4.3.4.0370 beta...and result? STILL THE SAME!
Started copying from 6x HDD (RAID5, Thin volume) with RAID 10 cache in 4x500GB Samsung ssd for write/read
Copying to 2x Samsung 960Pro 1TB in RAID 0 Static volume
Speed peaks to 380MB/s and after few seoncds it drops to 60-70MB/s...What a speed of light...
QTS 4.3.4 has got speed test and it looks all fine, see attachements.

Raid group 1 = 6x HDD RAID5
Raid group 2 = SSD cache RAID 10
Raid group 3 = 2x Samsung 960 in QM2 card RAID0


okenny wrote:I have 7 WD REDs RAID6 with 2x 500GB Samsung EVO SSDs as Cache in the m.2 slots.
I noticed when copying to or from the HDDs with read-write enabled performance would drop quite a lot after a short priod of time.
When I saw it happen it was exactly what I had read here, I comnverted back to read-only cache instead. Read only seems ok
You do not have the required permissions to view the files attached to this post.
okenny
Easy as a breeze
Posts: 292
Joined: Tue Dec 29, 2015 6:19 am
Location: Germany

Re: Slow transfer speed due to SSD cache acceleration

Post by okenny »

disappointing.....are you brave enough to try QTier 2.0?
QNAP TVS-1282-i7-7700K-40G with Corsiar H5 SF watercooling, 3x 80mm PWM Noctua fans, 1x 60mm PWM Noctua fan, Corsair SF450 PSU
QNAP TVS-872XT i9-9900k 64GB, Noctua Cooler
TS-653B 16GB
cryptochrome
Been there, done that
Posts: 577
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome »

I can confirm the results Vikino has seen in 4.3.4. What is interesting though is that the resource monitor no longer shows extremely high IOPS latency when the speed crashes. I was always able to see between 20.000 and 50.000 ms latency in the resource monitor while copying large amounts of files "through" the cache. Whenever it hit those ridiculous amounts of latency, the speed crashed down. Now the speed is still crashing, but IOPS latency looks fairly normal (between 20 and 100 ms).

Needless to say (and this is directed at QNAP): Ignorance is bliss.
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: Slow transfer speed due to SSD cache acceleration

Post by Trexx »

I am not sure how more plainly Qnap can state this as of late. Cache Acceleration is for Short duration, RANDOM I/O acceleration.

Large File copies between volumes are LONG SEQUENTIAL I/O.


Sent from my iPad using Tapatalk
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
cryptochrome
Been there, done that
Posts: 577
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome »

Trexx wrote:I am not sure how more plainly Qnap can state this as of late. Cache Acceleration is for Short duration, RANDOM I/O acceleration.

Large File copies between volumes are LONG SEQUENTIAL I/O.
No. It isn't just for random IO. They are using Facebook's Flashcache. It is very well equipped to handle sustained throughput of sequential IO. QNAP even expose the option in the UI. Not just when setting it up does the wizard ask you whether you want to tune the cache for random or sequential IO, even after it was set up the option is available in the cache settings. Why would they offer it if it doesn't work?

That's not the point though. The cache doesn't work with random small-block sized IO either. As long as you fill the cache quickly enough, it comes crashing down, no matter if random or sequential IO. Even if you set the block size to 256 KB (the smallest possible setting) it becomes slow. Much slower in fact than if you are not using a cache at all.

The problem is most likely tied to how and when they flush the cache (e.g. write back to target disk).
vikino
Know my way around
Posts: 131
Joined: Sun Jul 10, 2016 3:06 am

Re: Slow transfer speed due to SSD cache acceleration

Post by vikino »

Hi again,
my test were after flushing cache, so 1TB cache was clean and ready to use.
Bellow printscreen is read/write speed while flushing cache to disk after it was disabled - i would expect much higher transfer rates and it constansly flushes slower and slower, ending at 15MB/s, WTF?! but...it is only cache flushing.

Third printscreen copying exactly the same file to same location with cache disabled - 600MB/s no problem...

Nice, isnt it?
Synology RS815+ costs 1/10 price of this crap and works faster...
You do not have the required permissions to view the files attached to this post.
vikino
Know my way around
Posts: 131
Joined: Sun Jul 10, 2016 3:06 am

Re: Slow transfer speed due to SSD cache acceleration

Post by vikino »

okenny wrote:disappointing.....are you brave enough to try QTier 2.0?
If i can revert it back and disable Qtier then it would be not problem, but i havent found any signs to be able to downgrade, and as there are all my data then i affraid of...
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: Slow transfer speed due to SSD cache acceleration

Post by Trexx »

vikino wrote:
okenny wrote:disappointing.....are you brave enough to try QTier 2.0?
If i can revert it back and disable Qtier then it would be not problem, but i havent found any signs to be able to downgrade, and as there are all my data then i affraid of...
There is no downgrade option. You would have to backup and rebuild the storage pool.


Sent from my iPhone using Tapatalk
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
vikino
Know my way around
Posts: 131
Joined: Sun Jul 10, 2016 3:06 am

Re: Slow transfer speed due to SSD cache acceleration

Post by vikino »

Trexx wrote:
vikino wrote:
okenny wrote:disappointing.....are you brave enough to try QTier 2.0?
If i can revert it back and disable Qtier then it would be not problem, but i havent found any signs to be able to downgrade, and as there are all my data then i affraid of...
There is no downgrade option. You would have to backup and rebuild the storage pool.


Sent from my iPhone using Tapatalk
Thats what i thought so unusable for me...
okenny
Easy as a breeze
Posts: 292
Joined: Tue Dec 29, 2015 6:19 am
Location: Germany

Re: Slow transfer speed due to SSD cache acceleration

Post by okenny »

Trexx wrote:I am not sure how more plainly Qnap can state this as of late. Cache Acceleration is for Short duration, RANDOM I/O acceleration.

Large File copies between volumes are LONG SEQUENTIAL I/O.


Sent from my iPad using Tapatalk

But Trexx it shouldn't be slower with cache enabled, should it? Would you expect this from other platforms?
QNAP TVS-1282-i7-7700K-40G with Corsiar H5 SF watercooling, 3x 80mm PWM Noctua fans, 1x 60mm PWM Noctua fan, Corsair SF450 PSU
QNAP TVS-872XT i9-9900k 64GB, Noctua Cooler
TS-653B 16GB
robinqu
New here
Posts: 9
Joined: Sun Oct 22, 2017 7:57 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by robinqu »

I'm a 1282T3 owner, and SSD Cache acceleration **.

* 1282T3, QTS 4.3.3
* Seagate 10GB Enterprise Capcacity v6 * 8, static volume on RAID6;
* Samsung 850 EVO M.2 SATA 500GB * 2 as RAID0 read/write cache, LRU, non-blocksize-limit.


* about 250MB/s seq write speed over Thunderbolt
* about 110Mb/s for server-side benchmarks: dd if=/dev/zero of=/share/Storage/test1.img bs=4G count=1 conv=fsync
vikino
Know my way around
Posts: 131
Joined: Sun Jul 10, 2016 3:06 am

Re: Slow transfer speed due to SSD cache acceleration

Post by vikino »

robinqu wrote:I'm a 1282T3 owner, and SSD Cache acceleration **.

* 1282T3, QTS 4.3.3
* Seagate 10GB Enterprise Capcacity v6 * 8, static volume on RAID6;
* Samsung 850 EVO M.2 SATA 500GB * 2 as RAID0 read/write cache, LRU, non-blocksize-limit.


* about 250MB/s seq write speed over Thunderbolt
* about 110Mb/s for server-side benchmarks: dd if=/dev/zero of=/share/Storage/test1.img bs=4G count=1 conv=fsync
Seems also pretty slow...
Post Reply

Return to “System & Disk Volume Management”