Slow transfer speed due to SSD cache acceleration

Questions about SNMP, Power, System, Logs, disk, & RAID.
Post Reply
cryptochrome
Been there, done that
Posts: 577
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome »

nope, still an issue. QNAP don't even confirm that the issue exists.
Tsiox
New here
Posts: 2
Joined: Thu Dec 15, 2016 2:42 am

Re: Slow transfer speed due to SSD cache acceleration

Post by Tsiox »

I've been struggling with this one for months. TVS-1271U's being used for iSCSI disk-to-disk backups (mix of traffic). They have been very respectable for the year or so that they've been installed. Wanted improve their IOPS performance, so installed 500 GB SSD's into the internal cache slots. As soon as we turned on the SSD caching (no bypass block size, we wanted all writes cached), their performance dropped to 1/3rd and the disk latencies being reported by VMware went through the roof. We've had the same experience as everyone above, anything above 4 MByte block sizes, the IOPS drop and the disk latencies hit the 100k's us. We've dropped it to 1 MByte block, but that doesn't show much benefit.

We're about to rebuild one of the QNAPs, I'll put in a ticket then, and I'll have a few days to do performance testing.
Tsiox
New here
Posts: 2
Joined: Thu Dec 15, 2016 2:42 am

Re: Slow transfer speed due to SSD cache acceleration

Post by Tsiox »

I believe I've worked out the details of why the SSD's cause a bump up in disk latency. In my testing, it's not a issue with the hardware, it's how Linux works. I've gained a lot more respect for these boxes in my testing, they're capable of quite a bit more than what most people give them credit for.

The TVS-1271U's obviously run Linux. A SSH to them gives a very familiar shell command line, and most of the typical Linux commands are available.

I know a few things about Linux file system caching. You can Google "Linux dirty_ratio", and that should lead you to documentation that should give a better explanation of the individual tuning parameters.

I believe what is happening is that the write caching component in Linux is going to "blocking" based on one of the criteria which will cause that condition. This can't be avoided completely, but it can be tuned to the point where the impact can be minimized.

Here is my very short rendition of the Linux file system write caching parameters. There are people that can probably give more technically correct terms, but I can give a decent overview. All of these parameters can be found in /proc/sys/vm

In Linux, writes can be normal (background) or blocking. The objective, in this case, is to avoid any condition that causes writes to become "blocking". If the Linux file caching system goes blocking, all other file system interactions are put on hold until the entire contents of the write cache ("dirty" cache) are written to disk.

The following control the blocking behavior of storage writes.

dirty_bytes - How many bytes can be in the "dirty" cache before the file system goes blocked. Mutually exclusive with dirty_ratio.
dirty_ratio - Ratio of file system cache memory that can be dedicated to "dirty" writes before the file system goes blocked. Mutually exclusive with dirty_bytes.
dirty_expire_centisecs - Hundredths of seconds (100 = 1 second) that unwritten data can be "dirty" before the file system goes blocked.

Also, sync/fsync/volatile write cache/FUA will cause the file system to go blocked every time they are activated. All interaction with storage stops until the data in the dirty cache is written out to storage, this affects all hosts on a clustered iSCSI connection.

Normal writes are controlled by the following parameters.

dirty_background_bytes - Number of bytes of data in the "dirty" cache before they will be written out to storage in the background. Mutually exclusive with dirty_background_ratio
dirty_background_ratio - Ratio of the file system cache memory that can be dedicated to "dirty" writes before it will be written in the background. Mutually exclusive with dirty_background_bytes.
dirty_writeback_centisecs - How often the "dirty" cache is checked to determine if enough data exists to be written to storage. Lower numbers, checked more often, higher numbers, checked less often. (100 = 1 second)

Now, my theory is this. With the SSD's in place, enough data is being "received" that it's causing the file system to go blocked. Whether that's from receiving too much data in total (exceeding dirty_ratio), sitting in cache too long (exceeding dirty_expire_centisecs), or having a great deal of data in cache/SSD when a fsync is received and the fsync is being honored (volatile write cache).

Because the SSD's could have GB's of data stored up, they send this data at the storage "all at once", causing the storage to go blocked. This seems to be consistent with my testing. Here are the parameters I'm using.

cat 50 > /proc/sys/vm/dirty_ratio
cat 1 > /proc/sys/vm/dirty_background_bytes
cat 6000 > /proc/sys/vm/dirty_expire_centisecs
cat 100 > /proc/sys/vm/dirty_writeback_centisecs

With the above parameters, I've been testing with 10 GByte files written out via iSCSI followed by a sync, with the QNAP set to honor sync, and the performance is consistent with what you might expect for this type of configuration. The SSD cache block size is configured for "none".

The QNAPs begin to write out the data almost immediately (1 second writeback, and above 1 byte dirty cache), when the sync hits, the amount of data in write cache is significatly less than if the QNAP is using the Linux defaults, so the effect to storage latency is much less. Overall impact to performance is significantly better in this case. Additionally, I believe this configuration is more conservative ("safer" probably isn't the best term).

The drawback is that this configuration causes more activity on the array, and will probably reduce the life of the drives due to the additional activity.

This is my working theory, nothing more. QNAP would have to explain how the SSD cache interacts with the storage cache to get a correct interpretation of the issue.
cryptochrome
Been there, done that
Posts: 577
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome »

QNAP should hire you. They would have a pro on the team for a change.

Thanks for finding this. I will dig into this tomorrow. Good stuff.
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: Slow transfer speed due to SSD cache acceleration

Post by Trexx »

In 4.3, and not sure if in earlier versions, they appear to be using Flashcache for the SSD Acceleration.

You can see some of the data via the following:

Code: Select all

[~] # dmsetup status
cachedev1: 0 12947120128 flashcache stats:
	reads(231), writes(2044)
	read hits(227), read hit percent(98)
	replacement(0), write replacement(0)
	invalidates(59)
	pending enqueues(1074), pending inval(442)
	no room(0)
	disk reads(807643), disk writes(314569) ssd reads(1636317) ssd writes(9670640)
	uncached reads(545), uncached writes(6409), uncached IO requeue(5)
	disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
	uncached sequential reads(0), uncached sequential writes(0)
	lru hot blocks(242880), lru warm blocks(242942)
	lru promotions(86249), lru demotions(0)
Right now I don't have SSD Cache enabled (working on testing QTier), but this may help with some of your testing as well.
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
cryptochrome
Been there, done that
Posts: 577
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome »

Hm, that's really weird Paul. It appears to be using flashcache on 4.2 as well. I was sure they were using dm-cache before. I was probably wrong.
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: Slow transfer speed due to SSD cache acceleration

Post by Trexx »

There are a few counters that are exposed for FlashCache for easier monitoring.

Just as a disclaimer, I am resetting up my cache (was doing some QTIer testing) so right now it is still in Raid Resync.

Monitoring Errors:

Code: Select all

[/] # cat /proc/flashcache/CG0/flashcache_errors
disk_read_errors=0 disk_write_errors=0 ssd_read_errors=0 ssd_write_errors=0 memory_alloc_errors=0
thin_alloc_errors=0
ssd_errors=0
verify_errors1=0
verify_errors2=0
verify_errors3=0
verify_errors4=0
verify_errors5=0
verify_errors6=0
verify_errors7=0
Monitoring IO Size History:

Code: Select all

[/] # cat /proc/flashcache/CG0/flashcache_iosize_hist
1:0 2:0 4:64111 8:1722 16:265 32:72 64:22 128:144 256:74 512:775 1024:0
I am assuming those are in KB (so 1KB, 2KB, 4KB, 8KB, etc.)

Monitoring Cache Stats:

Code: Select all

[/] # cat /proc/flashcache/CG0/flashcache_stats
raw_reads:	1548
raw_writes:	68208
raw_read_hits:	0
raw_write_hits:	67149
raw_dirty_write_hits:	64986
reads:	100
writes:	2817
read_hits:	0
read_hit_percent:	0
write_hits:	2790
write_hit_percent:	99
dirty_write_hits:	2742
dirty_write_hit_percent:	97
replacement:	0
write_replacement:	0
write_invalidates:	0
read_invalidates:	0
direct:	24
fbc_busy:	0
fbc_inval:	0
defer_bio:	0
zero_sized_bio:	0
pending_enqueues:	0
pending_inval:	0
metadata_dirties:	28342
metadata_cleans:	298
metadata_batch:	21239
metadata_ssd_writes:	7401
cleanings:	298
fallow_cleanings:	24
full_cleanings:	6
partial_cleanings:	18
no_room:	0
front_merge:	171
back_merge:	67
disk_reads:	1548
disk_writes:	322
ssd_reads:	298
ssd_writes:	77133
uncached_reads:	0
uncached_writes:	24
uncached_IO_requeue:	0
uncached_sequential_reads:	0
uncached_sequential_writes:	0
total_blocks:	485888
cached_blocks:	1219
dirty_blocks:	802
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
pokrakam
Know my way around
Posts: 101
Joined: Wed Sep 17, 2008 5:27 am

Re: Slow transfer speed due to SSD cache acceleration

Post by pokrakam »

Interesting, I'm experiencing the same issue, discussed over here: viewtopic.php?f=180&t=123218

Do others on this thread also see tons of dm-kcopyd messages?

Code: Select all

[2086344.074480] dm-kcopyd track job out of array
(Login via SSH, run dmesg)

I have been working with QNAP support for a few weeks and they have been good, even though they haven't found the cause yet.

Also, what SSDs are folks using? I'm using two crucial MX300's, unfortunately not on the supported list (new-ish at the time and didn't want to wait around to see if they made the list. I figured as they had older models on the list it's worth a small risk).
Performance is much improved when the cache is initially activated and synched - I can run a VM over the network with fairly decent performance.
After a while it all goes to pot though, and there are loads of dm-kcopyd messages. It's possibly related to a filling cache, never seen bad performance when the cache showed under 70-80% full, but that could be coincidence.
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: Slow transfer speed due to SSD cache acceleration

Post by Trexx »

pokrakam wrote:Interesting, I'm experiencing the same issue, discussed over here: viewtopic.php?f=180&t=123218

Do others on this thread also see tons of dm-kcopyd messages?

Code: Select all

[2086344.074480] dm-kcopyd track job out of array
(Login via SSH, run dmesg)

I have been working with QNAP support for a few weeks and they have been good, even though they haven't found the cause yet.

Also, what SSDs are folks using? I'm using two crucial MX300's, unfortunately not on the supported list (new-ish at the time and didn't want to wait around to see if they made the list. I figured as they had older models on the list it's worth a small risk).
Performance is much improved when the cache is initially activated and synched - I can run a VM over the network with fairly decent performance.
After a while it all goes to pot though, and there are loads of dm-kcopyd messages. It's possibly related to a filling cache, never seen bad performance when the cache showed under 70-80% full, but that could be coincidence.
I just Installed (2) m.2 mx300's in mine and they are on supported list for the x73 family, so should be making to others soon. Not sure if 4.3 has any impact on that.

Which cache mode did you choose, random or sequential?


Sent from my iPad using Tapatalk
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
cryptochrome
Been there, done that
Posts: 577
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome »

Hey Pokrakam, where do you see these dm-kcopyd messages? I want to check if I have them too, but wouldn't know where to find them.

Also, I am using Samsung EVO 850 SSDs in the M.SATA slots for the cache.
pokrakam
Know my way around
Posts: 101
Joined: Wed Sep 17, 2008 5:27 am

Re: Slow transfer speed due to SSD cache acceleration

Post by pokrakam »

To see the messages, login to the NAS via ssh. On a mac you can do it using a terminal:
ssh admin@servername
On windows I suggest using PuTTY - just google putty ssh and download.

As to cache modes, I use sequential but have tried all combos.
cryptochrome
Been there, done that
Posts: 577
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome »

Yes, but which log file?
User avatar
Trexx
Ask me anything
Posts: 5393
Joined: Sat Oct 01, 2011 7:50 am
Location: Minnesota

Re: Slow transfer speed due to SSD cache acceleration

Post by Trexx »

cryptochrome wrote:Yes, but which log file?
I think he said t was in Dmesg.


Sent from my iPad using Tapatalk
Paul

Model: TS-877-1600 FW: 4.5.3.x
QTS (SSD): [RAID-1] 2 x 1TB WD Blue m.2's
Data (HDD): [RAID-5] 6 x 3TB HGST DeskStar
VMs (SSD): [RAID-1] 2 x1TB SK Hynix Gold
Ext. (HDD): TR-004 [Raid-5] 4 x 4TB HGST Ultastor
RAM: Kingston HyperX Fury 64GB DDR4-2666
UPS: CP AVR1350

Model:TVS-673 32GB & TS-228a Offline[/color]
-----------------------------------------------------------------------------------------------------------------------------------------
2018 Plex NAS Compatibility Guide | QNAP Plex FAQ | Moogle's QNAP Faq
cryptochrome
Been there, done that
Posts: 577
Joined: Sat Mar 09, 2013 7:26 pm

Re: Slow transfer speed due to SSD cache acceleration

Post by cryptochrome »

Where do you guys set random vs. sequential cache mode? Is that a 4.3 thing? I don't have such an option on my 4.2 box.
pokrakam
Know my way around
Posts: 101
Joined: Wed Sep 17, 2008 5:27 am

Re: Slow transfer speed due to SSD cache acceleration

Post by pokrakam »

dmesg is the command to list the kernel message buffer. The usual Linux /var/log/messages is empty on the QNAPs, but dmesg will give you the last few hundred or so.

The random/sequential terms are only used when setting up the cache. It's the 'Bypass block size' dropdown in the SSD cache setting, and the info button on there talks about it. Bypass 0 means cache everything = good for sequential IO (large files). Bypass 1MB means don't cache any requests over 1MB, good for random access (VM, database, anything that does many small operations).
Small requests always incur a latency overhead from a spinning disk, thus 100 random reads of 10k will be much much slower than 1 read of 1MB. So, depending on your usage patterns, you could e.g. make better use of cache capacity by setting it to buffer more of the expensive operations instead of filling it with a 5GB movie that streams just fine without a cache.
Post Reply

Return to “System & Disk Volume Management”