TVS-473: m.2 cache for virtualization

[FogDucker]

Assuming I had a TVS-473 with 4 disks in raid 5 and 32GB of memory.

Would adding a 250GB raid1 setup for read/write caching give me performance boost for VMs and not just regular file access?

Or, would I be better to get larger m.2s and run my VMs in their entirety off that raid 1 set?

For me, I'd like to keep it simple and just use m.2 as a read/write cache. I'm just unclear if there is virtual machines can benefit in the same was as the regular file system.

+ For example: Does caching not work as well if a VM image [say, a 500GB Windows machine] was larger than the cache? [ie: 256GB in raid 1]. Or, does virtualization work in a way that individual files on the guest OS can use caching?

--
Marc

[aarbee]

I think that the Virtual Machine is 1 file. And if that file is really 500GB, I do not think that it will fit into the M.2 of 250GB.
I for instance started a Windows machine at 160GB, and the system says that the image file is about 10GB big. That would fit on the Cache.
Somehow I remember a sentence somewhere, that the cache is more meant for smaller files. Not sure if that is the case.

[FogDucker]

Hmm. So the only way to gain performance with virtualization is to disable caching and instead use SSD drives as a pool and install the guest on those fast disks? Or, ensure that the guests are always smaller than the total cache size? As soon as the guest machine becomes bigger than the cache size, it skips cache altogether? That's hitting a wall!

So the guest machine file is simply the consumed space, in the case of Windows, pagefile.sys and all, correct?

cache > [total vm files] + [commonly used filesystem] + [apps] ?

@Rob: Can you share your configuration and experiences running guests on your TVS-473?

--
Marc

[aarbee]

I only have 2 guest OS'ses running. Linux Mint at 4GB and Windows Home Server at 8GB and 160GB disk.
My 673 config is in my signature.

I do not really know how the cache works. But it is more logical for smaller files.

[Trexx]

Hmm. So the only way to gain performance with virtualization is to disable caching and instead use SSD drives as a pool and install the guest on those fast disks? Or, ensure that the guests are always smaller than the total cache size? As soon as the guest machine becomes bigger than the cache size, it skips cache altogether? That's hitting a wall!

So the guest machine file is simply the consumed space, in the case of Windows, pagefile.sys and all, correct?

cache > [total vm files] + [commonly used filesystem] + [apps] ?

@Rob: Can you share your configuration and experiences running guests on your TVS-473?

--
Marc

Incorrect! SSD Caching works on a block level basis. So if you have m.2 cache that is say 225GB usable, but your VM Disk is say 300GB, you can still get benefit from the SSD cache from a read/write perspective. And remember the SSD cache applies to the entire VOLUME (VM files and others). The recommendation is to have the m.2 cache configured for larger block sizes for VM usage tuning, versus no block size restriction for more general use.

A different option is to configure the m.2's as a storage pool/volume. Then it is just another potential storage target that you can save the VM files on (just like spinning disk volume). That use case you would have to have your VM files be smaller than your m.2 Volume size.

[aarbee]

@Trex, you have more knowledge then I have on the Qnaps.

@Fogducker, I just saw, while I was doing a complete filebackup of a website of mine, from the internet to the 673, that the cache had a write hit rate of 70%. Odly I see more write hits than reads.
Both virtual machines I have got 2 cores. The Linux 4GB and the Windows 8GB of internal memory. Diskspace for the linux is 50GB. The Windows machine has 160GB, as that appeared to be the minimum.
And you should not forget to install the Guest tools in the windows VM. (Credits to Trexx), It makes a big difference, wether you run on the default drivers, or on the virtio drivers of Fedora. Not sure how valid they are, but these seems to be the latest https://fedorapeople.org/groups/virt/vi ... downloads/.

Just read a German forum, where somebody tested with and without Cache on a 253, where the cache was a single 2.5 SSD, and the first disk was 3TB disk, that there was no real performance gain. Smallnetbuilders did do the same test.
I can understand that it does not really work that fast, in case you are not having a heavy loaded NAS. However if the NAS is having a lot of traffic, I can imagine, that using the cache or not, makes a big difference.

[P3R]

Read cache: useful for repeated reads of the same data. Will be useful for NAS-internal traffic like when running VMs and/or having heavy random data access but pretty much useless for huge sequential data access like when streaming media.
Write cache: will be used all the time when writing to the NAS but again not very useful for large sequential writes as the data need to be flushed to mechanical disks anyway when the cache is filled.

Unless having an end-end 10 GbE network with fast clients, a RAID array of mechanical is often not a performance bottleneck (at least not as often as many seem to think today) and if the RAID isn't a bottleneck, a disk cache doesn't make much difference. At least not a positive difference.

[Trexx]

Read cache: useful for repeated reads of the same data. Will be useful for NAS-internal traffic like when running VMs and/or having heavy random data access but pretty much useless for huge sequential data access like when streaming media.
Write cache: will be used all the time when writing to the NAS but again not very useful for large sequential writes as the data need to be flushed to mechanical disks anyway when the cache is filled.

Another potential use case that I see P3R the cache makes a big difference in MySQL/Kodi (HD Station) performance. Likely due to the read caching of things.

[P3R]

Another potential use case that I see P3R the cache makes a big difference in MySQL/Kodi (HD Station) performance.

Well you have the real life experience I lack but being NAS internal (so not limited by a network bottleneck) and probably fairly disk intensive, it sounds logical that those are other scenarios where caching could be useful.

I was more focusing on when SSD caching isn't of much use because I think the current trend is so very sad where so many today overprovision their new NASes with ridiculous amounts of RAM, extreme CPUs and expensive SSDs. Yet they still don't buy the basic and really important stuff like a UPS for every data storage and arrange with proper backups on fully independent systems placed at remote locations.

Data is still lost as often today as is used to be but now from much more expensive NASes. It's such a waste of resources and beautiful technology...

[aarbee]

Maybe because a lot of people do not see the advantage of a UPS?
Neither did I. Mainly because of a wrong configered UPS-Server connection, because I thought that 500VA was 500 Watt/hour. So my Server using 100 watt/hour, could run 5 hours, according to me.
The truth was that it dropped out after 15-30 minutes. And I had to rebuild the 2-disk Raid anyway. And that was never a problem.
Thanks to several people on this forum, and especially Trexx, I know a lot more, and considering the fact that bigger disks and more disks, give more read errors, made me decide to buy a new one. APC 700. I am happily awaiting the device.

And several people in my neighberhood think in that way. UPS cost money to buy, uses electricity and and brings nothing in return. So why should I buy/use one then?

[xavierh]

Maybe because a lot of people do not see the advantage of a UPS?
Neither did I. Mainly because of a wrong configered UPS-Server connection, because I thought that 500VA was 500 Watt/hour. So my Server using 100 watt/hour, could run 5 hours, according to me.
The truth was that it dropped out after 15-30 minutes. And I had to rebuild the 2-disk Raid anyway. And that was never a problem.
Thanks to several people on this forum, and especially Trexx, I know a lot more, and considering the fact that bigger disks and more disks, give more read errors, made me decide to buy a new one. APC 700. I am happily awaiting the device.

And several people in my neighberhood think in that way. UPS cost money to buy, uses electricity and and brings nothing in return. So why should I buy/use one then?

hope that they don't end up having to rebuild an array from scratch because of a power failure, or lost a motherboard because of that. a UPs for me is an essential piece of equipment

[aarbee]

If you have a good backup. Or a Raid1 is beging rebuild within a few hours. I had the same reasoning. Yet now that I have the 673 with 4-5 disks in it, eventhough small, made me think about it.
As I tried to move Raid5-Raid6 in 2 evenings, without an issue... where is the problem? But if that happens a few days behind each other, is different cake.

[Trexx]

The critical thing people miss is that the purpose of the UPS is to prevent the ungraceful shutdown/corruption of the NAS due to either complete power loss, or brown-out.

Either of those conditions should they happen during a disk write can result in best case a garfed file, in worse case a corrupted RAID set. Thing about the MANY hours of time that it would take to rebuild the array, restore your apps, configure them by hand, restore all your data (assuming you were even bothering to do backups), etc. You are probably out of the hours category and into potentially DAYS.

That amount of lost free time is worth much more than the $100 I spent for a 1300va UPS.

[aarbee]

I see I am a more minimalistic guy.
Smaller disks, smaller SSD's, smaller UPS.

[FogDucker]

I was happy to read that caching is block level, but am surprised to read that caching didn't result in noticeable performance gains for guest operating systems. I suppose that once loaded, there really aren't repeated reads and that it might help a little with random writes.

A couple of more questions about virtualization: If I provision a 200GB VM, does that immediately create a 200GB file or will it simply grow up to the maximum of 200GB? Can I provision the VM on spinning disks today and move it to a m.2 volume later? Are there any drawbacks to running VMs on a m.2 volume? Does wear or heat play a role when using m.2 like this?

On the note of using UPS, I fully agree and have 3 in my home alone. I've always found the tricky part is getting the UPS to notify the devices to gracefully shutdown before the batteries run out.

My TVS-473 was delivered today - I can't wait to start using it so that I can retire my ~8 year old PC/server! The 16GB sticks are en route, but I think I'll wait on the m.2 until I get a better feel for performance.

Thank you all for your valuable input. It's much appreciated and ultimately resulted in me buying into the QNAP platform!

--
Marc