New TVS-473 built- so now what

[DarfNader]

I am a brand new QNAP user with about 20 years sysadmin/DevOps experience so I see myself as someone who will become an advanced user once I get the feel for this seemingly "magic bullet" of a device, but first I need to get the storage set up to best suit my needs so I can embark on my long list of applications that I wish to use it for so I can start removing my subscriptions for cloud-based or other services which I hopefully won't need.

So before I start setting stuff up, I need to start by setting up my storage and cache to best suit my needs, but I as still fuzzy on the details of how the storage model works. While it looks like the most flexible approach is to make your entire HDD array (I have 4 x 6TB 7200 rpm drives) in a reliable manner (I use RAID 5) and then slice it up a "virtual JBOD" so that storage is purposed out as needed and all speed and reliability provided by RAID 5 is done behind the scenes. The part I am having trouble deciding is how to best use my m2.SSD modules. I have 2 x 1 TB modules installed and they are kitted out with aftermarket copper heat dissipators so I'm ready to go. Question is, how? Do I put it all in QTier? Do I use some for SSH cache first, and if so, how much? There are so many options I am getting choice paralysis so I wanted to get input from the community based on the application profile that I am about to list. I realize this is a LOT of text and I don't expect the world to just read my plea and solve all of my problems, but if you have experience in a specific area, I'd love to hear what your impressions are and what advice you'd offer and why.

Please let me explain myself...

I figure the best place to start is to explain what I want to use this device for. While I see using the QNAP apps as not only an easy way to get up and running some specific functions, there appears to be some cases where I really don't have a choice because of the hardware limitations, specifically with video. Therefore, out of the gate, I see the need for doing some things natively and some things in VMs or even in Docker containers. I do have experience with that on other platforms, so assuming the same principles apply I should be able to work with it.

So without further ado...

I want to have my QNAP NAS also be a:
============================
- Set-top entertainment box: I understand that GPU acceleration only works in native mode and not within a VM. I have been advised to try out Kodi as my browser/media player, and while that's okay I guess, I would prefer to use a native Plex media player as well as separate media server as I already use it, pay for Plex pass, and really like it, but I may simply have to accept that is not an option as a native player on QNAP. Hoping for good news...
- Media Server: So many options here! I tired to get the Plex Media Server running just to see what it looks like and I couldn't even get the management page to load, so I'm not sure what's going on there. Anyway, I know that the QNAP store has multiple DLNA-based media servers for various media types so if people with experience with Plex and these apps can weigh the pros and cons, that'd be amazing.
- Music Server: Granted, this's normally would fall under "Media Server" but since there is some sort of iTunes music app for QNAP I am wondering if I should be tasking this out separately. Plex has support to scan iTunes libraries but that's on my Mac and I would love to get as much off my desktops/laptops on onto the QNAP as possible.
- Backup Server: I presently have a complex backup strategy but it has me covered. I use my desktop mac as a Time Machine server to back itself up and my laptop onto a pair of 4 TB external USB 3 drives that I swap out every week so one is always offsite. I then run an app called ARC which stream critical files to Amazon Cloud (until my unlimited storage runs out in January). I want to get all of this onto QNAP and then have my QNAP backup onto cloud in case of catastrophe. It seems like QNAP has Time Machine covered, but I still need to find an affordable cloud provider where I can back all my stuff too. B2 seems like the best contender, but I am not seeing anything on QNAP that supports it.
- Personal Cloud/Personal File Storage: This also seems like a no brainer, and I would love to be able to stop paying Google and Apple when it looks like I can get QNAP to do that for me. I have years of documents that are well organized and I need them to be both accessible and well secured. This seems like QNAPs wheelhouse, but I am again overwhelmed by options.
- VPN: I already use VyperVPN, so the QNAP app seems like the obvious choice- I just have to understand how to route my traffic through it and take advantage. Then again, the traffic that I want to encrypt will probably be hosted on the QNAP because...
- PTP Filesharing: Yes, I may be the last holdouts who hasn't taking this elsewhere like usenet, but it still works for me. Call me old fashioned. Presently use µTorrent on mac but I see the QNAP has a "Downloader" tool as well. Not sure if that is the intended purpose. I could just as easily set up a small linux VM or Docker container to use Transmission if that is best.
- Network monitoring: While most of my activity will be running on this single gold box of all-knowing, it would be good to monitor internal traffic on my LAN and not just watch my firewall. The perimeter software in ASUS-WRT is okay, but I would like to know now about what's going on, especially since my router has been a little flakey lately
- Virus/Malware scans: While I loathe antivirus software because it slows systems down worse than most viruses and malware, I feel that with a centralized system like this you'd be asking for trouble to just assume all is well, especially using P2P software, even it is in a little corral. Malware has a way to getting around when the foolish insist on running it.
- Home Automation: This area is still in development and it started with building a universal remote on my old iPad using Roomie/Simple Control. This started out great, and I love their media guides, but vthe software is buggy, some of the design decisions are a mix of peculiar and dubious, and their support is only sort of there. Also, their "hub" software seems to be problematic as my TV now keeps turning on randomly since I started using it. I know there are a lot of players in this space and it goes beyond the realm of QNAP, but I am curious to see how wither integrate QNAP into their HA world. (I also use Philips Hue heavily but have yet to pull the trigger on other IoT devices yet.)
- Mail Server: I was just thinking of using the same postfix setup that I run on a t1.micro in AWS on a small VM or container, but I am open to new methods as I have been doing that for over a decade now.
- Web services/Wordpress/etc..: it seems like QNAP has a lot of toys in the box for DIY web hosting so this looks like just trial and error, and I am sure I will probably want to roll my own in a VM or with containers for anything more advanced, but this is not a priority. I am not a big blogger or anything. Who has the time!

So okay, that's my list so far and I am sure I will find more things I want to do with it, but these are the essentials, or at least the stuff at the top is. Also, I want to play around with the VM and docker, but that will surely come later.

So any input would be appreciated on this tall order! I first need to set up storage, it realize, so that comes first, but I would love to hear from y'all on how you like to do things. It's open season, so pick your topic!

Thanks in advance!
-Darf


Sent from my iPad using Tapatalk Pro

[Trexx]

I am a brand new QNAP user with about 20 years sysadmin/DevOps experience so I see myself as someone who will become an advanced user once I get the feel for this seemingly "magic bullet" of a device, but first I need to get the storage set up to best suit my needs so I can embark on my long list of applications that I wish to use it for so I can start removing my subscriptions for cloud-based or other services which I hopefully won't need.

So before I start setting stuff up, I need to start by setting up my storage and cache to best suit my needs, but I as still fuzzy on the details of how the storage model works. While it looks like the most flexible approach is to make your entire HDD array (I have 4 x 6TB 7200 rpm drives) in a reliable manner (I use RAID 5) and then slice it up a "virtual JBOD" so that storage is purposed out as needed and all speed and reliability provided by RAID 5 is done behind the scenes. The part I am having trouble deciding is how to best use my m2.SSD modules. I have 2 x 1 TB modules installed and they are kitted out with aftermarket copper heat dissipators so I'm ready to go. Question is, how? Do I put it all in QTier? Do I use some for SSH cache first, and if so, how much? There are so many options I am getting choice paralysis so I wanted to get input from the community based on the application profile that I am about to list. I realize this is a LOT of text and I don't expect the world to just read my plea and solve all of my problems, but if you have experience in a specific area, I'd love to hear what your impressions are and what advice you'd offer and why.

Please let me explain myself...

I figure the best place to start is to explain what I want to use this device for. While I see using the QNAP apps as not only an easy way to get up and running some specific functions, there appears to be some cases where I really don't have a choice because of the hardware limitations, specifically with video. Therefore, out of the gate, I see the need for doing some things natively and some things in VMs or even in Docker containers. I do have experience with that on other platforms, so assuming the same principles apply I should be able to work with it.

So without further ado...

I want to have my QNAP NAS also be a:
============================
- Set-top entertainment box: I understand that GPU acceleration only works in native mode and not within a VM. I have been advised to try out Kodi as my browser/media player, and while that's okay I guess, I would prefer to use a native Plex media player as well as separate media server as I already use it, pay for Plex pass, and really like it, but I may simply have to accept that is not an option as a native player on QNAP. Hoping for good news...
- Media Server: So many options here! I tired to get the Plex Media Server running just to see what it looks like and I couldn't even get the management page to load, so I'm not sure what's going on there. Anyway, I know that the QNAP store has multiple DLNA-based media servers for various media types so if people with experience with Plex and these apps can weigh the pros and cons, that'd be amazing.
- Music Server: Granted, this's normally would fall under "Media Server" but since there is some sort of iTunes music app for QNAP I am wondering if I should be tasking this out separately. Plex has support to scan iTunes libraries but that's on my Mac and I would love to get as much off my desktops/laptops on onto the QNAP as possible.

Will try and cover these all in one shot.

• Media Server:
• Direct HDMI Media playback (w/GPU Accel) - Use KODI via HD Station (see my signature for link to it). Now you can use a Plex Add-on for KODI that basically uses KODI for playback, but Plex for backend Media DB.
• Direct HDMI Media playback (w/o GPU Accel?) - Use PlexMediaPlayer (PMP) add for HD Station. I am not 100% positive if this will use GPU accel for playback or not.
• Using PlexMediaServer (non-QNAP client) - Basically using PlexServer with separate playback device. The x73 family will generally support (2) 1080p playback streams with transcoding simultaneously. If the Plex client supports direct play of the format, then no transcoding is involved. Download PMS qpkg from Plex website (either release or plexpass x64 QNAP version).
• Use DLNA server - can use QNAP's native app, Plex, Twonky (3rd party - $20 for license), etc. - Many options, but I tend to stick with Plex as they have a client for pretty much almost anything. Especially if you already have PlexPass.
  
  - I generally use KODI for local playback and Plex for remote (out of the home). Both point to same media, and I use Trakt.tv to keep them synced (although could use Plex Kodi Plugin too if I wanted).
  
  Music Sever:
  I moved my iTunes Library and DB over to QNAP. So when I open on my macbook pro, it is pointing at the QNAP for everything. I haven't pointed Plex at it, but I subscribe to Apple Music, so don't really have a huge need to.

- Backup Server: I presently have a complex backup strategy but it has me covered. I use my desktop mac as a Time Machine server to back itself up and my laptop onto a pair of 4 TB external USB 3 drives that I swap out every week so one is always offsite. I then run an app called ARC which stream critical files to Amazon Cloud (until my unlimited storage runs out in January). I want to get all of this onto QNAP and then have my QNAP backup onto cloud in case of catastrophe. It seems like QNAP has Time Machine covered, but I still need to find an affordable cloud provider where I can back all my stuff too. B2 seems like the best contender, but I am not seeing anything on QNAP that supports it.
- Personal Cloud/Personal File Storage: This also seems like a no brainer, and I would love to be able to stop paying Google and Apple when it looks like I can get QNAP to do that for me. I have years of documents that are well organized and I need them to be both accessible and well secured. This seems like QNAPs wheelhouse, but I am again overwhelmed by options.

Backup Server: Install Hybrid Backup Sync, and Cloud Backup Sync - This will cover your TM backup, backup to external USB drive, and cloud backups. There are other options for if you are looking more for continuous sync vs. more of a 1x per day/week backup sync or more specific backup providers, but this setup handles most peoples need.

Personal Cloud: - there are a bunch of different options. Some QNAP, some 3rd party community packages. QNAP would be Qsync which is more PC focused though (similar to dropbox, etc.) which has automated sync. They have an android phone app out for qsync, but not for IOS yet.

Another options is storing stuff on QNAP in shared folders and then accessing via QFile, QVideo, QMusic apps. Not so much automated sync, but more remote access anywhere.

With any of these options, you still want an external (ideally offsite) backups. RAID protects from HW failure, but isn't a substitution for Backups.

There are also other 3rd party packages such as OwnCloud, NextCloud,etc.

- VPN: I already use VyperVPN, so the QNAP app seems like the obvious choice- I just have to understand how to route my traffic through it and take advantage. Then again, the traffic that I want to encrypt will probably be hosted on the QNAP because...
- PTP Filesharing: Yes, I may be the last holdouts who hasn't taking this elsewhere like usenet, but it still works for me. Call me old fashioned. Presently use µTorrent on mac but I see the QNAP has a "Downloader" tool as well. Not sure if that is the intended purpose. I could just as easily set up a small linux VM or Docker container to use Transmission if that is best.

Multiple ways to solve... I happen to have setup this week a docker container that has Deluge PTP client & OpenVPN client all integrated into one. So all traffic goes via VPN, but they are tied together. Rest of QNAP is unaffected by it so get best of both worlds.

But you can do PTP client (multiple community packages -Transmissions, RTorrent, etc.) & QVPN Client from QNAP.

- Network monitoring: While most of my activity will be running on this single gold box of all-knowing, it would be good to monitor internal traffic on my LAN and not just watch my firewall. The perimeter software in ASUS-WRT is okay, but I would like to know now about what's going on, especially since my router has been a little flakey lately
- Virus/Malware scans: While I loathe antivirus software because it slows systems down worse than most viruses and malware, I feel that with a centralized system like this you'd be asking for trouble to just assume all is well, especially using P2P software, even it is in a little corral. Malware has a way to getting around when the foolish insist on running it.

Not a lot out for network monitoring... although I have seen people deploy UTM's via VM on QNAPs before (essentially replacing your ASUS functionality with a QNAP multi-nic VM) and then using ASUS just for wifi.

Virus/Malware - install QNAP's malware remover app, and enable build in antivirus (basically clamav). Better bet is having better scanning on client side (PC/Mac/etc.).

- Home Automation: This area is still in development and it started with building a universal remote on my old iPad using Roomie/Simple Control. This started out great, and I love their media guides, but vthe software is buggy, some of the design decisions are a mix of peculiar and dubious, and their support is only sort of there. Also, their "hub" software seems to be problematic as my TV now keeps turning on randomly since I started using it. I know there are a lot of players in this space and it goes beyond the realm of QNAP, but I am curious to see how wither integrate QNAP into their HA world. (I also use Philips Hue heavily but have yet to pull the trigger on other IoT devices yet.)

There are I think 2-3 different community apps for this. QNAP also added IFTTT support recently as well.

- Mail Server: I was just thinking of using the same postfix setup that I run on a t1.micro in AWS on a small VM or container, but I am open to new methods as I have been doing that for over a decade now.
- Web services/Wordpress/etc..: it seems like QNAP has a lot of toys in the box for DIY web hosting so this looks like just trial and error, and I am sure I will probably want to roll my own in a VM or with containers for anything more advanced, but this is not a priority. I am not a big blogger or anything. Who has the time!

Mail - either community QPGK, Docker Container, or VM...
Web Service - I tried doing this, and moved to a 3rd party hosted solution. Removes potential hacking target, and you don't want your website down every time you do firmware update, tinker, etc.

Storage Setup:

HDD's: Do a Raid-5, Storage Pool w/Thick Provisioning. Start with default 50% utilization (unless you must use more out of box) and save some room back for snapshot reserve. You can always expand volume your default volume after the fact, but you can't shrink them. Don't worry about Virtual JBOD. That only applies if you have multiple NAS's.

SSD's: You basically have 3 options... create a Raid-1 volume (use like HDD's), use for SSD Caching, or QTier (where QNAP moves data blocks between SSD & HDD).

For QTier there are a couple considerations. Currently it can only be setup when you create your volume... so you setup the SSD & HDD for QTier at the same time. Once you turn it on, there is no turning it off without starting over from scratch. Qtier works at the block level, so you may have parts of a file on SSD and other parts on HDD depending on access/usage patterns. Probably more beneficial for 10GbE environment than 1GbE unless you are using Port Trunking or "in box" usage as talked about below.

For SSD caching - mainly comes in handy for Random I/O and for heavily "IN BOX" applications (VM's, Docker, etc.). Otherwise 1GbE NW ports tend to be more limiting factor (~110MB/s throughput) than HDD throughput. SSD benefits increase quite a bit though when you move to 10GbE (via PCIe expansion).

For SSD Volume - would mainly use for things like Docker Containers, VM's, etc. When the majority of the traffic is staying "in the box" then SSD's can really have some advantages, OR you are running 10GbE then SSD Volumes can be an asset.


This will give you some things to start to think about anyway.

[DarfNader]

This will give you some things to start to think about anyway.

Wow, I'll say! That was fantastic! I really appreciate all the help you've given me as I break into this "new world" of purpose built home servers that service all kinds of devices. I will see how I do and report back. Thank again! I owe you a beer!


Sent from my iPad using Tapatalk Pro

[Trexx]

This will give you some things to start to think about anyway.

Wow, I'll say! That was fantastic! I really appreciate all the help you've given me as I break into this "new world" of purpose built home servers that service all kinds of devices. I will see how I do and report back. Thank again! I owe you a beer!


Sent from my iPad using Tapatalk Pro

Not a problem. Biggest thing to remember, as long as you have your critical data backed up, you can always erase and start over.

[DarfNader]

For QTier there are a couple considerations. Currently it can only be setup when you create your volume... so you setup the SSD & HDD for QTier at the same time. Once you turn it on, there is no turning it off without starting over from scratch. Qtier works at the block level, so you may have parts of a file on SSD and other parts on HDD depending on access/usage patterns. Probably more beneficial for 10GbE environment than 1GbE unless you are using Port Trunking or "in box" usage as talked about below.

For SSD caching - mainly comes in handy for Random I/O and for heavily "IN BOX" applications (VM's, Docker, etc.). Otherwise 1GbE NW ports tend to be more limiting factor (~110MB/s throughput) than HDD throughput. SSD benefits increase quite a bit though when you move to 10GbE (via PCIe expansion).

I have both a my ASUS RT-AC87R and a Netgear ProSafe switch which both can be configured to support trunking. The ASUS can take two gigE ports and and aggregate them to make them 2 GigE total, so if I trunk two of the QNAP NICs and put a second NIC in my workstation that would allow me better storage performance, but alas, it'd still be a bottleneck as you said which would reduce the value of QTier if that as my sole purpose for it. However though, as you pointed out, it seems like most of the action will be local to the QNAP so I hope this thing has a fast backplane! The amount of actually remote shared storage that needs to be fast for all conditions (R, W, RW, RandRW, etc...) and be data that is repeatedly used may not be that high unless I am working with media like running large Handbrake jobs which I will probably still run faster if I use my desktop and write the encoded files to the NAS. Otherwise, the SSD QTier storage is going to be entirely for VMs and Docker applications, which seems like the ideal use for that storage anyway.

It sounds like I don't want to skip using SSD as cache though, even if it's just a read only cache. (A R/W cache would probably be advisable as RAID 1 like QTier SSD in case of hardware failure, right?) I am just wondering if the payoff is that big since SSDs, while much faster than magnetic media, don't compare to RAM module cache which of course is blisteringly fast. Therefore, even if I used a good amount of my SSD for cache, would it make a big difference assuming I had a diverse set of tasks running? It's hard to quantify whether QTier SSD vs Cache SSD (and that could be R or R/W cache at that) would yield noticeably different performance for a mixed environment on my QNAP (assuming no network bottleneck as disk usage is all local) and I was running and unknown mixture of different native, VM, and Docker applications. The documentation gives an inkling on best choice for which application, this but is not that revealing about specifics pros and cons of either especially if you are doing a mish-mash of stuff. I figure that since I will be doing a mixture of applications, I should have a setup that offers different speed boost techniques with my SSDs but of course I have no idea how much of each I should use, and whether I should burn 50% of my potential SSD cache by making it RW over just read-only. The QNap docs are not that specific when it comes to size. They just give minimums. I also suspect I will have to resort to trial and error, but I want to start with a hypothesis that's at least vague sounds. What do you think about going with 500 GB QTier SSD at RAID 1 to accompany my 4 x 6 TB 7200 HDs, have 500 GB (2 x 250 GB) of read-only cache SSD, and finally 250 GB of RW cache SSD running RAID 1? Or is that just way to complicated? I know I an always just start fresh if I screw it up, but I want to at least base this on a reasonable hypothesis if I can, and if I am making some wrongful assumptions about QTier SSD vs Cache SSD I figure now is the time to get correct. Also, I think I might be assuming the using of SSD cache is pretty automated, quite QTier offers your direct control over what uses the SDD storage over the HDs.

As for the GigE vs 10GigE, I will probably just go with trunking for now as I said, since my actual shared storage needs that require high performance are not that high right now (or so I think) but if I find that it is something I do need and I am using the other two NICs for some workhorse VMs or a Docker host, I can add the 10GigE card (or HBA is that's available) as you suggest and do the same on my workstations that would mount the storage. If I did need to go that route, I'd probably get a Brocade switch or something just for the storage network and kit my workstation(s) out with HBAs if I can, or whatever is the appropriate way to get a DAS going. Hopefully that can wait.

Thanks again in advance for the wealth of your experience. I really appreciate it!
-Darf


Sent from my iPad using Tapatalk Pro

[Trexx]

Hi Darf -

Just a suggestions, readers digest version of these post (bullet form, etc.) make it a lot easier to digest your question/options/etc. I appreciate the detailed info, but maybe just a little more condensed.

Networking first - Port trunking will do nothing in terms of speeding up a SINGLE connection right now. It gives you more AGGREGATE throughput at the QNAP level, but at a CLIENT level, you are still limited to 1GbE. So right now 2 NICs in your workstation does no good.

Microsoft did come out with some enhancements to SMB in the latest version in 2016 that MIGHT give some improvement, but QNAP doesn't support those SMB changes yet. I would trunk on the ProSafe, not the ASUS.

I would never do a read/write SSD cache/QTier w/o RAID-1. Qnap will let you do it (for CACHE anyway), but I wouldn't.

SSD's can't be "subdivided" for QTIER & CACHE usage. QNAP uses the entire m.2 modules. So it is an either/or.

As for control, the only option that gives you DIRECT control is SSD Volume. One option would be install QTS on the SSD's (Volume1) and then also keep your apps/VM's/Dockers/etc. on it as well. Then create a 2nd volume (volume2) out of the HDD's where you store your media/etc. This will give you max performance for the things where it matters, and for streaming media (where it doesn't) you can use your spinning disk.

SSD Caching & QTier you can "influence" the decisions the systems make via various settings, but you don't have 100% control over them.

[DarfNader]

Hi Darf -

Just a suggestions, readers digest version of these post (bullet form, etc.) make it a lot easier to digest your question/options/etc. I appreciate the detailed info, but maybe just a little more condensed.

Noted! I will be more mindful of that in the future as I have been known to ramble...

Networking first - Port trunking will do nothing in terms of speeding up a SINGLE connection right now. It gives you more AGGREGATE throughput at the QNAP level, but at a CLIENT level, you are still limited to 1GbE. So right now 2 NICs in your workstation does no good.

Microsoft did come out with some enhancements to SMB in the latest version in 2016 that MIGHT give some improvement, but QNAP doesn't support those SMB changes yet. I would trunk on the ProSafe, not the ASUS.

Really? I'm really surprised that if you had both the QNAP and the workstation with two NICs and set up to be trunked (a.k.a. bonded) and aggregating traffic (as in not just running in failover) and were using a switch that not only supported port trunking in the manner AND has a fast enough backplane that that you would be getting true 2 GigE. What am I missing here?

I would never do a read/write SSD cache/QTier w/o RAID-1. Qnap will let you do it (for CACHE anyway), but I wouldn't.

I agree, and I was going to attempt it. Were to use Cache and were I NOT to mirror it, I'd only use it as read-only as it seems like a way to introduce date-loss into your system needlessly.[/quote]

SSD's can't be "subdivided" for QTIER & CACHE usage. QNAP uses the entire m.2 modules. So it is an either/or.

Yeah, that's what I figured since I was not able to find any way of subdividing the SSD in any way. It wants to use all or nothing, so what you say comes as no surprise, though it's a tad dissapointing, but sometimes the truth hurts![/quote]

As for control, the only option that gives you DIRECT control is SSD Volume. One option would be install QTS on the SSD's (Volume1) and then also keep your apps/VM's/Dockers/etc. on it as well. Then create a 2nd volume (volume2) out of the HDD's where you store your media/etc. This will give you max performance for the things where it matters, and for streaming media (where it doesn't) you can use your spinning disk.

SSD Caching & QTier you can "influence" the decisions the systems make via various settings, but you don't have 100% control over them.

Interesting. So if I am hearing you right, don't bother with QTier at all. Set the SSD modules to be RAID 1 and use SSD storage explicitly for tasks that would really benefit from the speed boost like a VM or Docker server that did some hard labor like transcoding, or something else that was going to be I/O and CPU intensive. I guess what I as thinking was that QTier was kind of elfin magic that would identify these sort of operations anyway as being the more I/O intensive than say simply playing some media file and therefore would populate the modules with the operational storage of my busy VM/Docker containers and if there was not any of those sort of tasks running at the time and hogging the I/O bus at the moment, QTier would put the next busiest data there so it wouldn't go to waste. I saw that as a risk were I to just manually reserve it for VMs. I was of course puttijng a lot of faith in QTier which might have been overly rosy. The other concern is that not all of the storage on my VMs will need to be "turbocharged " which is why the concept of QTier sounded appealing as the idea was only storage that was getting read and written repeatedly would be put on SSD while more static data would stay on disk. Of course I may simply be giving QTier more credit than it has fairly earned, especially since I haven't seen it in action yet. If you are pretty sure that QTier is not going to subtly smart enough to know that pretty much all VM or app should be given priority to be the SSD over plain files, it sounds like QTier may end up slowing things down if it was busy shuffling things or wasn't programmed to almost always give the SSD storage to VMs, apps, or Docker containers.

Thanks again for the advice! I might give both configs a test to see if I can tell if one is faster than the other.



Sent from my iPad using Tapatalk Pro

[Moogle Stiltzkin]

The part I am having trouble deciding is how to best use my m2.SSD modules. I have 2 x 1 TB modules installed and they are kitted out with aftermarket copper heat dissipators so I'm ready to go.

Question is, how? Do I put it all in QTier? Do I use some for SSH cache first, and if so, how much? There are so many options I am getting choice paralysis so I wanted to get input from the community based on the application profile that I am about to list. I realize this is a LOT of text and I don't expect the world to just read my plea and solve all of my problems, but if you have experience in a specific area, I'd love to hear what your impressions are and what advice you'd offer and why.

M.2 SSD should consider installing some sort of heatsink.

If you do opt for Qtier or SSD cache read/write mode, just be aware that SSD are prone to data loss during sudden power loss. So unless you have like no blackouts or any chance of your server losing power, or you already have a UPS; then i would suggest you not use those.

SSD cache acceleration read only may be okay. To my understanding that keeps an extra copy from HDD on the SSD for cache read only? or at least that what i was told anyway.

On forum there are topics regarding qtier vs ssd cache acceleration, you should check that out.

It's recommended to use raid 1 using a pair of the same m.2 SSD (this however does not protect you from sudden power loss, just keep that in mind)

[Moogle Stiltzkin]

As for control, the only option that gives you DIRECT control is SSD Volume. One option would be install QTS on the SSD's (Volume1) and then also keep your apps/VM's/Dockers/etc. on it as well. Then create a 2nd volume (volume2) out of the HDD's where you store your media/etc. This will give you max performance for the things where it matters, and for streaming media (where it doesn't) you can use your spinning disk.

SSD Caching & QTier you can "influence" the decisions the systems make via various settings, but you don't have 100% control over them.

wait are you speaking about the M.2 SSD? or the SSD you install in the front panel bays?

Can the M.2 SSD create 2 volumes like you said for that setup, AS WELL AS configuring the same ssd pairs for raid1 ssd cache acceleration (is that possible or does that not make sense doing so?). Is that how that setup would work?

The only setup for M.2 SSD i knew about was this (but in raid5 hdds)

[Trexx]

Really? I'm really surprised that if you had both the QNAP and the workstation with two NICs and set up to be trunked (a.k.a. bonded) and aggregating traffic (as in not just running in failover) and were using a switch that not only supported port trunking in the manner AND has a fast enough backplane that that you would be getting true 2 GigE. What am I missing here?

Read this document that explains 802.3ad Port Trunking much better than I probably will. http://www.ieee802.org/3/hssg/public/ap ... 1_0407.pdf

Interesting. So if I am hearing you right, don't bother with QTier at all. Set the SSD modules to be RAID 1 and use SSD storage explicitly for tasks that would really benefit from the speed boost like a VM or Docker server that did some hard labor like transcoding, or something else that was going to be I/O and CPU intensive.

I guess what I as thinking was that QTier was kind of elfin magic that would identify these sort of operations anyway as being the more I/O intensive than say simply playing some media file and therefore would populate the modules with the operational storage of my busy VM/Docker containers and if there was not any of those sort of tasks running at the time and hogging the I/O bus at the moment, QTier would put the next busiest data there so it wouldn't go to waste. I saw that as a risk were I to just manually reserve it for VMs. I was of course puttijng a lot of faith in QTier which might have been overly rosy. The other concern is that not all of the storage on my VMs will need to be "turbocharged " which is why the concept of QTier sounded appealing as the idea was only storage that was getting read and written repeatedly would be put on SSD while more static data would stay on disk. Of course I may simply be giving QTier more credit than it has fairly earned, especially since I haven't seen it in action yet. If you are pretty sure that QTier is not going to subtly smart enough to know that pretty much all VM or app should be given priority to be the SSD over plain files, it sounds like QTier may end up slowing things down if it was busy shuffling things or wasn't programmed to almost always give the SSD storage to VMs, apps, or Docker containers.

See this for a good overview of QTier. https://www.qnap.com/solution/qtier-auto-tiering/en/

Remember QTIER works at a block level not a file level. So it may find that some of your docker blocks are hot but others aren't. There is also a lag factor for "cold" data to become "hot" data and get promoted to QTIER. Biggest issue is right now there is no way to "un-QTier" once you go that route. It is a delete/rebuild, restore from backup. I haven't heard of significant slow downs for the migration up/down between tiers in QTier, and I believe you can do some tuning on it to .

Also in QTier 2.0, you can do the following, which I think will address your VM question:

Qtier 2.0 supports "On-Demand Allocation Priority". Administrators can manually configure the storage tier for a folder's new data to be placed in. This provides customizable flexibility to arrange tiered storage suited for system and application demands, and allows for maximum utilization of SSDs for IOPS-demanding tasks.

[Trexx]

wait are you speaking about the M.2 SSD? or the SSD you install in the front panel bays?

Can the M.2 SSD create 2 volumes like you said for that setup, AS WELL AS configuring the same ssd pairs for raid1 ssd cache acceleration (is that possible or does that not make sense doing so?). Is that how that setup would work?

I was referring to Darf's m.2 ssd. No you can't "partition" them to use some space for a volume, and the rest for ssd cache.

[DarfNader]

Also in QTier 2.0, you can do the following, which I think will address your VM question:

Qtier 2.0 supports "On-Demand Allocation Priority". Administrators can manually configure the storage tier for a folder's new data to be placed in. This provides customizable flexibility to arrange tiered storage suited for system and application demands, and allows for maximum utilization of SSDs for IOPS-demanding tasks.

I saw that when I was setting up my QTier volumes... that's pretty cool.

I have to admit that all of this is pretty dizzying. Just when I feel like I have a basic grasp on concepts I start on the task and feel like I haven't a grip on what I am doing. I decided to just start over and give QTier a try and recreate my storage pool with the SSDs in them and see how it goes. But now I am wondering if I should just make the whole thing a status volume like I did before or, considering that I am looking to leverage both QTS as much as it makes sense, and to make purpose-built VMs (or containers when, and if, I wanna get fancy) where you have made some clear arguments why the QTS apps may not the best way to go, do I go with making thick/thin volumes for different purposes? Or is that just complicating things when in fact I am going to be the primary user of this beast and it may not be necessary to slice it up six ways to Sunday.

Well, I am going to read the manual section on Disk, Volumes, and Pools again so I understand how QTS uses these concepts and what the ramifications are for going a certain way considering I am going to end up with a mixed-use system where I am going to have a some heavily used VMs while also relying on native QTS apps, if just for the fact that I need to run video natively.

Sheesh... is there a school for this stuff? I have been a systems engineer all of my life and write chef code to deploy infrastructure, but this is kind of overwhelming. Maybe it's because I don't have enough time to spend on it. Signing off.... thanks again.

[Trexx]

I saw that when I was setting up my QTier volumes... that's pretty cool.

I have to admit that all of this is pretty dizzying. Just when I feel like I have a basic grasp on concepts I start on the task and feel like I haven't a grip on what I am doing. I decided to just start over and give QTier a try and recreate my storage pool with the SSDs in them and see how it goes. But now I am wondering if I should just make the whole thing a status volume like I did before or, considering that I am looking to leverage both QTS as much as it makes sense, and to make purpose-built VMs (or containers when, and if, I wanna get fancy) where you have made some clear arguments why the QTS apps may not the best way to go, do I go with making thick/thin volumes for different purposes? Or is that just complicating things when in fact I am going to be the primary user of this beast and it may not be necessary to slice it up six ways to Sunday.

Well, I am going to read the manual section on Disk, Volumes, and Pools again so I understand how QTS uses these concepts and what the ramifications are for going a certain way considering I am going to end up with a mixed-use system where I am going to have a some heavily used VMs while also relying on native QTS apps, if just for the fact that I need to run video natively.

Sheesh... is there a school for this stuff? I have been a systems engineer all of my life and write chef code to deploy infrastructure, but this is kind of overwhelming. Maybe it's because I don't have enough time to spend on it. Signing off.... thanks again.

I will simplify it down to 3 different config options for you as it can be overwhelming some when you start.

Option 1: (QTier)

• SSD's: Raid-1
• HDD's: Raid-5 Storage Pool, Thick Provisioned (50% utilization or however much you need), Storage Pool 20% (can be adjusted up/down)
• Pin VM Shared folder for "Must have" VM's to SSD

Option 2: (SSD Volume)

• SSD's (Volume1): Raid-1 Storage Pool, 1 Thick Provisioned (50%+ utilization), Storage Pool 10%, for QTS/Critical VM's, Docker, etc.
• HDD's (Volume2): Raid-5 Storage Pool, 1 Thick Provisioned (50% utilization or however much you need), Storage Pool 20% (can be adjusted up/down), for all media, less critical apps, etc.

Option 3: (SSD Caching)

• SSD's: Raid-1 Cache Acceleration. LRU w/2MB block cut off
• HDD's (Volume1): Raid-5 Storage Pool, Thick Provisioned (50% utilization or however much you need), Storage Pool 20% (can be adjusted up/down)

USP and External backups required for all 3 options. If you don't like one of the options, try the next one. I would probably try them in the order listed. Don't overcomplicate things.. simpler is better. I am not a big fan of thin provisioned for most home usage as it can cause many quirks/issues. You either have the space or your don't for what you want to do. For separating/slicing dicing things up... think separate shared folder or snapshot folders.

Here is some info on snapshot folders. https://www.qnap.com/en-in/news/2017/qn ... le-folders

I would say only use is for critical data (docs/spreedsheet/baby photo's/etc.). It is a quicker restore/recovery option vs. volume level snapshot recovery.

[DarfNader]

I will simplify it down to 3 different config options for you as it can be overwhelming some when you start.

Option 1: (QTier)

• SSD's: Raid-1
• HDD's: Raid-5 Storage Pool, Thick Provisioned (50% utilization or however much you need), Storage Pool 20% (can be adjusted up/down)
• Pin VM Shared folder for "Must have" VM's to SSD

Option 2: (SSD Volume)

• SSD's (Volume1): Raid-1 Storage Pool, 1 Thick Provisioned (50%+ utilization), Storage Pool 10%, for QTS/Critical VM's, Docker, etc.
• HDD's (Volume2): Raid-5 Storage Pool, 1 Thick Provisioned (50% utilization or however much you need), Storage Pool 20% (can be adjusted up/down), for all media, less critical apps, etc.

Option 3: (SSD Caching)

• SSD's: Raid-1 Cache Acceleration. LRU w/2MB block cut off
• HDD's (Volume1): Raid-5 Storage Pool, Thick Provisioned (50% utilization or however much you need), Storage Pool 20% (can be adjusted up/down)

USP and External backups required for all 3 options. If you don't like one of the options, try the next one. I would probably try them in the order listed. Don't overcomplicate things.. simpler is better. I am not a big fan of thin provisioned for most home usage as it can cause many quirks/issues. You either have the space or your don't for what you want to do. For separating/slicing dicing things up... think separate shared folder or snapshot folders.

Here is some info on snapshot folders. https://www.qnap.com/en-in/news/2017/qn ... le-folders

I would say only use is for critical data (docs/spreedsheet/baby photo's/etc.). It is a quicker restore/recovery option vs. volume level snapshot recovery.

Thank you! This is great info an is a great place to start.

Here's a question I had with option #1. When I create my first Storage Pool with QTier (after removing the one I made previously) the Storage configuration wizard appears to ask me how much space on the m2.SSDs should be reserved for snapshots. This doesn't seem right. Is this what it is in fact asking, or am I misinterpreting the wizard as it is actually asking me how much of the total pool I should be reserving for snapshots? I hope that it is the latter as putting snapshots on m2.SSD doesn't seem to make a lot of sense- unless my interpretation of what a snapshot is all wrong.

As for the volumes, when you suggest "Thick Provisioned (50% utilization or however much you need)" I presume this is volume that would be for all storage that is set aside for use by QTS apps and the the data that those apps store. When sizing this, should I consider things like the amount of audio and video media that I have, or should I put data that its own volume? I realize that with setting up a media server, I could go either way with either using one of the available QTS apps to serve media or I could make a VM that runs Plex Media Server on it. (Seeing that Plex offers a QNAP-specific server, I will probably go that route.) Still, it would seem that having the media in a location that is easily and efficiently available to a media server as well as a file over the network if, say, I am going to do any video processing on my computer instead of slamming the NAS with it. (Though I do expect the NAS to perform transcoding on-the-fly wherever necessary.)

Also, when you say "Storage Pool" I presume you mean space for shared files and the like, such as documents and the like that I would simply have as shared volume as one would think that a classic NAS offers, or did you mean something else?

My apologies for asking such noob questions, but I want to be clear on meaning since the concepts of "Volume", "Provisioning", "Snapshots", and the like seem to have very specific meaning with QNAP (as it does with any storage/virtualization platform) so I want to be sure I am speaking the same language as all of you folks who know this stuff.

Thanks again! If there is a way I could buy you a "virtual beer" let me know! I owe you at least three by now!

[Trexx]

I will simplify it down to 3 different config options for you as it can be overwhelming some when you start.

Option 1: (QTier)

• SSD's: Raid-1
• HDD's: Raid-5 Storage Pool, Thick Provisioned (50% utilization or however much you need), Storage Pool 20% (can be adjusted up/down)
• Pin VM Shared folder for "Must have" VM's to SSD

Option 2: (SSD Volume)

• SSD's (Volume1): Raid-1 Storage Pool, 1 Thick Provisioned (50%+ utilization), Storage Pool 10%, for QTS/Critical VM's, Docker, etc.
• HDD's (Volume2): Raid-5 Storage Pool, 1 Thick Provisioned (50% utilization or however much you need), Storage Pool 20% (can be adjusted up/down), for all media, less critical apps, etc.

Option 3: (SSD Caching)

• SSD's: Raid-1 Cache Acceleration. LRU w/2MB block cut off
• HDD's (Volume1): Raid-5 Storage Pool, Thick Provisioned (50% utilization or however much you need), Storage Pool 20% (can be adjusted up/down)

USP and External backups required for all 3 options. If you don't like one of the options, try the next one. I would probably try them in the order listed. Don't overcomplicate things.. simpler is better. I am not a big fan of thin provisioned for most home usage as it can cause many quirks/issues. You either have the space or your don't for what you want to do. For separating/slicing dicing things up... think separate shared folder or snapshot folders.

Here is some info on snapshot folders. https://www.qnap.com/en-in/news/2017/qn ... le-folders

I would say only use is for critical data (docs/spreedsheet/baby photo's/etc.). It is a quicker restore/recovery option vs. volume level snapshot recovery.

Thank you! This is great info an is a great place to start.

Here's a question I had with option #1. When I create my first Storage Pool with QTier (after removing the one I made previously) the Storage configuration wizard appears to ask me how much space on the m2.SSDs should be reserved for snapshots. This doesn't seem right. Is this what it is in fact asking, or am I misinterpreting the wizard as it is actually asking me how much of the total pool I should be reserving for snapshots? I hope that it is the latter as putting snapshots on m2.SSD doesn't seem to make a lot of sense- unless my interpretation of what a snapshot is all wrong.

As for the volumes, when you suggest "Thick Provisioned (50% utilization or however much you need)" I presume this is volume that would be for all storage that is set aside for use by QTS apps and the the data that those apps store. When sizing this, should I consider things like the amount of audio and video media that I have, or should I put data that its own volume? I realize that with setting up a media server, I could go either way with either using one of the available QTS apps to serve media or I could make a VM that runs Plex Media Server on it. (Seeing that Plex offers a QNAP-specific server, I will probably go that route.) Still, it would seem that having the media in a location that is easily and efficiently available to a media server as well as a file over the network if, say, I am going to do any video processing on my computer instead of slamming the NAS with it. (Though I do expect the NAS to perform transcoding on-the-fly wherever necessary.)

Also, when you say "Storage Pool" I presume you mean space for shared files and the like, such as documents and the like that I would simply have as shared volume as one would think that a classic NAS offers, or did you mean something else?

My apologies for asking such noob questions, but I want to be clear on meaning since the concepts of "Volume", "Provisioning", "Snapshots", and the like seem to have very specific meaning with QNAP (as it does with any storage/virtualization platform) so I want to be sure I am speaking the same language as all of you folks who know this stuff.

Thanks again! If there is a way I could buy you a "virtual beer" let me know! I owe you at least three by now!

Ssd snapshot would be at the volume level (ssd, hdd, etc.). Start with 20%, or post a screenshot would be helpful. Snapshot reserve can be adjusted after the fact so don’t stress on the % too much. Easier to shrink later than grow though so better off high to start with.

Here is a good overview of Qnap storage management which shows (minus QTier) how the pieces all fit together.

https://www.qnap.com/en/how-to/tutorial ... management

Think of volume as being similar to a hard drive partition. The first 1 (volume1) is like your c: drive where Qnap stores some “OS files” as well as your default location for app installs, multi-media, etc.

So if you have say 30TB of useable space (after raid-5), by default Qnap will say let’s allocate 15tb to the volume. That leaves you free space for snapshot reserve, iSCSI Luns, snapshot folder, etc.

So if you already have 20TB of media, then we’ll need to up the 15tb, if not...stick with the default as one of the benefits of thick luns is that we can grow them. So later you could expand it from 15tb to say 20TB when you get low on space.

Once you expand it though, you can’t shrink it. Which is why you usually don’t want to allocate everything right away.

For Plex, run the native Plex qpkg that you’ll download from Plex.tv. It will be more current than Qnap’s version in the app center.

Your media will be in one or can be several (personal preference) shared folders. Those folders can also be shared externally on your home network to your pc, Xbox, media players, etc.

You may want also want to watch some of Qnap’s YouTube videos where they go into some of these topics/ideas in more detail. https://www.youtube.com/user/QNAPTV




Sent from my iPad using Tapatalk