[guide] pfsense VM on QNAP in 2020

[Moogle Stiltzkin]

Traffic Totals package had an issue with the database, it was probably due to improper shutdown. to fix this had to disable the graph then re-enable. and if that doesn't work, to reset the graphic. it's under advanced settings for traffic totals package.

[Moogle Stiltzkin]

2020 Getting started with pfsense 2.4 Tutorial: Network Setup, VLANs, Features & Packages
https://www.youtube.com/watch?v=fsdm5uc_LsU

excellent video by lawrence covering the entire process for setting up pfsense. highly recommended watch


@`45:13

Just a bookmark for myself. still learning the part for segregating networks by modifying the firewall rules, and how to implement that

https://forums.serverbuilds.net/t/guide ... sense/5777

https://www.youtube.com/watch?v=xlx2REV3-wo

[Moogle Stiltzkin]

pfsense roadmap for future update

https://redmine.pfsense.org/projects/pfsense/roadmap

https://www.servethehome.com/pfsense-2- ... es-coming/

Code: Select all

Disable "Hardware Checksum Offloading" if VM is detected

convenient but not that game changing, except make things easier for layman users who would not realize they needed to disable this for vm pfsense.

Looking Ahead
Looking ahead, something many of our users need to be aware of is that there is a big change coming for pfSense 2.5.0. With the next version, we will finally see the depreciation of the built-in pfSense load balancer relayd since it will not work with the newer OpenSSL versions. As a result, the pfSense ecosystem is going to effectively be migrated to using HAproxy for load balancing and reverse proxy duties. Moving load balancing to a package will have the impact of changing how the pfSense features are bundled and will require users to make a change if they are using the built-in load balancer. At STH, we were using pfSense and the HAProxy HA/ load balancer in 2015.

Another big change coming is that we will see an update to FreeBSD 12 with the new pfSense 2.5.0. That is causing some of these changes but should again bring better hardware support. On the subject of hardware, pfSense will not require AES-NI with 2.5.0 as originally planned. For some low-end appliance users, this is great news, but at the same time, we recommend to all STH readers to only buy new hardware with crypto acceleration. The time has come where this should be a must-have feature.

If you see any pfSense 2.4.5 or 2.5.0 feature changes that jump out at you, feel free to call them out in the comments.

[Moogle Stiltzkin]

Was testing around setting up VPN.

2 parts here
- VPN to access my network remotely (this is for accessing network remotely, securely as possible)
- VPN for Mullvad subscription (this is for tunneling my traffic to a difference country to do stuff like geo unblocking etc)


Here are some resources on vpns if anyone else is interested to try it out on pfsense

Tutorial: pfsense OpenVPN Configuration For Remote Users 2020
https://www.youtube.com/watch?v=PgielyUFGeQ

@2:20 lawrence discusses whether suricata is a good idea or not for home users that don't even port forward or run any type of server for remote



Configure OpenVPN step by step from scratch - PFSense (lawrence didn't cover some stuff, so i used this as a supplementary source)
https://www.youtube.com/watch?v=QrZ4TzUprNI


Setting up PIA VPN on pfSense for your whole network and Configuring Selective Routing
https://www.youtube.com/watch?v=ov-xddVpxhc

I use mullvad instead of PIA. So i cross check lawrence guide against Mullvads own guide to get a better idea for this setup
https://mullvad.net/en/help/using-pfsense-mullvad/


I was testing the settings around but i did not fully implement, i disable most of the vpn stuff half way in case later i have an actual need for it. For the Mullvad VPN scenario, it's possible to do split tunneling so only some devices use the VPN, while others are exempted, especially for client pc devices that require low latency for gaming. That said, i'm very use to the mullvad desktop app which i can easily on/off and change server locations quite easily.



Now i'm just working myself up whether to go ahead with trying suricata again. it's a lot of work. Obviously do a backup of config before proceeding. My main concern with Suricata is, just how much maintenance is required to avoid false positives. There are guides out there that minimize false positives for beginners such as myself
https://www.reddit.com/r/PFSENSE/commen ... se_an_ids/


Suricata Network IDS/IPS Installation, Setup, and How To Tune The Rules & Alerts on pfSense 2020
https://www.youtube.com/watch?v=S0-vsjhPDN0

pfsense With Suricata Intrusion Detection System: How & When it works and What It Misses
https://www.youtube.com/watch?v=7gZYbIr_Qj4

[Moogle Stiltzkin]

Here is an interesting case use

Pvk33

1 point 3 months ago

Why would you forward a port? I would prefer to use HAProxy as a reverse proxy. Install the certificates on pfSense and then forward the traffic encrypted or unencrypted to the server. With that only HTTP traffic will go through port 443, it can be inspected to be only traffic for the intended domain and certificate, and IDS/IPS can inspect the traffic.

https://www.reddit.com/r/PFSENSE/commen ... se_an_ids/


How To Setup ACME, Let's Encrypt, and HAProxy HTTPS offloading on pfsense
https://www.youtube.com/watch?v=gVOEdt-BHDY

How To Create pfsense Let's Encrypt Wildcard Certificates using HAProxy
https://www.youtube.com/watch?v=jpyUm53we-Y



In lawrence's example @ 18:30 he is using the haproxy on pfsense for syncthing for his nas



in a future pfsense, you may have to redo haproxy depending on those new changes

Looking Ahead
Looking ahead, something many of our users need to be aware of is that there is a big change coming for pfSense 2.5.0. With the next version, we will finally see the depreciation of the built-in pfSense load balancer relayd since it will not work with the newer OpenSSL versions. As a result, the pfSense ecosystem is going to effectively be migrated to using HAproxy for load balancing and reverse proxy duties. Moving load balancing to a package will have the impact of changing how the pfSense features are bundled and will require users to make a change if they are using the built-in load balancer. At STH, we were using pfSense and the HAProxy HA/ load balancer in 2015.

[Moogle Stiltzkin]

in regards to pfblocker, i was wondering why it could block ads on sites like cnn.com but when i checked youtube, those ads are still there.

sishgupta

Youtube ads have the same domain names and hosts from the same servers as the actual videos you want to watch. So if you were to block youtube ad servers at the router, you'd also just straight up block youtube.

Pfblockerng can not inspect the content of each page. As such, a comprehensive blocking strategy does not just depend on router level blocking, but also browser level blocking.

TLDR: you need to install something like ublock origin in order to block inline video ads by their page elements.

https://www.reddit.com/r/pfBlockerNG/co ... utube_ads/

real_weirdcrap

You are correct. PFBlockerNG and Pi-Hole are DNS based blocking only and since the ads are served mostly from the same domains as the legit video content you can't block one without the other.

Ublock and other adblockers have other blocking methods I believe that allow them to more effectively block these ads though I'm fuzzy on exactly how it all works.

So only solution i could find was just to also install ublock and perhaps also umatrix to fix the youtube ad problem
https://12bytes.org/articles/tech/firef ... t-page-10/


but there is no ublock for my smart tv.... so i was trying to find a solution x_x;
https://www.reddit.com/r/pfBlockerNG/co ... droid_app/


for android there is vanced
https://vanced.app/


but for smart tv... mine uses tizen. so the ecosystem does not have vanced... this is not the regular android :{



a second thing i noticed, even for sites like cnn.com that is does block adds, you still see ugly broken ad boxes, just minus the actual ads. Using ublock it cleans up all that ...stuff.... so the browsing looks much cleaner without all these broken elements.

So in the end you still should use ublock as well as pfblocker.

[Moogle Stiltzkin]

i redid the iperf test

iperf uses a pretty small window size by default.

Increase the window or run it in UDP mode and it will easily saturate a 1 Gbit/s connection.

*update

refer to the other post below for the correct results. thx for the fix iperf command onecd



https://tynick.com/blog/07-08-2019/how- ... t-to-host/

https://www.slashroot.in/iperf-how-test ... ebandwidth



So yes, my network does indeed fully use the 1gbe both ways, so no issues with network performance

[OneCD]

So yes, my network does indeed fully use the 1gbe both ways, so no issues with network performance

Not quite. You're only showing 1Mb/s there.

This is because you ran the tests with the default UDP bandwidth limiter: 1Mb/s.

To measure the "possible" UDP bandwidth, you’ll need to raise the bandwidth limit. Let's try it at 1Gb/s (shown here between my Debian workstation and Debian NAS):

Code: Select all

$ iperf3 -c 10.0.0.2 -u -b 1g
Connecting to host 10.0.0.2, port 5201
[  5] local 10.0.0.10 port 36691 connected to 10.0.0.2 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   113 MBytes   949 Mbits/sec  81910  
[  5]   1.00-2.00   sec   113 MBytes   951 Mbits/sec  82130  
[  5]   2.00-3.00   sec   113 MBytes   951 Mbits/sec  82125  
[  5]   3.00-4.00   sec   113 MBytes   951 Mbits/sec  82123  
[  5]   4.00-5.00   sec   113 MBytes   951 Mbits/sec  82112  
[  5]   5.00-6.00   sec   113 MBytes   951 Mbits/sec  82109  
[  5]   6.00-7.00   sec   113 MBytes   951 Mbits/sec  82118  
[  5]   7.00-8.00   sec   113 MBytes   948 Mbits/sec  81808  
[  5]   8.00-9.00   sec   113 MBytes   951 Mbits/sec  82124  
[  5]   9.00-10.00  sec   113 MBytes   951 Mbits/sec  82134  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.11 GBytes   951 Mbits/sec  0.000 ms  0/820693 (0%)  sender
[  5]   0.00-10.04  sec  1.11 GBytes   947 Mbits/sec  0.013 ms  83/820679 (0.01%)  receiver

iperf Done.

Not-too-shabby: 951Mb/s (which isn't really Mb/s in this display) converts to 997,195,776 bits-per-second.

Or, try running it without the '-u' parameter to check using TCP:

Code: Select all

$ iperf3 -c 10.0.0.2 
Connecting to host 10.0.0.2, port 5201
[  5] local 10.0.0.10 port 41542 connected to 10.0.0.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   945 Mbits/sec    0    363 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   936 Mbits/sec    0    363 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   938 Mbits/sec    0    363 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   936 Mbits/sec    0    363 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   938 Mbits/sec    0    382 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   937 Mbits/sec    0    382 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   936 Mbits/sec    0    382 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   939 Mbits/sec    0    382 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   939 Mbits/sec    0    489 KBytes       
[  5]   9.00-10.00  sec   112 MBytes   940 Mbits/sec    0    489 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   938 Mbits/sec    0             sender
[  5]   0.00-10.04  sec  1.09 GBytes   932 Mbits/sec                  receiver

iperf Done.

[Moogle Stiltzkin]

...

thx i'll try again and post back


Re-ran the command ok that worked. tested connection speed to the router and to the ts-877 qnap nas



and for the reverse command i used this

Code: Select all

iperf3 -c 192.168.0.210 -R -u -b 1g

i'll add this to the other thread for future reference
viewtopic.php?t=140883


*update

apparently the R is case sensitive. So r previous was incorrect, so i updated it

[Moogle Stiltzkin]

should i be doing jumbo frames? on my network setup. or will that just cause issues o-O; yeah probly better not
https://www.youtube.com/watch?v=vRiPqoybVhI
https://www.youtube.com/watch?v=jJFC-8z3AfM

[Moogle Stiltzkin]

i was testing out vpn, but later disabled when not using.

Initially i just wanted to go through the process for setting up a remote vpn access for the network, and another vpn for mullvad. i got as far as i felt comfortable, then decided to disable it but leave the entries intact but presumably not active.

then noticed pfsense crash later (the next day). So i rebooted the router but i still couldn't access pfsense.

to find out why it wasn't working, i connected my monitor and keyboard to the qotom pfsense router.

that was when i found out why when i rebooted pfsense it didn't fully boot up. because during the loading, there vpn auth password prompt which blocked pfsense from fully booting. so i had to press enter "blank" for it to procede.

so once i got back in pfsense, i then from shell rebooted to single user mode. then ran this command to see if there was any corruption that needed fixing (ran the code 5 times as recommended).

Code: Select all

fsck -fy /

i don't know why it did that (the vpn prompt for pfsense after rebooting each time), so i tried to delete any vpn settings in my pfsense config i had set but disabled earlier. hoping that would fix it.


But thats when i discovered that undoing vpn isn't so simple or clearcut. Because i had a vpn cert that couldn't be removed. and from what i gathered, even if it seemingly looks disabled, it may still be active.


now learned removing vpn isn't so easy
https://www.informaticar.net/how-to-rev ... e-openvpn/

https://forum.netgate.com/topic/137573/ ... -and-certs

https://www.reddit.com/r/PFSENSE/commen ... delete_ca/

>_>: ...


i managed to solve this by restoring an earlier config, before i began messing with the VPN. I can't stress enough how much i appreciate pfsense' backup and restore functionality which is more reliable compared to my last router.

Seems like i need to learn more about it's quirks before i touch that again

[Moogle Stiltzkin]

keeping historical logs from pfsense
https://forum.netgate.com/topic/31716/l ... sh-issue/7

Syslog is the best way, push them off to that same Linux box with syslog and you don't have to do anything manually. You can store and rotate them there.

All we not care so much about logs BEFORE some troubles happened.
And only after troubles we start seeking thru the logs to find saver on question “what exactly happened”.

https://forum.netgate.com/topic/149393/ ... emory-card


qnap has a syslog server feature
https://community.cambiumnetworks.com/u ... 1b99a.jpeg

https://community.cambiumnetworks.com/u ... 2ad49.jpeg


so in pfsense go to Status > System Logs > Settings

enable remote server log

add the ip and port for your qnap syslog, apply. might have to reboot router.


So now.... IF after my router crashes for some reason, i would then have logs SURVIVE the crash, so i can then read what the heck just happened. Before i did not have these saved logs. But now i should


@4:00 this video briefly covers about syslogs on pfsense
https://www.youtube.com/watch?v=9nnHbYUKNio

[Moogle Stiltzkin]

hm i managed to crash even when using qotom pfsense.

could be a pfsense configuration rather than qnap pfsense fault? i'll have to find out what setting i can change to fix this.

Nachtfalke Nov 26, 2012,

Perhaps something of the followuing could help you:

Check that the MBUF Usage isn't reaching the limit - could be increased with kern.ipc.nmbclusters

Check kern.ipc.somaxconn. The description I found for that is: The kern.ipc.somaxconn sysctl variable limits the size of the listen queue for accepting new TCP connections. The default value of 128 is typically too low for robust handling of new connections in a heavily loaded web server environment. For such environments, it is recommended to increase this value to 1024 or higher. I set it to 2048

I increased net.inet.tcp.sendbuf_max to 16777216

I increased net.inet.tcp.recvbuf_max to 16777216

I increased net.inet.ip.intr_queue_maxlen to 3000

I put these sysctls into SYSTEM –> Advanced --> System Tunables. Some people in the forum say that they will not work there and you need t put the in /root/loader.conf.local or /root/loader.conf.
What is for sure that you need a reboot after you did these changes.

https://forum.netgate.com/topic/50584/t ... i-fix-this

https://forum.netgate.com/topic/126380/ ... ownloading

https://forum.netgate.com/topic/129293/ ... big-time/2



But brute force downloading a ton of torrents causes crashes :{

1 maybe 2 torrents is fine

May 20, 2016

Realtek cards are not OK for a router. Enable netmap (IPS on opnsense) and then download a few popular Linux distros via torrent.
Your box will lock up soon enough if you're using the default mtu. The card stops processing packets.

A workaround seems to use a higher mtu, but that's just black magic. The card seems to reset the queue when overloaded and things keep working.

Also, you'll reach maybe 70% of your Gigabit connection vs 90%+ with an Intel card.

So, it kind of works, but it's not great, especially if you have a decent WAN and a few diverse users in your LAN.

https://forums.freebsd.org/threads/high ... pts.55277/


i'm sure the qotom uses intel nics


some other suggestions

KOM Apr 5, 2018,
Perhaps you need to look into some traffic shaping so that your torrent app doesn't kill your network.

this is likely to help, but.... i wish i can max out my speed ideally.

x2rl Apr 5, 2018,

Guess ill have to look into that. Could it be the hardrive in the pfsense box its not great

hm... i'm using whatever ssd came prebuilt. i see some people just buy the box but buy separately the ssd and ram. maybe i should have done that, but was lazy. something for me to consider the next time i get one of these nucs for pfsense.

KOM Apr 6, 2018,
Try going to System - Routing - Gateways. Edit your gateway and check the Disable Gateway Monitoring checkbox and try again. It seems that your torrent app is filling your pipe to the point that the upstream monitor thinks your link is dying.

SammyWoo Apr 7, 2018,


I suggest save your conf, then turn on Traffic Shaper and see what it does, always can revert back by restoring conf.

If you use the Traffic Shaper's Wizard, it guides you through pages, and on the second page I believe, it presents you with common scenarios you want to deal with and one of them is ta-da, peer-to-peer protocol, u can simply ENABLE it and give it LOW Priority and see what happens.

In Cisco-land, Traffic Shaper is called Priority Queuing, which is a term simpler to understand what it's doing underneath.

In Traffic Shaper, a service sits on the WAN port controlling the uploads. a second service sits on the LAN port controlling the downloads.

The gists of it is, rather than letting traffic pass though these ports as they come, the TS services hold the packets on queues (or buffer if u prefer), each queue has a priority label, another service takes the packets from the queues and shoot them out the interface. The highest priority queue gets serviced more often than the lower queues. Think of a traffic cop sitting at a intersection and letting go of the commuter lane for 1 minute, while let go of the next passenger lane only for 15 seconds. This way the packets flow is controlled, giving a chance for everybody to go through, eventually.

Well there is more to it, if you really get into it, but last paragraph is the gist and I hope easy to understand.

Bottom line is, without flow control, an app, in this case peer-to-peer often takes over and everybody else get stuck/freeze.

Don't know why your consumer grade box works, it could be just a coincidence. There is the possibility that it came with QOS (another Traffic Shaper term) turned on. Lots of consumer boxes these days come with a variety versions of QOS, in part to deal with VOIP.

[Moogle Stiltzkin]

i got guest wifi to work on pfsense. posted a guide here
viewtopic.php?f=45&t=155460&p=766764#p766764

any questions, just ask. it took trial and error, and finally asking help on reddit for this to get figured out

[Qmann]

I also posted this over in a 872XT memory upgrade thread, but thought i would post here too since it's pfSense related. I got my unsupported 64GB Crucial memory installed, and pfSense has these errors now. Turn pfblockerNg off, and everything works again. Hmmm. pfSense had 2GB allocated and worked fine, so I gave it 3GB and still has this problem. All other systems seem to work fine with the memory.

There were error(s) loading the rules: /tmp/rules.debug:24: cannot define table pfB_Top_v4: Cannot allocate memory - The line in question reads [24]: table <pfB_Top_v4> persist file "/var/db/aliastables/pfB_Top_v4.txt"
@ 2020-10-11 15:00:56

Anyone know of a way to run a memtest on the QNAP that could verify ALL memory is available to the device, and working properly? I can't hardly see why just a module in pfSense wouldn't work, but the system and all other apps are working fine. It is officially an unsupported amount of memory, but still, weird it's just one module of pfSense not working right....