Automated Nginx Reverse Proxy for Docker

Post Reply
nocsi
New here
Posts: 3
Joined: Mon Jan 18, 2016 1:13 pm

Automated Nginx Reverse Proxy for Docker

Post by nocsi » Fri Sep 02, 2016 9:10 am

So as you can imagine, setting up a proper Docker environment is a pain on QNAP systems. I'm going to detail my setup and what I've had to do in order to get around restraints set by QNAP. Concerning Docker, each container typically gets initialized NATed to an internal network (10.X.X.X) and a random port is exposed and bounded to the QNAP host (32000-33000). Now depending on your network, you can do a few things to readily gain access. For most people, there aren't network restrictions and ports can be readily accessed. I don't particularly think that's good security practice however, my home network has a set of allowed ports on the LAN and 32000-33000 are definitely not in scope.

I've toyed with trying to set NGinx to replace the Qthttpd that runs in QNAP, but quickly realized that they really intend on you not messing around with that. It was to the point where I needed to track different configs and what was having an effect on which along with tracking the services I was disabling to prevent the server from overwriting the config. This proved to be difficult as I wanted NGinx running bounded to :80 and :443 to correctly perform its operations. Qthttpd however is bounded to those ports. Ultimately I decided to reset everything and just allow Qthttpd to live freely segmented on its own network nic.

So what then? We can setup NGinx as a reverse proxy to allow traffic into the exposed ports to reach the Docker containers. Since the ports are randomly generated on startup, that means the NGinx configuration we setup will have to be changed each time for the life of the container. We can automate this as well.

Requirements
* QNginx (viewtopic.php?f=320&t=113790)
* Dedicated Ethernet NIC (https://www.forum-nas.fr/viewtopic.php?f=55&t=4183)
* DNS Server for subdomain resolution (Will leave as an exercise to the reader because everyone's setup is different)
* Golang (viewtopic.php?t=109670 --I have gogs installed and so that's where golang is coming from)

Firstly, setup and install QNginx from the provided link above. It'll default bind to :89 on all interfaces, however for my network setup I don't want to allow port 89 at all. On the French forum there's a tutorial you can follow in setting up QNginx to bind to a second interface. We'll do that as to preserve the first interface for QNAP stuff and move all our Container Station stuff to bind to Eth1 NIC.

.qpkg/QNginx/etc/nginx/nginx.conf

Code: Select all

user httpdusr everyone;
worker_processes  8;
worker_rlimit_nofile 40000;

events {
    worker_connections  1024;
    use epoll;
    multi_accept on;
}


http {
    include       mime.types;
    default_type  application/octet-stream;
    keepalive_timeout  65;
    sendfile        on;
    tcp_nopush     on;
    tcp_nodelay    on;

    #gzip  on;
    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    include conf.d/*.conf;
    include sites-enabled/*;
}


The configuration is set to load everything in sites-enabled/ for dictating each reverse proxy. If you don't care about automating the generation of the config for reverse proxying, then you can stop here and go about your life.

sites-enabled/default.conf will be a reserved configuration file. If you are looking to reverse proxy to whatever other service that is running on your host then keep a separate file as sites-enabled/default.conf is overwritten every time a container starts. 192.168.134.88 in this case is my Eth1 server address, separate network to Eth0 that is now dedicated to just QNAP stuff. This is bounded in the config by specifying the listening address to include the QNAP address on that NIC.

Note I haven't setup ssl/443 yet for proxying. I will be doing a letsencrypt/purchase an SSL cert type setup in the future.

.qpkg/QNginx/etc/nginx/sites-enabled/services.conf

Code: Select all

server {
        server_name carpet.com;
        listen 192.168.134.88:80;
        location /couchpotato {
                proxy_pass http://127.0.0.1:5050/couchpotato;
                proxy_set_header  Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        # sonarr
        location /api/wanted
        {
                return $scheme://$host/sonarr$request_uri;
        }
        location /sonarr
        {
                proxy_pass http://127.0.0.1:8989/sonarr;
                proxy_set_header  Host localhost:8989;
                proxy_redirect    default;
        }
        # nzbget
        location /nzbget
        {
                proxy_pass http://127.0.0.1:6789/nzbget;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
}

server {
        server_name carpet.com;
        listen 192.168.134.88:443 ssl http2;
        client_max_body_size 1024M;
        location /couchpotato {
                proxy_pass https://127.0.0.1:5050;
        }
        location /api/wanted {
                return $scheme://$host/sonarr$request_uri;
        }
        location /nzbget {
                proxy_pass https://127.0.0.1:6789/nzbget;
        }
        location /gogs {
                proxy_pass https://127.0.0.1:3000/gogs;
        }
        location /sonarr {
                proxy_pass https://127.0.0.1:9898/sonarr;
        }
}


Automating NGinx Reverse Proxy
Jason Wilder provides some great utilities that aid in this. I experimented with having an nginx-proxy container running to handle it all but ultimately decided to figure out how to move its operation straight into the QNginx on the server. Mainly I didn't feel like figuring out dual reverse proxying and felt it was a bit redundant. Docker-gen is his utility that monitors /var/run/docker.sock for changes and reads an NGinx config template file to fill in the needed info for the reverse proxying. Simply, it grabs the container's exposed port(random or not) and dynamically generates the config into sites-enabled/default.conf.


Host Install
* amd64 - https://github.com/jwilder/docker-gen/r ... 7.3.tar.gz
* i386 - https://github.com/jwilder/docker-gen/r ... 7.3.tar.gz

Check the git repository for the latest docker-gen binary (https://github.com/jwilder/docker-gen). My system architecture is amd64. Untar it and place it alongside the other docker binaries:

Code: Select all

$ which docker
$ .qpkg/container-station/bin/docker
$ which docker-gen
$ .qpkg/container-station/bin/docker-gen


Create a templates/ directory in the QNginx qpkg folder (.qpkg/QNginx). Place the nginx.tmpl (https://github.com/jwilder/docker-gen/b ... nginx.tmpl) inside the new directory (.qpkg/QNginx/templates). We're going to have to edit portions of it to make it relevant to running on a QNAP host:

.qpkg/QNginx/templates/nginx.tmpl

Code: Select all

server {
        listen [ETH1_addr]:80 default_server;
        server_name _; # This is just an invalid value which will never trigger on a real hostname.
        return 503;
}

{{ range $host, $containers := groupByMulti $ "Env.VIRTUAL_HOST" "," }}
upstream {{ $host }} {

{{ range $index, $value := $containers }}

        {{ $addrLen := len $value.Addresses }}
        {{ $network := index $value.Networks 0 }}

        {{/* If only 1 port exposed, use that */}}
        {{ if eq $addrLen 1 }}
                {{ with $address := index $value.Addresses 0 }}
                        # {{$value.Name}}
                        server 127.0.0.1:{{ $address.HostPort }};
                {{ end }}

        {{/* If more than one port exposed, use the one matching VIRTUAL_PORT env var */}}
        {{ else if $value.Env.VIRTUAL_PORT }}
                {{ range $i, $address := $value.Addresses }}
                        {{ if eq $address.Port $value.Env.VIRTUAL_PORT }}
                        # {{$value.Name}}
                        server 127.0.0.1:{{ $address.HostPort }};
                        {{ end }}
                {{ end }}

        {{/* Else default to standard web port 80 */}}
        {{ else }}
                {{ range $i, $address := $value.Addresses }}
                        {{ if eq $address.Port "80" }}
                        # {{$value.Name}}
                        server 127.0.0.1:{{ $address.HostPort }};
                        {{ end }}
                {{ end }}
        {{ end }}
{{ end }}
}

server {
        server_name {{ $host }};
        proxy_buffering off;
        listen [ETH1_addr]:80;

        location / {
                proxy_pass http://{{ trim $host }};
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;

                # HTTP 1.1 support
                proxy_http_version 1.1;
                proxy_set_header Connection "";
        }
}
{{ end }}


Lastly to get this all going we create a docker_gen.sh and set it to autostart. Every time a container startsup or shutsdown, the nginx config template is referenced and a new config is generated to replace default.conf. QNginx is notified to restart and it will load the new config with the current exposed ports.

docker_gen.sh

Code: Select all

docker-gen -only-published -watch -notify "QNginx.sh restart" templates/nginx.tmpl  etc/nginx/sites-enabled/default.conf


Lastly when configuring Docker containers in Container Station. You have to set VIRTUAL_HOST=containersubdomain1.yourdomain.tld and have a DNS server to resolve that to the address that QNginx is bounded to. Container Station makes it difficult to change the setting after the fact and I haven't figured out a sound method to have the VIRTUAL_HOST variable persist.

I suppose I should try stopping the docker daemon and THEN editing the config.v2.json file. I also might try creating a new docker that imports the old container and sets the VIRTUAL_HOST that way. But for now, QNAP wants you to set all your settings on creation of the container.

I'll update once I figure out TLS/SSL for the containers and the overall setup.

Some Gotchas
* Container Station does not let you specific "ADDRESS::PORT" -> "PORT" for the forwarding as it should. Docker documentation states that exposing ADDRESS::PORT allows you to bind it to a specific NIC on the system and that just isn't passed through by the interface. Container Station will just reject the input and default to NAT.
* Container Station starts docker containers with all ports exposed by default
* Docker-gen will only bind the first accessible port, typically port 80 of the container. VIRTUAL_PORT can be set to indicate the port to be bound and caught by docker-gen

guachin
First post
Posts: 1
Joined: Thu Feb 16, 2017 12:44 am

Re: Automated Nginx Reverse Proxy for Docker

Post by guachin » Thu Feb 16, 2017 12:46 am

Hey! I also need to setup docker behind nginx as reverse proxy. following your thread and this tutorial on docker and nginx as reverse proxy I was able to make it work!!
thanks!!!

User avatar
oyvindo
Experience counts
Posts: 1019
Joined: Tue May 19, 2009 2:08 am
Location: Norway, Oslo

Re: Automated Nginx Reverse Proxy for Docker

Post by oyvindo » Thu Mar 23, 2017 3:01 am

guachin wrote:Hey! I also need to setup docker behind nginx as reverse proxy. following your thread and this tutorial on docker and nginx as reverse proxy I was able to make it work!!
thanks!!!

in the tutorial you link to, Step 1 recommends using docker-gen to solve the random assigning of IP and ports. But why not instead set up the container in host mode? Wouldn't that also solve this problem? Or am I just confused? :'
NAS:
QNAP TS-453Be 16Gb
4x3TB RAID5
QTS 4.4.1
Dockerized installations of:
Madsonic, Logitech Media Server, Pi-Hole, Home-Assistant, MQTT

QNAP HS-251 2G
2x2TB RAID0
QTS 4.4.1
Kodi, Rainloop, Guacamole, L2TP

QNAP TS-119
Single Disk 1Tb
QTS 4.3.3
Used mainly as Reverse Proxy

xavierh
Experience counts
Posts: 1075
Joined: Wed Jan 30, 2008 6:15 am
Location: Denton, Texas

Re: Automated Nginx Reverse Proxy for Docker

Post by xavierh » Thu Mar 30, 2017 6:10 am

oyvindo wrote:
guachin wrote:Hey! I also need to setup docker behind nginx as reverse proxy. following your thread and this tutorial on docker and nginx as reverse proxy I was able to make it work!!
thanks!!!

in the tutorial you link to, Step 1 recommends using docker-gen to solve the random assigning of IP and ports. But why not instead set up the container in host mode? Wouldn't that also solve this problem? Or am I just confused? :'


i was thinking the same thing.

QNAP TVS-951x QTS 4.4.1.1146 build 20191206 OS Storage Pool: Samsung 860 EVO 250GB SSD x 4 (RAID 5), Data Storage Pool: WD WD30EFRX (Red) 3TB x 4 (RAID 5), 16GB RAM WD Easystore 10TB External USB 3.0 Services: SMB, Appletalk, QPKG: Container Station, Virtualization Station, Plex Media Server, Hybrid Backup Sync
QNAP TS-453A QTS 4.4.1.1146 build 20191206 Services: SMB, Appletalk,QVPN
Network: UDM, UDM Beacon, Unifi 8 Port Switch x 3

xavierh
Experience counts
Posts: 1075
Joined: Wed Jan 30, 2008 6:15 am
Location: Denton, Texas

Re: Automated Nginx Reverse Proxy for Docker

Post by xavierh » Thu Mar 30, 2017 6:28 am

oyvindo wrote:
guachin wrote:Hey! I also need to setup docker behind nginx as reverse proxy. following your thread and this tutorial on docker and nginx as reverse proxy I was able to make it work!!
thanks!!!

in the tutorial you link to, Step 1 recommends using docker-gen to solve the random assigning of IP and ports. But why not instead set up the container in host mode? Wouldn't that also solve this problem? Or am I just confused? :'

QNAP TVS-951x QTS 4.4.1.1146 build 20191206 OS Storage Pool: Samsung 860 EVO 250GB SSD x 4 (RAID 5), Data Storage Pool: WD WD30EFRX (Red) 3TB x 4 (RAID 5), 16GB RAM WD Easystore 10TB External USB 3.0 Services: SMB, Appletalk, QPKG: Container Station, Virtualization Station, Plex Media Server, Hybrid Backup Sync
QNAP TS-453A QTS 4.4.1.1146 build 20191206 Services: SMB, Appletalk,QVPN
Network: UDM, UDM Beacon, Unifi 8 Port Switch x 3

Post Reply

Return to “Container Station”