MISC-27 Building a self hosted DNS over HTTPS Server ======================================================== 2019-05-01 This document is the compressed setup process based on exploration in https://projects.bentasker.co.uk/jira_projects/browse/MISC-27.html and should result in a working DoH server running Nginx. This process should result in an actual article with cogent explanations in some point at the future :) Install the DNS-over-HTTPS Server software --------------------------------------------- apt-get update apt-get install curl software-properties-common build-essential git mkdir build cd build # Need Go >= 1.10 to build DoH server # so fetch latest wget https://dl.google.com/go/go1.12.2.linux-amd64.tar.gz tar xzf go1.12.2.linux-amd64.tar.gz mv go /usr/local/ export GOROOT=/usr/local/go export PATH=$GOPATH/bin:$GOROOT/bin:$PATH go version mkdir ~/gopath export GOPATH=~/gopath wget https://github.com/m13253/dns-over-https/archive/v2.0.1.tar.gz tar xzf v2.0.1.tar.gz cd dns-over-https-2.0.1/ make make install # Install the config cat << EOM > /etc/dns-over-https/doh-server.conf # HTTP listen port listen = [ "127.0.0.1:8053", "[::1]:8053", ] # TLS certification file # If left empty, plain-text HTTP will be used. # You are recommended to leave empty and to use a server load balancer (e.g. # Caddy, Nginx) and set up TLS there, because this program does not do OCSP # Stapling, which is necessary for client bootstrapping in a network # environment with completely no traditional DNS service. cert = "" # TLS private key file key = "" # HTTP path for resolve application path = "/dns-query" # Upstream DNS resolver # If multiple servers are specified, a random one will be chosen each time. upstream = [ "127.0.0.1:5353" ] # Upstream timeout timeout = 10 # Number of tries if upstream DNS fails tries = 3 # Only use TCP for DNS query tcp_only = false # Enable logging verbose = false # Enable log IP from HTTPS-reverse proxy header: X-Forwarded-For or X-Real-IP # Note: http uri/useragent log cannot be controlled by this config log_guessed_client_ip = false EOM systemctl restart doh-server systemctl status doh-server NGinx Install --------------- apt-get install gnupg2 ca-certificates lsb-release echo "deb http://nginx.org/packages/debian `lsb_release -cs` nginx" >> /etc/apt/sources.list.d/nginx.list curl -fsSL https://nginx.org/keys/nginx_signing.key | apt-key add - apt-key fingerprint ABF5BD827BD9BF62 apt-get update apt-get -y install nginx # We're going to set rate limits just in case the public gain access # This sets 300 requests a second cat << EOM > /etc/nginx/conf.d/00-rate-limits.conf limit_req_zone \$binary_remote_addr zone=doh_limit:10m rate=300r/s; EOM # Create the config - remember to replace server_name with whatever name you are using cat << EOM > /etc/nginx/conf.d/doh.conf upstream dns-backend { server 127.0.0.1:8053; keepalive 30; } server { listen 80 server_name dns.bentasker.co.uk; root /tmp/NOEXIST; location /dns-query { limit_req zone=doh_limit burst=50 nodelay; proxy_set_header X-Real-IP \$remote_addr; proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for; proxy_set_header Host \$http_host; proxy_set_header X-NginX-Proxy true; proxy_http_version 1.1; proxy_set_header Upgrade \$http_upgrade; proxy_set_header Connection ""; proxy_redirect off; proxy_set_header X-Forwarded-Proto \$scheme; proxy_read_timeout 86400; proxy_pass http://dns-backend/dns-query; } location / { return 404; } } EOM systemctl restart nginx Verify it's started root@debian-9-doh-newbuild:~/build/dns-over-https-2.0.1# netstat -lnp | grep 80 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 4890/nginx: master tcp 0 0 127.0.0.1:8053 0.0.0.0:* LISTEN 4029/doh-server tcp6 0 0 ::1:8053 :::* LISTEN 4029/doh-server udp6 0 0 fe80::343e:5bff:fee:123 :::* 834/ntpd SSL ---- Next we need to sort out SSL cat << EOM > /etc/nginx/conf.d/00-cert-stapling.conf ssl_stapling on; ssl_stapling_verify on; resolver 127.0.0.1:5353; EOM We're going to set up certbot to use LetsEncrypt, make sure your DNS name (in my case dns.bentasker.co.uk) resolves to your box apt-get install certbot python-certbot-nginx certbot --nginx -d dns.bentasker.co.uk Agree all the terms, and when prompted, choose redirect. Certbot's default SSL options can are a little over liberal though, so overwrite them with some more conservative values cat << EOM > /etc/letsencrypt/options-ssl-nginx.conf ssl_session_cache shared:le_nginx_SSL:1m; ssl_session_timeout 1440m; ssl_protocols TLSv1.1 TLSv1.2 TLSv1.3; ssl_prefer_server_ciphers on; ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256'; add_header Strict-Transport-Security "max-age=31536000;" always; EOM systemctl restart nginx Nginx should now be listening on port 443 root@debian-9-doh-newbuild:~/build/dns-over-https-2.0.1# netstat -lnp | grep nginx tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 5466/nginx: master tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 5466/nginx: master Edit the doh file to enable HTTP/2 root@debian-9-doh-newbuild:/usr/local/etc/unbound# cat /etc/nginx/conf.d/doh.conf upstream dns-backend { server 127.0.0.1:8053; keepalive 30; } server { server_name dns.bentasker.co.uk; root /tmp/NOEXIST; location /dns-query { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_set_header X-NginX-Proxy true; proxy_set_header Connection ""; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_redirect off; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 86400; proxy_pass http://dns-backend/dns-query; } location / { return 404; } listen 443 ssl http2; # managed by Certbot ssl_certificate /etc/letsencrypt/live/dns.bentasker.co.uk/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/dns.bentasker.co.uk/privkey.pem; # managed by Certbot } Unbound Setup --------------- Start by restricting access so that outside users cannot access unbound while we're working on it iptables -I INPUT -p tcp --dport 53 -j REJECT iptables -I INPUT -p udp --dport 53 -j REJECT iptables -I INPUT -i lo -j ACCEPT apt-get install dnsutils Packaged Unbound ------------------ If you're going to be running this on a LAN, at this point, you can just run apt-get install unbound Compiled Unbound ----------------- However, if you're setting up in public address space, you're probably going to want ECS support (so that CDNs route you to the PoP nearest you, not nearest to your DoH instance), in which case we need to download and compile unbound apt-get install libssl-dev libexpat1-dev cd ~/build wget http://www.unbound.net/downloads/unbound-latest.tar.gz tar xzf unbound-latest.tar.gz cd unbound-1.9.1/ ./configure --enable-subnet make make install ln -s /usr/local/etc/unbound/ /etc/unbound ldconfig useradd -r unbound mkdir /usr/local/etc/unbound/trust mkdir /usr/local/etc/unbound/local.d mkdir /usr/local/etc/unbound/unbound.conf.d chown unbound /usr/local/etc/unbound/trust sudo -u unbound unbound-control-setup -d /usr/local/etc/unbound/trust/ sudo -u unbound unbound-anchor -a /usr/local/etc/unbound/trust/root.key cat << EOM > /lib/systemd/system/unbound.service [Unit] Description=Unbound DNS server Documentation=man:unbound(8) After=network.target Before=nss-lookup.target Wants=nss-lookup.target [Service] Type=simple Restart=on-failure EnvironmentFile=-/etc/default/unbound ExecStartPre=/usr/local/sbin/unbound-anchor -a /usr/local/etc/unbound/trust/root.key ExecStart=/usr/local/sbin/unbound -c /usr/local/etc/unbound/unbound.conf -d \$DAEMON_OPTS ExecReload=/usr/local/sbin/unbound-control reload [Install] WantedBy=multi-user.target EOM systemctl daemon-reload Configuring Unbound --------------------- If you used apt-get to install unbound, then you'll need to change paths below to be /etc/unbound cat << EOM > /usr/local/etc/unbound/unbound.conf server: module-config: "subnetcache validator iterator" chroot: "/usr/local/etc/unbound" directory: "/usr/local/etc/unbound" username: "unbound" interface: 127.0.0.1 port: 5353 do-daemonize: yes verbosity: 1 # Enable UDP, "yes" or "no". do-udp: yes # Enable TCP, "yes" or "no". do-tcp: yes auto-trust-anchor-file: "/usr/local/etc/unbound/trust/root.key" # ECS support client-subnet-zone: "." client-subnet-always-forward: yes max-client-subnet-ipv4: 24 max-client-subnet-ipv6: 48 # Randomise case to make poisioning harder use-caps-for-id: yes # Minimise QNAMEs qname-minimisation: yes harden-below-nxdomain: yes # This is where we'll put our adblock config include: local.d/*.conf include: unbound.conf.d/*.conf remote-control: control-enable: yes control-interface: 127.0.0.1 control-port: 8953 server-key-file: "/usr/local/etc/unbound/trust/unbound_server.key" server-cert-file: "/usr/local/etc/unbound/trust/unbound_server.pem" control-key-file: "/usr/local/etc/unbound/trust/unbound_control.key" control-cert-file: "/usr/local/etc/unbound/trust/unbound_control.pem" EOM If you want to just forward queries on rather than have unbound resolve them itself, you can set forwarders like this cat << EOM > /usr/local/etc/unbound/unbound.conf.d/forwarders.conf forward-zone: name: "." forward-addr: 8.8.8.8 forward-addr: 8.8.4.4 forward-addr: 1.1.1.1 EOM Now start unbound and it should be bound to loopback on port 5353: systemctl start unbound root@debian-9-doh-newbuild:/usr/local/etc/unbound# netstat -lnp | grep unboun tcp 0 0 127.0.0.1:5353 0.0.0.0:* LISTEN 16753/unbound udp 0 0 127.0.0.1:5353 0.0.0.0:* 16753/unbound At this point, we _should_ be able to place requests via our setup root@debian-9-doh-newbuild:/usr/local/etc/unbound# curl -s "https://dns.bentasker.co.uk/dns-query?name=projects.bentasker.co.uk&type=A" | python -m json.tool { "AD": false, "Answer": [ { "Expires": "Wed, 01 May 2019 11:05:14 UTC", "TTL": 3600, "data": "projects.balanced.bentasker.co.uk.", "name": "projects.bentasker.co.uk.", "type": 5 }, { "Expires": "Wed, 01 May 2019 10:05:44 UTC", "TTL": 30, "data": "51.255.232.237", "name": "projects.balanced.bentasker.co.uk.", "type": 1 } ], "CD": false, "Question": [ { "name": "projects.bentasker.co.uk.", "type": 1 } ], "RA": true, "RD": true, "Status": 0, "TC": false, "edns_client_subnet": "134.209.27.0/24" } Adblock Setup --------------- Now we want to set up an auto-pull of a list of adblocked domains so that Unbound will return 127.0.0.1 for those Set up a cron to periodically refresh the list: curl -o /root/update_ads.sh https://www.bentasker.co.uk/adblock/static/update_ads.sh chmod +x /root/update_ads.sh echo "0 0 * * * root /root/update_ads.sh" > /etc/cron.d/ad_domains_update Now go and manually populate an instance of the list cd /usr/local/etc/unbound/local.d/ curl -o adblock.new "https://www.bentasker.co.uk/adblock/autolist.txt" mv adblock.new adblock.conf systemctl restart unbound curl -s "https://dns.bentasker.co.uk/dns-query?name=google-analytics.com&type=A" | python -m json.tool # Should give an IP of 127.0.0.1 Firewall ---------- iptables -F INPUT iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -A INPUT -i lo -s 127.0.0.0/8 -j ACCEPT iptables -A INPUT -p tcp --dport 443 -j ACCEPT iptables -A INPUT -p tcp --dport 80 -j ACCEPT iptables -A INPUT -p tcp --dport 22 -j ACCEPT iptables -A INPUT -j REJECT ip6tables -F INPUT ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT ip6tables -A INPUT -i lo -s ::1 -j ACCEPT ip6tables -A INPUT -p tcp --dport 443 -j ACCEPT ip6tables -A INPUT -p tcp --dport 22 -j ACCEPT ip6tables -A INPUT -j REJECT apt-get -y install iptables-persistent Client Setup =============== Firefox --------- It's possible to set up Firefox to perform queries itself by pointing it's Trusted Recursive Resolver (TRR) config to your server. Go to about:config Set: - network.trr.uri to https://dns.bentasker.co.uk/dns-query - network.trr.mode 2 - network.trr.disable-ECS false OS Level - Android -------------------- Go to Google's Play store and download Intra - https://play.google.com/store/apps/details?id=app.intra&hl=en_GB - Open Intra - Go into Settings and choose "Select DNS over HTTPS Server" - Choose custom server URL - enter https://dns.bentasker.co.uk/dns-query OS Level - Linux ------------------- wget https://dl.google.com/go/go1.12.2.linux-amd64.tar.gz tar xvf go1.12.2.linux-amd64.tar.gz sudo mv go /usr/local/ export GOROOT=/usr/local/go export PATH=$GOPATH/bin:$GOROOT/bin:$PATH wget https://github.com/m13253/dns-over-https/archive/v2.0.1.tar.gz tar xzf v2.0.1.tar.gz mkdir ~/gopath export GOPATH=~/gopath make sudo make install sudo vi /etc/dns-over-https/doh-client.conf Change the URL under [[upstream.upstream_ietf]] to "https://dns.bentasker.co.uk/dns-query": [[upstream.upstream_ietf]] url = "https://dns.bentasker.co.uk/dns-query" weight = 100 systemctl enable doh-client.service systemctl start doh-client.service netstat -lnp | grep 53 dig @127.0.0.1 projects.bentasker.co.uk If you're using NetworkManager, override the DNS: sudo vi /etc/NetworkManager/system-connections/Wired\ connection # Add Under the [ipv4] section: ignore-auto-dns=true dns=127.0.0.1; Otherwise, just drop it into resolv.conf echo nameserver 127.0.0.1 | sudo tee /etc/resolv.conf Additional Notes =================== Split-Horizon DNS -------------------- One of the reasons for doing this may be that you want to use DoH on mobile devices (such as yourp phone and laptop) so that your queries aren't interfered with by network operators when out and about, but don't want to need to remember to toggle it off when you get home and want to resolve local domains (Note: alternatively, if you configure Firefox with network.trr.mode 2 it'll automatically fall back to the OS configured resolver if it cannot resolve a name, so LAN names will still resolve if you haven't also configured the OS to use DoH. The downside of that is that apps etc don't get the DoH protection). If it's your intention to do that, then you'll want to essentially end up with two DoH servers, one that's in publicly routable address space (so that your devices can reach it when out and about) and one on the LAN, carrying LAN specific records/overrides. You'd then use traditional split-horizon DNS on the name to send LAN clients to the LAN DoH server, and out-and-about devices to the main one. Note that you don't necessarily need to run a full-fat DoH server on your public instance, and could instead proxy through to cloudflare or Google: server { server_name dns.bentasker.co.uk; root /tmp/NOEXIST; location /dns-query { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host 1.1.1.1; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_redirect off; proxy_read_timeout 86400; proxy_pass https://1.1.1.1/dns-query; } location / { return 404; } listen 443 ssl http2; # managed by Certbot ssl_certificate /etc/pki/cert/cert.pem; ssl_certificate_key /etc/pki/private/cert.key; } You will need a full-fat version on the LAN though so that you can configure overrides Authentication and Access Control ------------------------------------ If your server is in publicly routable address space, you will likely wish to consider implementing some form of access control. There isn't a defined way of supporting this, so the approach you take will depend on the clients you expect to be using your DoH server. If the clients will only be Firefox, then there's one way, if it's more likely to be mixed, then there's an alternative approach Firefox Clients ---------------- If you're simply using Firefox's TRR, it's possible to provide it with the right-hand side of an authorisation header in network.trr.credentials So you might do something like the following (please use a better password) cd /etc/nginx printf "dohuser:$(openssl passwd -crypt supersec)\n" >> .htpasswd vi conf.d/doh.conf # Add the following within the dns-query location statement: satisfy any; deny all; auth_basic "Authentication is a must"; auth_basic_user_file /etc/nginx/.htpasswd; service nginx reload Then generate a string to pass to Firefox echo "Basic `echo -n "dohuser:supersec" | base64`" Basic ZG9odXNlcjpzdXBlcnNlYw== Set that as the value for network.trr.credentials Intra/DoH-client/Mixed ------------------------- Most DoH clients do not include explicit support for authentication. Assuming you're not able to predict what each client's IP will be at any point, you will need to rely on a means of putting authentication tokens into the path, and reconfiguring your client to use those tokens. That may be as simple as changing your NGinx location block location /mysupersecretstring/dns-query { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_set_header X-NginX-Proxy true; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_redirect off; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 86400; proxy_pass http://dns-backend/dns-query; } location /dns-query { return 404; } to frustrate bots/attackers that might test for responses under dns-query. Or, for slightly more complex implementations (such as having a 'user' per client) you could also switch to using OpenResty so that you can use LUA to validate credentials passed in the query string. Under that scheme you'd reconfigure Intra (or whatever) to include a fixed query string containing the credentials.