Building a NixOS router for a UK FTTP ISP – The Basics
I use NixOS as a router for my FTTP ISP in the UK. The details in this post should apply to most PPPoE-delivered services.
In a later post I'll add 4G failover in a way that's more specific to my ISP, then add multiple VLANs and 1:1 NAT to make the most of the additional IPv4 addresses my ISP offers.
I picked up one of those cheap AliExpress tiny PCs a few months ago to have something dedicated to the router role in my setup. I'd previously used the same machine that self hosts my various services for the role, along with a USB ethernet dongle for the WAN side, and I really wanted to decouple them, with the router having a much smaller footprint and attack surface.
The particular model I got was the “SZBOX G48S Alder Lake N100 Soft Router”, with 8Gb RAM, 256Gb NVMe SSD and a 4-core Intel N100 CPU. It was £151 delivered, including UK power cable. It's £131 today!
I don't necessarily recommend it, so I'm not linking to it directly. It rejected a 16Gb stick of Crucial RAM I tried that works fine in another system (with a lot of hard-hangs). I'm sure not expecting updates to the BIOS. But I wiped the Kingston SSD it shipped with, tried out some things, and reassured myself that it's not behaving nefariously in a way that I could detect. Be careful out there.
The thing most attractive to me about it was the fanless design and 4x 2.5GbE Intel I226-V NICs. Perfect (or massive overkill) for a “home” router.
I'm not going to get into the weeds on NixOS in this post, and will assume you're comfortable installing it. I use a flake-based approach to my systems, all defined in a single Git repo. It's a bit of a time sink getting familiar with it, but for me has been worth it for how quickly I can now configure new systems and services.
Below is a simplified version of the module for this host. It's missing things like an SSH server, a local non-root user, and some commands you'll probably want to add to debug things (tcpdump, etc).
It's opinionated, I prefer to use systemd-networkd where possible, and use nftables directly for messing with packets rather than the NixOS firewall module.
I use dnsmasq for DHCP (IPv4) and internal DNS, with AdGuard Home for ad-blocking DNS. My internal DNS domain is home.arpa.
I make use of agenix to keep encrypted secrets in my config repo. It's also possible to provide the password directly in the PPP config, but I wouldn't recommend it.
{ config, lib, pkgs, ... }:
{
imports =
[
./hardware-configuration.nix
];
# allocate some of our RAM to use as compressed swap
# which I use instead of disk-backed swap
zramSwap.enable = true;
boot.kernel.sysctl = {
# be more swappy as we're using zramswap
"vm.swappiness" = 100;
# enable IPv4 and IPv6 forwarding on all interfaces
"net.ipv4.conf.all.forwarding" = true;
"net.ipv6.conf.all.forwarding" = true;
"net.ipv4.conf.all.arp_filter" = 1;
"net.ipv4.conf.default.arp_filter" = 1;
};
# sleep any attached screen after 5 minutes
boot.kernelParams = [ "consoleblank=300" ];
networking = {
hostName = "hermes";
useDHCP = false;
useNetworkd = true;
nftables.enable = true;
# we write our own rules
firewall.enable = false;
# these will be available in the local DNS
# along with DHCP hostnames
extraHosts = ''
192.168.32.1 hermes.home.arpa
2001:x:x:x::1 hermes.home.arpa
'';
};
# the basic network configuration for systemd-networkd
systemd.network = {
enable = true;
wait-online.enable = false;
links = {
"10-lan" = {
# port 0 is plugged into my lan switch
matchConfig = {
MACAddress = "aa:aa:aa:aa:aa:00";
};
linkConfig = {
Name = "lan";
};
};
# port 1 is unused
# port 2 will be attached to our failover 4G router
# but that's for a future post
"20-wan-fttp" = {
# port 3 is plugged into my ONT
matchConfig = {
MACAddress = "aa:aa:aa:aa:aa:03";
};
linkConfig = {
Name = "wan-fttp";
};
};
};
networks = {
"10-lan" = {
linkConfig.RequiredForOnline = "yes";
matchConfig.Name = "lan";
address = [
"192.168.32.1/23"
"2001:x:x:x::1/64"
];
dns = [ "192.168.32.1" ];
domains = [ "home.arpa" ];
networkConfig = {
# have networkd send IPv6 router advertisements
IPv6SendRA = "yes";
};
ipv6SendRAConfig = {
# RAs should include the router's IP for DNS
DNS = "2001:x:x:x::1";
};
ipv6Prefixes = [
{
Prefix = "2001:x:x:x::/64";
Assign = true;
}
];
};
"20-wan-fttp" = {
# networkd should ignore the NIC connected to the ONT
matchConfig.Name = "wan-fttp";
linkConfig.Unmanaged = "yes";
linkConfig.RequiredForOnline = "no";
};
};
};
# fttp pppoe
age.secrets.pppoe-chap-secrets = {
file = ../../secrets/pppoe-chap-secrets.age;
path = "/etc/ppp/chap-secrets";
owner = "root";
group = "root";
mode = "0600";
};
services.pppd = {
enable = true;
peers = {
# the peer name here, and ifname below can be specific to your setup
aaisp-pppoe = {
autostart = true;
enable = true;
# wan-fttp is the named NIC from the networkd configuration
# if your ISP doesn't offer baby-jumbo frames, set mtu to 1492
config = ''
plugin pppoe.so wan-fttp
name "<USERNAME>"
noipdefault
hide-password
lcp-echo-interval 1
lcp-echo-failure 4
noauth
persist
maxfail 0
holdoff 5
mtu 1500
noaccomp
default-asyncmap
+ipv6
ipv6cp-use-ipaddr
ifname pppoe-aaisp
'';
};
};
};
systemd.services."pppd-aaisp-pppoe".preStart = ''
# if your ISP doesn't offer baby-jump frames, remove this line
${pkgs.iproute2}/bin/ip link set wan-fttp mtu 1508
# bring up the interface so ppp can use it
${pkgs.iproute2}/bin/ip link set wan-fttp up
'';
# routing (pppoe)
environment.etc.ppp-up = {
# this script runs after PPP has established a connection
# we'll use it to log, and add the default IPv4 and IPv6 routes
enable = true;
target = "ppp/ip-up";
mode = "0755";
text = ''
#!${pkgs.bash}/bin/bash
${pkgs.logger}/bin/logger "$1 is up"
if [ $IFNAME = "pppoe-aaisp" ]; then
${pkgs.logger}/bin/logger "AAISP - FTTP PPPoE online"
${pkgs.logger}/bin/logger "Add default routes via PPPoE"
${pkgs.iproute2}/bin/ip route add default dev pppoe-aaisp scope link metric 100
${pkgs.iproute2}/bin/ip -6 route add default dev pppoe-aaisp scope link metric 100
fi
'';
};
environment.etc.ppp-down = {
# this script runs after the PPP connection drops
# we'll use it to log, and remove the default routes
enable = true;
target = "ppp/ip-down";
mode = "0755";
text = ''
#!${pkgs.bash}/bin/bash
${pkgs.logger}/bin/logger "$1 is down"
if [ $IFNAME = "pppoe-aaisp" ]; then
${pkgs.logger}/bin/logger "AAISP - FTTP PPPoE offline"
${pkgs.logger}/bin/logger "Remove default routes via PPPoE"
${pkgs.iproute2}/bin/ip route del default dev pppoe-aaisp scope link metric 100
${pkgs.iproute2}/bin/ip -6 route del default dev pppoe-aaisp scope link metric 100
fi
'';
};
# dnsmasq configuration, providing DHCP and internal DNS
# (based on DHCP clients and /etc/hosts)
services.dnsmasq = {
enable = true;
resolveLocalQueries = false;
settings = {
# bind to 8053, we want adguard to provide DNS
# and we'll let resolved own the loopback port 53
port = 8053;
no-resolv = true;
bind-dynamic = true;
dhcp-authoritative = true;
domain-needed = true;
domain = "home.arpa";
local = "/home.arpa/";
dhcp-range = [
"set:lan,192.168.32.100,192.168.33.200,255.255.254.0,12h"
];
dhcp-option = [
"tag:lan,option:dns-server,192.168.32.1"
];
dhcp-host = [
# add any static DHCP IPs you want to assign here
"aa:aa:aa:aa:aa:10,192.168.32.5,core-switch"
];
};
};
services.adguardhome = {
enable = true;
# any changes made through the web UI will be thrown away
# on rebuild with this setting...
mutableSettings = false;
settings = {
# note this configuration logs queries by default
# check the docs if you want to avoid this
# this will allow unauthenticated access to the adguard UI
# to any host on your LAN.
# Change it to 127.0.0.1 if you do not want this
host = "192.168.32.1";
dns = {
bind_hosts = [
# trusted lan
"192.168.32.1"
"2001:x:x:x::1"
];
port = 53;
# some optimisations I found necessary
ratelimit = 0;
cache_size = 67108864;
max_goroutines = 500;
use_http3_upstreams = true;
upstream_dns = [
# you may prefer to use your own ISPs DNS
"https://dns.quad9.net/dns-query"
"https://dns.mullvad.net/dns-query"
"https://cloudflare-dns.com/dns-query"
# requests for the local domain go to dnsmasq
"[/home.arpa/]127.0.0.1:8053"
];
local_ptr_upstreams = [
# reverse lookups for local IPs go to dnsmasq
"127.0.0.1:8053"
];
bootstrap_dns = [
# you may prefer to use your own ISPs DNS
"9.9.9.10"
"149.112.112.10"
"2620:fe::10"
"2620:fe::fe:10"
];
};
};
};
networking.nftables.ruleset = ''
table inet firewall {
chain rpfilter {
type filter hook prerouting priority mangle + 10; policy drop;
meta nfproto ipv4 udp sport . udp dport { 68 . 67, 67 . 68 } accept comment "DHCPv4 client/server"
fib saddr . mark oif exists accept
}
chain input {
type filter hook input priority filter; policy drop;
# assuming we trust our LAN clients
iifname { "lo", "lan" } accept comment "trusted interfaces"
# handle packets according to connection state
ct state vmap { invalid : drop, established : accept, related : accept, new : jump input-allow, untracked : jump input-allow }
# if we make it here, block and log
tcp flags syn / fin,syn,rst,ack log prefix "refused connection: " level info
}
chain input-allow {
# make your own choice on whether to allow SSH from outside
tcp dport 22 accept comment "ssh from anywhere"
icmp type echo-request accept comment "allow ping"
icmpv6 type != { nd-redirect, 139 } accept comment "Accept all ICMPv6 messages except redirects and node information queries (type 139). See RFC 4890, section 4.4."
ip6 daddr fe80::/64 udp dport 546 accept comment "DHCPv6 client"
}
chain forward {
type filter hook forward priority 0; policy drop;
# no internet egress to RFC1918 IPs
oifname "pppoe-aaisp" ip daddr { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 } reject with icmp type net-unreachable comment "outbound rfc1918 not permitted"
# established/related allowed, invalid dropped
ct state vmap { established : accept, related : accept, invalid : drop }
# internal interfaces outbound allowed
iifname "lan" oifname "pppoe-aaisp" accept comment "internal networks out via ISP"
# allow icmp
icmp type echo-request accept comment "allow ping"
icmpv6 type != { nd-redirect, 139 } accept comment "Accept all ICMPv6 messages except redirects and node information queries (type 139). See RFC 4890, section 4.4."
# log anything that was blocked
tcp flags syn / fin,syn,rst,ack log prefix "refused forward: " level info
}
}
table ip nat {
chain pre {
type nat hook prerouting priority dstnat; policy accept;
# we'll add rules for our 1:1 NAT here later
}
chain post {
type nat hook postrouting priority srcnat; policy accept;
iifname "lan" oifname "pppoe-aaisp" masquerade comment "LAN NAT to FTTP"
}
chain out {
type nat hook output priority mangle; policy accept;
# we'll add rules for our 1:1 NAT here later
}
}
'';
system.stateVersion = "24.11";
}