Slightly Nerdy

NixOS is great for self-hosting stuff

Fri, 03 Apr 2026 22:50:00 +0100

This isn't a how-to or guide, just a show and tell of something I think is cool.

I've been using NixOS for a few years now. I continue to find it frustrating, in large part because I still haven't internalised the language and how everything hangs together to build a system. Maybe I'll get there eventually, but in the mean time I know just enough to ~~be dangerous~~ assemble useful collections of services.

For a server, there's something magical about having pre-written modules to enable and configure a whole bunch of services.

This week, after a particularly trying experience with Google Docs at work, I wanted to try out HedgeDoc. So I threw up a copy on my home server by writing this file and dropping it into my Nix repo, building on what I've setup previously...

{ ... }:
{
  # I run a bunch of services inside a NixOS container,
  # which is "system" container (ala Incus or Proxmox)
  # rather than an application container (ala Docker)
  # I've got a wildcard DNS entry pointing to this
  # container's IP
  containers.intsvcs = {
    bindMounts = {
      "/var/lib/hedgedoc" = {
        # on the host /services is a ZFS dataset, and
        # I create a dataset for each application.
        # By being in here, it'll get snapshotted
        # every 10 minutes and sent over to my Netcup VPS
        # for fast recovery if I need it, in addition
        # to a daily snapshot that's captured by restic,
        # stored locally and rsync'd to a Hetzner Storage
        # Box
        hostPath = "/services/hedgedoc";
        isReadOnly = false;
      };
    };
    config = {
      # I use traefik as my frontend load balancer, it's
      # configured elsewhere with it's basic settings and 
      # wired up with a wildcard LetsEncrypt cert managed
      # by the NixOS `services.acme` module.
      # Here, I just need to tell it about my new service
      services.traefik.dynamicConfigOptions.http = {
        services.docs = {
          loadBalancer = {
            servers = [ { url = "http://localhost:3001"; } ];
          };
        };
        routers.docs = {
          rule = "Host(`docs.mydomain`)";
          service = "docs";
          tls = { };
        };
      };
      services.gatus.settings.endpoints = [
        # I use Gatus to keep an eye on my services, alert
        # me if they're down, or the certifate renewal is
        # getting nearer than it really should. I just need
        # to add an entry into it's endpoints list...
        {
          name = "Hedgedoc";
          group = "Intsvcs";
          interval = "30s";
          url = "https://docs.mydomain";
          conditions = [
            "[STATUS] == 200"
            "[CERTIFICATE_EXPIRATION] > 240h"
          ];
          alerts = [ { type = "ntfy"; } ];
        }
      ];
      services.victoriametrics.prometheusConfig.scrape_configs = [
        # victoria metrics collects all the metrics from my
        # various services, we just need to let it know
        # about this new one...
        {
          job_name = "hedgedoc";
          static_configs = [ { targets = [ "http://127.0.0.1:3001/metrics" ]; } ];
          relabel_configs = [
            {
              source_labels = [ "__address__" ];
              regex = ".*";
              target_label = "instance";
              replacement = "intsvcs";
            }
          ];
        }
      ];
      services.postgresql = {
        # quite a few of the services running in this container
        # make use of postgres, sometimes the NixOS module does all
        # the work for me, other times, like this one, I just need
        # to add to the existing configuration to let it know I'd like
        # a new DB and user (relies on a system user, which the hedgedoc
        # module will create for us)
        ensureUsers = [
          {
            name = "hedgedoc";
            ensureDBOwnership = true;
          }
        ];
        ensureDatabases = [ "hedgedoc" ];
      };
      services.hedgedoc = {
        # and finally the service itself, a handful of lines
        # here to get the hedgedoc config right for my selfup
        # and we're good to go.
        enable = true;
        settings = {
          port = 3001;
          db = {
            dialect = "postgresql";
            host = "/run/postgresql";
            database = "hedgedoc";
          };
          domain = "docs.mydomain";
          protocolUseSSL = true;
          allowOrigin = [ "docs.mydomain" ];
          allowAnonymous = false;
          # not shown are a handful of extra lines configuring
          # it for OAuth2 authentication with Keycloack, the one
          # bit of manual setup I had to do, creating a new client
          # there and dropping in the details here
        };
      };
    };
  };
}

A minute of nixos-rebuild later, and HedgeDoc is up and running.

I like how I can divide up the configuration for common components (Traefik, PostgreSQL, Gatus and VictoriaMetrics) so that the relevant parts live along the services they're being used for.

If I wanted to get fancier, I could save myself some boilerplate and make a module with a small handful of inputs (hostnames, ports, paths) that turns all the common stuff into a couple of lines.

Maybe I'll do that next, or maybe it's time I got around to having native PostgreSQL backups instead of relying on just ZFS snapshots, services.postgresqlBackup should make that a breeze.

And if I decide in a couple of weeks that HedgeDoc isn't for me? I can just delete the file, nixos-rebuild and it's gone.

Disabling Javascript

Fri, 03 Apr 2026 08:45:00 +0100

I was reading a blog post¹ the other day, and in my usual ADHD way, was flicking back and forth to other tabs. I got a little confused when I went to go back to it, and the title and favicon had changed to something entirely different. Returning to the tab, an overlay had been placed over the page, making the case for disabling Javascript by default, and linking to disable-javascript.org.

It turns out, you don't need to do anything as drastic as disabling JS for the entire browser these days, you can ask uBlock Origin to instead have it disabled by default and selectively allow it as needed.

So I gave it a try. Here's what I found...

The frustrating parts

Many sites straight up don't work, and of those ~20% show a message about requiring Javascript to function, the rest leave you in limbo.

Of these, some make sense given their purpose, and for others it's just baffling why they're built that way.

One site containing an article had some coloured blocks and no text, but clicking the Firefox reader mode presented the otherwise invisible article.

It's a shame Mastodon doesn't return the contents of a toot when directly linked to, but it does at least let you know directly that it needs Javascript.

The PEBKAC part

After a few days, I forgot about it, and ocassionally landed on a new site, or one of my own self-hosted services I'd not visited in a while and thought something was wrong with my Internet. The pink (instead of the usual grey) blocked resource counter in the uBlock extension is a good, but not always consistent reminder of this.

The good parts

Many sites mostly work, and are even improved. All those data sharing consents, social login prompts, giant begging overlays and demands to register, "this is your last free article", etc just stop existing. News sites especially are much improved, usually at the cost of non-working embedded videos and articles where they've tried to do fancy presentation tricks. And clicking around just feels snappier.

And then there's the satisfaction of all the sites that work just fine where you see that uBlock has blocked Javascript and know that whatever it was, it probably wasn't there to improve your experience.

In conclusion

Parts of the web are greatly enhanced by turning Javascript off. Sometimes it's annoying having to do three extra clicks every time you visit a new site that doesn't function without it.

For me, it's worthwhile, so I'm going to stick with it.

¹ With apologies for not crediting the blog, I can't recall it now, and because the post was entirely unrelated, I've not been able to find it again. But you can take a look here, where the same code is running. Just click away to another tab for a moment.

Building a NixOS router for AAISP - L2TP Failover over 4G

Sat, 14 Dec 2024 16:24:00 +0000

Following on from setting up the basics of our router, here's how I add a second WAN connection via a 4G router, and use Andrews & Arnold's L2TP service to keep my public IPv4 and IPv6 networks available in the event of a failure of my fibre service.

You can simplify the configuration here to just use the 4G connection directly if you don't have an ISP that's quite so fancy. Let me know if you'd like me to write about that.

The 4G router is one I picked up a while ago from the Three network, a ZTE MF286D. I do nothing special to it's configuration, and have it running in router mode. It does offer a bridge mode where the external IPs are passed to a connected device, but we don't need that here.

I have an 80Gb/month prepaid 26 month SIM (affiliate link) from Safecom that uses the Three network. It was £60 when I bought it, and is more expensive now, but it's worth checking out there various options over time to get the best prices. I reckon it should cover 2-4 days in a month, so long as we avoid watching a lot of streaming TV.

Things are configured such that when the fibre's PPPoE connection ends, the L2TP session is initiated. And when the PPPoE connection starts, the L2TP session is terminated. Notably this doesn't cover the router booting whilst the fibre is down, which is something to fix.

In Linux L2TP is typically implemented with a combination xl2tpd and pppd. There's a NixOS module for xl2tpd, but it makes some assumptions that mean it won't work for this use case. Typically L2TP is used with IPsec for some old school VPN solutions, rather than on it's own, as we need it.

So the first job is to create a new NixOS module, which we can base on the upstream version and just chop it up a bit to pass more configurability up the stack. I'll be the first to admit my Nix-fu is not strong yet, so I'm sure there are better ways to do this. But it works for me!

# modules/nixos/xl2tpd-flexible.nix
{ config, pkgs, lib, ... }:

with lib;

{
  options = {
    services.xl2tpd-flexible = {
      enable = mkEnableOption "xl2tpd, with more flexible configuration";

      xl2tpOptions = mkOption {
        type        = types.lines;
        description = "xl2tpd configuration file content";
        default     = "";
      };

      pppOptions = mkOption {
        type        = types.lines;
        description = "ppp options file content";
        default     = "";
      };

    };
  };

  config = mkIf config.services.xl2tpd-flexible.enable {
    systemd.services.xl2tpd-flexible = let
      cfg = config.services.xl2tpd-flexible;

      pppd-options = pkgs.writeText "ppp-options-xl2tpd.conf" cfg.pppOptions;

      xl2tpd-conf = pkgs.writeText "xl2tpd.conf" ''
        ${cfg.xl2tpOptions}
        pppoptfile = ${pppd-options}
      '';

      xl2tpd-ppp-wrapped = pkgs.stdenv.mkDerivation {
        name         = "xl2tpd-ppp-wrapped";
        phases       = [ "installPhase" ];
        nativeBuildInputs  = with pkgs; [ makeWrapper ];
        installPhase = ''
          mkdir -p $out/bin

          makeWrapper ${pkgs.ppp}/sbin/pppd $out/bin/pppd \
            --set LD_PRELOAD    "${pkgs.libredirect}/lib/libredirect.so" \
            --set NIX_REDIRECTS "/etc/ppp=/etc/xl2tpd/ppp"

          makeWrapper ${pkgs.xl2tpd}/bin/xl2tpd $out/bin/xl2tpd \
            --set LD_PRELOAD    "${pkgs.libredirect}/lib/libredirect.so" \
            --set NIX_REDIRECTS "${pkgs.ppp}/sbin/pppd=$out/bin/pppd"
        '';
      };
    in {
      description = "xl2tpd flexible server";

      requires = [ "network-online.target" ];
      wantedBy = [ "multi-user.target" ];

      preStart = ''
        mkdir -p -m 700 /etc/xl2tpd/ppp
        mkdir -p -m 700 /run/xl2tpd
      '';

      serviceConfig = {
        ExecStart = "${xl2tpd-ppp-wrapped}/bin/xl2tpd -D -c ${xl2tpd-conf} -p /run/xl2tpd/pid -C /run/xl2tpd/control";
        KillMode  = "process";
        Restart   = "on-success";
        Type      = "simple";
        PIDFile   = "/run/xl2tpd/pid";
      };
    };
  };
}

We need to configure our additional WAN port, which builds on our configuration from the previous post. xl2tpd doesn't support connecting to an IPv6 endpoint, and although my SIM gives me a /64 network, I chose not to configure it here to keep things simple.

# nixos/hermes/default.nix

{ config, pkgs, ... }:
{
  systemd.network = {
    links = {
      "20-wan-4g" = {
        matchConfig = {
          MACAddress = "aa:aa:aa:aa:aa:02";  # port 2
        };
        linkConfig = {
          Name = "wan-4g";
        };
      };
    };
    networks = {
      "20-wan-4g" = {
        matchConfig.Name = "wan-4g";
        networkConfig = {
          DHCP = "ipv4";
          IPv6AcceptRA = "no";
        };
        dhcpV4Config = {
          ClientIdentifier = "mac";
          RouteMetric = 200;
        };
        linkConfig.RequiredForOnline = "no";
      };
    };
  };
}

Note the RouteMetric for the wan-4g is 200, where I used 100 for pppoe-aaisp. This means we won't use the 4G connection's default route so long as the fibre is up.

We'll use our xl2tpd-flexible module and some additional PPP up/down scripts to configure the L2TP service, and modify our PPPoE PPP scripts to trigger the connection. And we'll add a new reference to our CHAP secrets specifically for the pppd used by xl2tpd.

# nixos/hermes/default.nix
{ config, pkgs, ... }:
{
  # l2tp failover
  age.secrets.xl2tpd-chap-secrets = {
    file = ../../secrets/pppoe-chap-secrets.age;
    path = "/etc/xl2tpd/ppp/chap-secrets";
    owner = "root";
    group = "root";
    mode = "0600";
  };
  services.xl2tpd-flexible = {
    enable = true;
    xl2tpOptions = ''
      [global]
      max retries = 36

      [lac aaisp]
      lns = 194.4.172.12
      require authentication = no
      redial = yes
      redial timeout = 10
    '';
    pppOptions = ''
      +ipv6
      ipv6cp-use-ipaddr
      name <USERNAME>
      noauth
      ifname l2tp-aaisp
    '';
  };

  # routing (pppoe)
  environment.etc.ppp-up = {
    enable = true;
    target = "ppp/ip-up";
    mode = "0755";
    text = ''
      #!${pkgs.bash}/bin/bash
      ${pkgs.logger}/bin/logger "$1 is up"
      if [ $IFNAME = "pppoe-aaisp" ]; then
        ${pkgs.logger}/bin/logger "AAISP - FTTP PPPoE online"

        ${pkgs.logger}/bin/logger "Bring L2TP down and remove routes"
        echo "d aaisp" > /run/xl2tpd/control

        ${pkgs.logger}/bin/logger "Add default routes via PPPoE"
        ${pkgs.iproute2}/bin/ip route add default dev pppoe-aaisp scope link metric 100
        ${pkgs.iproute2}/bin/ip -6 route add default dev pppoe-aaisp scope link metric 100
      fi
    '';
  };
  environment.etc.ppp-down = {
    enable = true;
    target = "ppp/ip-down";
    mode = "0755";
    text = ''
      #!${pkgs.bash}/bin/bash
      ${pkgs.logger}/bin/logger "$1 is down"
      if [ $IFNAME = "pppoe-aaisp" ]; then
        ${pkgs.logger}/bin/logger "AAISP - FTTP PPPoE offline"

        ${pkgs.logger}/bin/logger "Remove default routes via PPPoE"
        ${pkgs.iproute2}/bin/ip route del default dev pppoe-aaisp scope link metric 100
        ${pkgs.iproute2}/bin/ip -6 route del default dev pppoe-aaisp scope link metric 100

        ${pkgs.logger}/bin/logger "Connect via LT2P failover"
        echo "c aaisp" > /run/xl2tpd/control

      fi
    '';
  };

  # routing (l2tp)
  environment.etc.l2tp-up = {
    enable = true;
    target = "xl2tpd/ppp/ip-up";
    mode = "0755";
    text = ''
      #!${pkgs.bash}/bin/bash
      ${pkgs.logger}/bin/logger "$1 is up"
      if [ $IFNAME = "l2tp-aaisp" ]; then
        ${pkgs.logger}/bin/logger "AAISP - L2TP failover online"

        ${pkgs.logger}/bin/logger "Add default routes via L2TP"
        ${pkgs.iproute2}/bin/ip route add 194.4.172.12/32 via 192.168.100.1 metric 10
        ${pkgs.iproute2}/bin/ip route add default dev l2tp-aaisp scope link metric 100
        ${pkgs.iproute2}/bin/ip -6 route add default dev l2tp-aaisp scope link metric 100
      fi
    '';
  };
  environment.etc.l2tp-down = {
    enable = true;
    target = "xl2tpd/ppp/ip-down";
    mode = "0755";
    text = ''
      #!${pkgs.bash}/bin/bash
      ${pkgs.logger}/bin/logger "$1 is down"
      if [ $IFNAME = "l2tp-aaisp" ]; then
        ${pkgs.logger}/bin/logger "AAISP - L2TP failover offline"

        ${pkgs.logger}/bin/logger "Remove default route via L2TP"
        ${pkgs.iproute2}/bin/ip route del default dev l2tp-aaisp scope link metric 100
        ${pkgs.iproute2}/bin/ip -6 route del default dev l2tp-aaisp scope link metric 100
        ${pkgs.iproute2}/bin/ip route del 194.4.172.12/32 via 192.168.100.1 metric 10
      fi
    '';
  };

}

Changes you might want to make include references to 194.4.172.12, which is the L2TP service endpoint for Andrews & Arnold, and 192.168.100.1 which is IP of the 4G router. We add a static route for the L2TP endpoint via this address to prevent it inceptioning.

Lastly, we need to modify our firewall rules to allow traffic through the new l2tp-aaisp interface and to NAT traffic on the way out.

{ config, pkgs, ... }:
{
  networking.nftables.ruleset = ''
    table inet firewall {
      chain rpfilter {
        type filter hook prerouting priority mangle + 10; policy drop;
        meta nfproto ipv4 udp sport . udp dport { 68 . 67, 67 . 68 } accept comment "DHCPv4 client/server"
        fib saddr . mark oif exists accept
      }

      chain input {
        type filter hook input priority filter; policy drop;

        # assuming we trust our LAN clients
        iifname { "lo", "lan" } accept comment "trusted interfaces"

        # handle packets according to connection state
        ct state vmap { invalid : drop, established : accept, related : accept, new : jump input-allow, untracked : jump input-allow }

        # if we make it here, block and log
        tcp flags syn / fin,syn,rst,ack log prefix "refused connection: " level info
      }

      chain input-allow {
        # make your own choice on whether to allow SSH from outside
        tcp dport 22 accept comment "ssh from anywhere"

        icmp type echo-request accept comment "allow ping"
        icmpv6 type != { nd-redirect, 139 } accept comment "Accept all ICMPv6 messages except redirects and node information queries (type 139).  See RFC 4890, section 4.4."
        ip6 daddr fe80::/64 udp dport 546 accept comment "DHCPv6 client"
      }

      chain forward {
        type filter hook forward priority 0; policy drop;

        # no internet egress to RFC1918 IPs
        oifname { "pppoe-aaisp", "l2tp-aaisp" } ip daddr { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 } reject with icmp type net-unreachable comment "outbound rfc1918 not permitted"

        # established/related allowed, invalid dropped
        ct state vmap { established : accept, related : accept, invalid : drop }

        # internal interfaces outbound allowed
        iifname "lan" oifname { "pppoe-aaisp", "l2tp-aaisp" } accept comment "internal networks out via ISP"

        # allow icmp
        icmp type echo-request accept comment "allow ping"
        icmpv6 type != { nd-redirect, 139 } accept comment "Accept all ICMPv6 messages except redirects and node information queries (type 139).  See RFC 4890, section 4.4."

        # log anything that was blocked
        tcp flags syn / fin,syn,rst,ack log prefix "refused forward: " level info
      }
    }

    table ip nat {
      chain pre {
        type nat hook prerouting priority dstnat; policy accept;
        # we'll add rules for our 1:1 NAT here later
      }

      chain post {
        type nat hook postrouting priority srcnat; policy accept;

        iifname "lan" oifname { "pppoe-aaisp", "l2tp-aaisp" } masquerade comment "LAN NAT to FTTP"
      }

      chain out {
        type nat hook output priority mangle; policy accept;
        # we'll add rules for our 1:1 NAT here later
      }
    }
  '';
}

And that's it! I'm able to bring down the FTTP link by pulling the cable or powering off the ONT, and in under 10 seconds, everything is connected again, albeit with much less bandwidth and much higher latency!

In the next post, I'll add VLANs for IoT devices, guests, self hosted services, and private clients/services that I prefer to reach the Internet via Mullvad's VPN service.

Building a NixOS router for a UK FTTP ISP - The Basics

Fri, 13 Dec 2024 14:10:00 +0000

I use NixOS as a router for my FTTP ISP in the UK. The details in this post should apply to most PPPoE-delivered services.

In a later post I'll add 4G failover in a way that's more specific to my ISP, then add multiple VLANs and 1:1 NAT to make the most of the additional IPv4 addresses my ISP offers.

I picked up one of those cheap AliExpress tiny PCs a few months ago to have something dedicated to the router role in my setup. I'd previously used the same machine that self hosts my various services for the role, along with a USB ethernet dongle for the WAN side, and I really wanted to decouple them, with the router having a much smaller footprint and attack surface.

The particular model I got was the "SZBOX G48S Alder Lake N100 Soft Router", with 8Gb RAM, 256Gb NVMe SSD and a 4-core Intel N100 CPU. It was £151 delivered, including UK power cable. It's £131 today!

I don't necessarily recommend it, so I'm not linking to it directly. It rejected a 16Gb stick of Crucial RAM I tried that works fine in another system (with a lot of hard-hangs). I'm sure not expecting updates to the BIOS. But I wiped the Kingston SSD it shipped with, tried out some things, and reassured myself that it's not behaving nefariously in a way that I could detect. Be careful out there.

The thing most attractive to me about it was the fanless design and 4x 2.5GbE Intel I226-V NICs. Perfect (or massive overkill) for a "home" router.

I'm not going to get into the weeds on NixOS in this post, and will assume you're comfortable installing it. I use a flake-based approach to my systems, all defined in a single Git repo. It's a bit of a time sink getting familiar with it, but for me has been worth it for how quickly I can now configure new systems and services.

Below is a simplified version of the module for this host. It's missing things like an SSH server, a local non-root user, and some commands you'll probably want to add to debug things (tcpdump, etc).

It's opinionated, I prefer to use systemd-networkd where possible, and use nftables directly for messing with packets rather than the NixOS firewall module.

I use dnsmasq for DHCP (IPv4) and internal DNS, with AdGuard Home for ad-blocking DNS. My internal DNS domain is home.arpa.

I make use of agenix to keep encrypted secrets in my config repo. It's also possible to provide the password directly in the PPP config, but I wouldn't recommend it.

{ config, lib, pkgs, ... }:

{
  imports =
    [
      ./hardware-configuration.nix
    ];

  # allocate some of our RAM to use as compressed swap
  # which I use instead of disk-backed swap
  zramSwap.enable = true;

  boot.kernel.sysctl = {
    # be more swappy as we're using zramswap
    "vm.swappiness" = 100;

    # enable IPv4 and IPv6 forwarding on all interfaces
    "net.ipv4.conf.all.forwarding" = true;
    "net.ipv6.conf.all.forwarding" = true;

    "net.ipv4.conf.all.arp_filter" = 1;
    "net.ipv4.conf.default.arp_filter" = 1;
  };

  # sleep any attached screen after 5 minutes
  boot.kernelParams = [ "consoleblank=300" ];

  networking = {
    hostName = "hermes";
    useDHCP = false;
    useNetworkd = true;
    nftables.enable = true;

    # we write our own rules
    firewall.enable = false;

    # these will be available in the local DNS
    # along with DHCP hostnames
    extraHosts = ''
      192.168.32.1 hermes.home.arpa
      2001:x:x:x::1 hermes.home.arpa
    '';
  };

  # the basic network configuration for systemd-networkd
  systemd.network = {
    enable = true;
    wait-online.enable = false;
    links = {
      "10-lan" = {
        # port 0 is plugged into my lan switch
        matchConfig = {
          MACAddress = "aa:aa:aa:aa:aa:00";
        };
        linkConfig = {
          Name = "lan";
        };
      };
      # port 1 is unused
      # port 2 will be attached to our failover 4G router
      # but that's for a future post
      "20-wan-fttp" = {
        # port 3 is plugged into my ONT
        matchConfig = {
          MACAddress = "aa:aa:aa:aa:aa:03";
        };
        linkConfig = {
          Name = "wan-fttp";
        };
      };
    };
    networks = {
      "10-lan" = {
        linkConfig.RequiredForOnline = "yes";
        matchConfig.Name = "lan";
        address = [
          "192.168.32.1/23"
          "2001:x:x:x::1/64"
        ];
        dns = [ "192.168.32.1" ];
        domains = [ "home.arpa" ];
        networkConfig = {
          # have networkd send IPv6 router advertisements
          IPv6SendRA = "yes";
        };
        ipv6SendRAConfig = {
         # RAs should include the router's IP for DNS
          DNS = "2001:x:x:x::1";
        };
        ipv6Prefixes = [
          {
            Prefix = "2001:x:x:x::/64";
            Assign = true;
          }
        ];
      };
      "20-wan-fttp" = {
        # networkd should ignore the NIC connected to the ONT
        matchConfig.Name = "wan-fttp";
        linkConfig.Unmanaged = "yes";
        linkConfig.RequiredForOnline = "no";
      };
    };
  };

  # fttp pppoe
  age.secrets.pppoe-chap-secrets = {
    file = ../../secrets/pppoe-chap-secrets.age;
    path = "/etc/ppp/chap-secrets";
    owner = "root";
    group = "root";
    mode = "0600";
  };
  services.pppd = {
    enable = true;
    peers = {
      # the peer name here, and ifname below can be specific to your setup
      aaisp-pppoe = {
        autostart = true;
        enable = true;
        # wan-fttp is the named NIC from the networkd configuration
        # if your ISP doesn't offer baby-jumbo frames, set mtu to 1492
        config = ''
          plugin pppoe.so wan-fttp
          name "<USERNAME>"

          noipdefault
          hide-password
          lcp-echo-interval 1
          lcp-echo-failure 4
          noauth
          persist
          maxfail 0
          holdoff 5
          mtu 1500
          noaccomp
          default-asyncmap
          +ipv6
          ipv6cp-use-ipaddr
          ifname pppoe-aaisp
        '';
      };
    };
  };
  systemd.services."pppd-aaisp-pppoe".preStart = ''
    # if your ISP doesn't offer baby-jumbo frames, remove this line
    ${pkgs.iproute2}/bin/ip link set wan-fttp mtu 1508  
    # bring up the interface so ppp can use it
    ${pkgs.iproute2}/bin/ip link set wan-fttp up
  '';
  # routing (pppoe)
  environment.etc.ppp-up = {
    # this script runs after PPP has established a connection
    # we'll use it to log, and add the default IPv4 and IPv6 routes
    enable = true;
    target = "ppp/ip-up";
    mode = "0755";
    text = ''
      #!${pkgs.bash}/bin/bash
      ${pkgs.logger}/bin/logger "$1 is up"
      if [ $IFNAME = "pppoe-aaisp" ]; then
        ${pkgs.logger}/bin/logger "AAISP - FTTP PPPoE online"

        ${pkgs.logger}/bin/logger "Add default routes via PPPoE"
        ${pkgs.iproute2}/bin/ip route add default dev pppoe-aaisp scope link metric 100
        ${pkgs.iproute2}/bin/ip -6 route add default dev pppoe-aaisp scope link metric 100
      fi
    '';
  };
  environment.etc.ppp-down = {
    # this script runs after the PPP connection drops
    # we'll use it to log, and remove the default routes
    enable = true;
    target = "ppp/ip-down";
    mode = "0755";
    text = ''
      #!${pkgs.bash}/bin/bash
      ${pkgs.logger}/bin/logger "$1 is down"
      if [ $IFNAME = "pppoe-aaisp" ]; then
        ${pkgs.logger}/bin/logger "AAISP - FTTP PPPoE offline"

        ${pkgs.logger}/bin/logger "Remove default routes via PPPoE"
        ${pkgs.iproute2}/bin/ip route del default dev pppoe-aaisp scope link metric 100
        ${pkgs.iproute2}/bin/ip -6 route del default dev pppoe-aaisp scope link metric 100

      fi
    '';
  };

  # dnsmasq configuration, providing DHCP and internal DNS
  # (based on DHCP clients and /etc/hosts)
  services.dnsmasq = {
    enable = true;
    resolveLocalQueries = false;
    settings = {
      # bind to 8053, we want adguard to provide DNS
      # and we'll let resolved own the loopback port 53
      port = 8053;
      no-resolv = true;
      bind-dynamic = true;
      dhcp-authoritative = true;
      domain-needed = true;

      domain = "home.arpa";
      local = "/home.arpa/";

      dhcp-range = [
        "set:lan,192.168.32.100,192.168.33.200,255.255.254.0,12h"
      ];
      dhcp-option = [
        "tag:lan,option:dns-server,192.168.32.1"
      ];

      dhcp-host = [
        # add any static DHCP IPs you want to assign here
        "aa:aa:aa:aa:aa:10,192.168.32.5,core-switch"
        ];

    };
  };

  services.adguardhome = {
    enable = true;
    # any changes made through the web UI will be thrown away
    # on rebuild with this setting...
    mutableSettings = false;
    settings = {
      # note this configuration logs queries by default
      # check the docs if you want to avoid this

      # this will allow unauthenticated access to the adguard UI
      # to any host on your LAN.
      # Change it to 127.0.0.1 if you do not want this
      host = "192.168.32.1";

      dns = {
        bind_hosts = [
          # trusted lan
          "192.168.32.1"
          "2001:x:x:x::1"
        ];
        port = 53;
        # some optimisations I found necessary
        ratelimit = 0;
        cache_size = 67108864;
        max_goroutines = 500;
        use_http3_upstreams = true;
        upstream_dns = [
          # you may prefer to use your own ISPs DNS
          "https://dns.quad9.net/dns-query"
          "https://dns.mullvad.net/dns-query"
          "https://cloudflare-dns.com/dns-query"
          # requests for the local domain go to dnsmasq
          "[/home.arpa/]127.0.0.1:8053"
        ];
        local_ptr_upstreams = [
          # reverse lookups for local IPs go to dnsmasq
          "127.0.0.1:8053"
        ];
        bootstrap_dns = [
          # you may prefer to use your own ISPs DNS
          "9.9.9.10"
          "149.112.112.10"
          "2620:fe::10"
          "2620:fe::fe:10"
        ];
      };

    };
  };

  networking.nftables.ruleset = ''
    table inet firewall {
      chain rpfilter {
        type filter hook prerouting priority mangle + 10; policy drop;
        meta nfproto ipv4 udp sport . udp dport { 68 . 67, 67 . 68 } accept comment "DHCPv4 client/server"
        fib saddr . mark oif exists accept
      }

      chain input {
        type filter hook input priority filter; policy drop;

        # assuming we trust our LAN clients
        iifname { "lo", "lan" } accept comment "trusted interfaces"

        # handle packets according to connection state
        ct state vmap { invalid : drop, established : accept, related : accept, new : jump input-allow, untracked : jump input-allow }

        # if we make it here, block and log
        tcp flags syn / fin,syn,rst,ack log prefix "refused connection: " level info
      }

      chain input-allow {
        # make your own choice on whether to allow SSH from outside
        tcp dport 22 accept comment "ssh from anywhere"

        icmp type echo-request accept comment "allow ping"
        icmpv6 type != { nd-redirect, 139 } accept comment "Accept all ICMPv6 messages except redirects and node information queries (type 139).  See RFC 4890, section 4.4."
        ip6 daddr fe80::/64 udp dport 546 accept comment "DHCPv6 client"
      }

      chain forward {
        type filter hook forward priority 0; policy drop;

        # no internet egress to RFC1918 IPs
        oifname "pppoe-aaisp" ip daddr { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 } reject with icmp type net-unreachable comment "outbound rfc1918 not permitted"

        # established/related allowed, invalid dropped
        ct state vmap { established : accept, related : accept, invalid : drop }

        # internal interfaces outbound allowed
        iifname "lan" oifname "pppoe-aaisp" accept comment "internal networks out via ISP"

        # allow icmp
        icmp type echo-request accept comment "allow ping"
        icmpv6 type != { nd-redirect, 139 } accept comment "Accept all ICMPv6 messages except redirects and node information queries (type 139).  See RFC 4890, section 4.4."

        # log anything that was blocked
        tcp flags syn / fin,syn,rst,ack log prefix "refused forward: " level info
      }
    }

    table ip nat {
      chain pre {
        type nat hook prerouting priority dstnat; policy accept;
        # we'll add rules for our 1:1 NAT here later
      }

      chain post {
        type nat hook postrouting priority srcnat; policy accept;

        iifname "lan" oifname "pppoe-aaisp" masquerade comment "LAN NAT to FTTP"
      }

      chain out {
        type nat hook output priority mangle; policy accept;
        # we'll add rules for our 1:1 NAT here later
      }
    }
    '';

  system.stateVersion = "24.11";

}

Switching from Virgin Media cable to Andrews & Arnold over OpenReach FTTP

Wed, 11 Dec 2024 13:06:00 +0000

This started as a post about how I use NixOS to build a router, but the aside on AAISP was getting a little unwieldy. I'll get back to that post another time.

Earlier this year, Fibre To The Premises (FTTP) arrived in my street, and so I took the opportunity to end my 23 year relationship with Virgin Media (previously ntl), the UK's last cable operator standing, and jumped at the chance to switch to Andrews & Arnold (AAISP), probably the nerdiest generally available ISP in the country.

Aside from all the great things I'd heard from others, I knew I was in good hands when I emailed support 30 minutes after placing the order to request a (free) additional IPv4 /30 subnet for a service that wouldn't start until 6 weeks later. I got a reply in under half an hour with my assigned CIDR.

In some ways, one might call the switch a downgrade. With cable I had 1130Mbit/s down, 104Mbit/s up, for £43.52 per month. Now I have 1000Mbit/s down, 115Mbit/s up for £85. So, slower download, faster upload and near twice the price.

What gives?

Well, for one thing, officially the price of the cable package is around £72 per month. Keeping the price down requires invoking the 30-day cancellation period at the end of each 18-month contract cycle, and then waiting for a call a few days or weeks later where they finally offer you the best prices. With AAISP, the price is the price.

And talking of price, at the last negotiation I was paying £40 per month, but then the annual "RPI + a little bit extra for our profit margin" rise kicked in, boosting the price for the remaining 6 months of the contract to £43.52. Illegal in the UK for new contracts from 17th January 2025! VM doesn't seem to be in much of a hurry to get aboard that train until it absolutely has to, with the small print on their current offers stating:

Monthly price of Virgin Media’s main services and O2 Airtime Plan will increase each April from April 2025 by the Retail Price Index rate of inflation announced in February each year plus 3.9%.

No such shenanigans with AAISP.

More importantly, I appreciate how transparent AAISP is, right down to the technical details.

There will always be times when things go wrong, but having visibility and honesty in those times makes for a much less frustrating experience. I generally found VM to be pretty reasonable in my area, but occasionally we'd lose service, and half the time the failures weren't even acknowledged, let alone explained.

Let's discuss AAISP features I use, which VM does not offer on their consumer packages.

IPv6

It's just about to tick over into the year of our lawd 2025, and one of the UK's largest ISPs doesn't offer IPv6! Meanwhile, AAISP assigns my account a /48 IPv6 network, from which I'm free to assign multiple /64 and /60 networks to each of my lines, giving me the freedom to assign separate networks to each of my VLANs (a topic for another time).

Multiple IPv4 addresses

I like to self-host a bunch of services. I'd prefer to keep the publicly accessible ones on a separate IP from the bulk of our family Internet usage. I used to do this via a small VPS with Wireguard, now I can do it directly to home, thanks to AAISP offering a free /30 or /29 IPv4 network, even to their home customers, on request. Notably this block is in addition to the fixed primary address.

Failover

AAISP offer a service called L2TP, which gives you Internet access through their network over the top of an existing connection. I used it for a while to get IPv6 for my home network.

It's also included in their standard broadband packages as a backup. In practice this means I've been able to setup a 4G router with cheap prepaid SIM, connected to my main router. Should there be an issue with the fibre connection, an L2TP connection is automatically initiated over 4G, and the IPs associated with my fibre service are now routed over the cellular network, with the attending drop in bandwidth and increase in latency.

Which is really useful when you're using those IPs to host all the things!

Finally, whilst VM is on-paper faster, at least for downloads, the raw advertised throughput to your ISP is not the only metric for the speed of your broadband. The details of their internal infrastructure and peering to the places you want to go will also impact your experience. I got a little hint of this having previously used AAISP's L2TP service, finding that latency to many places (at least measured by ping) was better with their service running over Virgin Media, than from Virgin Media directly!

Subjectively, it does feel like everything is just a little more snappy now. Objectively, measuring the ping times between my home in Bedford to my VPS in Cambridge, I typically saw 15-20ms over cable, and now 5-8ms with fibre. That's around 30 miles, though it's unlikely the actual path is anything close to the line of sight distance. I don't know for sure, but I would guess the technology differences between cable and FTTP are a factor here, in addition to my choice of ISP.

In conclusion, I'm happy to, and fortunate to be able to pay a premium for a quality service from a company with values I align with.

No money was exchanged for this post, except me paying for my broadband.

My self-hosted services

Tue, 10 Dec 2024 13:09:00 +0000

Here's a snapshot of the services I'm currently running for myself, and approximately how. Some will go, new ones will be added, but here we are for December 2024.

Many, if not all are strictly unnecessary, there are free/cheap hosted alternatives available, but there's something very satisfying about knowing these are sitting on a shelf in the cupboard in my office. Not least that for the most part, these services are incredibly responsive!

In the past I've run a lot of these things in a single-node Kubernetes cluster, mostly using Helm charts. NixOS modules have made things much simpler, whilst still allowing a declarative approach to managing most of these services.

I typically have /services/<service>/<subcomponent> directories on host machines, and have these mounted into the various containers and VMs as necessary to store state. This means the rest of the container/VM is usually ephemeral, and can be re-created at any time. Indeed, I've been able to migrate between machines very easily thanks to this. It also provides a gives me a single volume to back up from each host.

I recently got to grips with microvm.nix, so am starting to migrate things to VMs where that level of isolation feels appropriate.

One thing to be mindful of when using a single Nix expression to define a system composed of many containers/VMs is that nixos-rebuild is pretty expensive in CPU and RAM, taking around 2 minutes to make a simple change and using approaching 10Gb RAM on the machine that hosts the majority of the services below!

AdGuard Home

My network-wide DNS server, also used by Tailscale/Headscale clients, like my phone, to do surprisingly effective ad blocking at the network level.

Deployed using the NixOS module directly on my router host.

Atuin

I'm running my own instance of the sync server for this amazing utility. Once you get used to cross-machine shell history with blazing-fast search, it's hard to go back.

Deployed using the NixOS module which also takes care of configuring a PostgreSQL instance for persistence, all in a NixOS container (a systemd-nspawn container built and managed by NixOS).

Emoncms

Quoting from the website, "Emoncms is a powerful open-source web-app for processing, logging and visualising energy, temperature and other environmental data".

I use it to collect electricity usage data from clamps on my incoming grid power cable, and from my solar PV inverter, in addition to an optical sensor on the electricity meter. It turns out the little flashing LED flashes at a very exact interval according to your electricity use.

It also connects with an EmonEVSE charging station for my car, providing it via MQTT with solar generation data so it can modulate charge according to what's coming from the roof.

Deployed on an EmonPi, a dedicated Raspberry Pi 3 unit in the cupboard with my distribution board, that includes a "hat" for connecting to the clamps, etc.

Forgejo

A nice and surprisingly feature complete alternative to Github/Gitlab that's pretty simple to operate at small scale. Handling SSH Git interactions through my load balancer was probably the hardest part of getting this running.

Deployed using the NixOS module, which takes care of configuring PostgreSQL, all in a NixOS container.

Grafana and Prometheus

Prometheus scrapes metrics from a bunch of these services, including Home Assistant, and from each host device using it's node-exporter tool for data on CPU, memory, network, temperatures, etc.

Grafana lets me build dashboards to query Prometheus and visualise all that data. There are pre-built dashboards for many of these services, and I've got custom ones I use all the time for energy data sources from my battery inverter and EmonCMS (via Home Assistant).

Each is deployed in it's own NixOS container, using the Grafana and Prometheus NixOS modules.

Headscale

An alternative to using the official service-side part of the Tailscale mesh VPN. I installed this recently, and I'm not sure I'm ready to commit to using it, versus manually configuring Wireguard. But it does what it says on the tin. I have noticed lots of connections continue to be made by the client apps to the official endpoints.

Using it to provide DNS where the DNS server is in the Tailnet seems to result in delayed/failed queries for a few moments if the connection has gone idle. This wasn't a problem I saw when previously using vanilla Wireguard for the same thing, likely due to the static configuration.

I also noticed the Wireguard configuration was lighter on my iPhone's battery, likely because it had a much simpler job of connecting to a single, static endpoint.

Deployed in microvm.nix VM using the NixOS module which also configured PostgreSQL.

Home Assistant

The centre of my home automation universe, acting as my Zigbee coordinator, bridging lights, our central heating and ACs to HomeKit.

Deployed as a VM using HAOS using Incus on NixOS. There's a NixOS module, but it's complicated, and the Home Assistant folks are pretty clear that they only actively support their own OS.

Jellyfin

Although I stopped adding to it when streaming came along, I've got a few thousand tracks mostly ripped from CDs, some bought from iTunes back in the day, that have come with me across each new machine. Now I've dropped them in to Jellyfin, along with a few DVD rips, and can access them anywhere, which is nice.

Deployed directly on a NixOS host with the module. Recently moved out of a Docker container inside an Incus Debian container, and probably not in it's final form yet.

Keycloak

I'm running enough services here that maintaining separate logins for them all gets quickly tedious. So wherever a service supports OIDC, which many listed here do, I connect them to Keycloak as my single sign on provider. It's a single place to maintain an account, including 2FA with multiple passkeys, YubiKeys and TOTP codes.

It's configured primarily through the web UI right now, which is something I'd like to fix eventually.

Deployed in a NixOS container using the module which also takes care of a supporting PostgreSQL instance.

Linkding

One of my favourites. I used Linkding as a read-it-later service. I looked at a few alternatives, and I find this one suits me best. It's got a browser extension for quick-adding links, and an iOS app which doesn't have a share-sheet action out the box, but it's pretty simple to create a Shortcut for.

No native Nix package or module for this one! And the author makes Docker the default easy path for deployment, which is understandable. This one might finally get me to try my hand at writing a Nix package and module.

Deployed with Docker Compose inside a dedicated Debian container in Incus.

Mastodon

This is what really got me started on the path to self-hosting. I'm fortunate to have had a very pleasant social media experience on Mastodon, perhaps in part because my single-user instance brings with it some limits on what I see.

Mastodon is by far the most resource intensive service listed here. But it's still pretty small and lives happily alongside all the other stuff on a reasonably modest small form factor machine.

I do use fedifetcher to grab extra context for the toots appearing in my timeline, which is often useful, but definitely adds to the resources. It's packaged in NixOS, but there's no module for it. However it's trivial to create the necessary systemd units in the nix config.

Deployed in a NixOS container using the module which takes care of all supporting services, except for Minio.

Miniflux

Like Linkding, Miniflux is one of many options I looked at, and it's the one that fits me best as an RSS aggregator and reader. I love it's minimalist UI, and it's broad API support for connecting to apps on iOS and elsewhere. And indeed I can save directly to my Linkding instance from the Miniflux web UI.

Deployed in a NixOS container using the module, which also takes care of it's PostgreSQL database.

Minio

Providing an S3-compatible service for object storage, for now just being used for my Mastodon instance's media. Arguably I could have skipped this and let Mastodon manage the media in it's own filesystem, but here we are. I've worked in DevOps long enough to know having something S3-like often ends up being handy.

Deployed as a NixOS container, using the NixOS module.

Nextcloud

Primarily I use it like one might use Dropbox or OneDrive, as a file sync service across devices. Syncthing might be more suitable for this narrow use case.

I do appreciate Nextcloud supporting iOS's native Photos, so that any new photos taken on my phone are automatically uploaded to my server.

It feels like a large attack service, and so is something I want to make VPN-only.

Deployed using the NixOS module, which also takes care of configuring PostgreSQL, Redis and nginx, in a NixOS container. Definitely a candidate for moving to a VM.

Restic

My backup tool of choice, I use the built-in server to receive backups for my various machines on the local network, and remotely over VPN.

I also use rclone to nightly sync these encrypted backups to OneDrive, which has been the cheapest cloud storage I've found. Effectively 6Tb for around £50 per year when you pick up the Microsoft 365 "gift cards" on offer. With the news of them increasing the cost to force AI into the offering, I suspect I'll eventually move on from this solution. But for now, because they allow stacking the gift cards, I'm covered for the next 3 years. Assuming they don't renege on the deal.

Deployed using the module directly on a NixOS host.

SearXNG

My default search "engine" is one I self host! SearXNG is a meta-search tool, running queries against multiple public search services, and gives a clean, ad-free set of results that are usually sufficient.

Deployed using the module in a NixOS container.

Snikket

This is my most recent addition, prompted by Neil's recent post linking to his article on using Snikket/XMPP for over a year.

I previously ran Matrix Synapse for a while, which is pretty effective, but never became embedded in my day-to-day. I'd been meaning to try an XMPP-based solution for a while, so took a look at Snikket, and Prosody, which it's built on.

Prosody has a NixOS module, where Snikket does not. However, it's a pretty complex beast, so there was something quite appealing about dipping my toe with an opinionated all-in-one solution.

Another project where the author targets Docker for deployment. I don't like running Docker natively on my NixOS host that I use for my server, because I like to be in full control of it's network configuration and nftables rules.

I started by trying to use Podman and the oci-containers NixOS module within a NixOS container. Giving the container sufficient privileges to run containers didn't sit well, and it required some hacky use of the module to pass the correct arguments. So this was the project that pushed me to get to grips with microvm.nix, which ended up being quite straightforward, and led me to quickly convert a few containers to VMs.

Deployed using the NixOS oci-containers module, with Podman, inside a microvm.nix VM.

Traefik

Traefik is the glue that binds all my services together and makes them available to the Internet (or just my home network). It's a fairly lightweight load balancer with a configuration format that makes sense to me. I could easily swap in Nginx or similar here, but Traefik suits me well.

I use the security.acme NixOS module to manage certificates with Lets Encrypt, and pass these to the various places they're needed with mounts. I mostly use wildcard certificates to reduce the number required, and do dns-01 validation against my domains hosted with Mythic Beasts, using the built-in support for them in Lego.

The public instance is deployed in a micronix.vm VM using the NixOS module. The private instance uses the same module on the NixOS host.

Vaultwarden

I've been a 1Password user for many years. Two decades of using a Mac will do that for you. Now I'm back to Linux, I wanted something that I could host myself that provided a reasonable user experience on the desktop and iOS. Bitwarden is pretty decent, but I've not quite mustered the enthusiasm to migrate fully yet, so continue to mainly use 1Password. The UX just isn't quite as good in Bitwarden. With that said, I have only positive things to say about Vaultwarden.

Deployed in a NixOS container using the module which uses SQLite by default, but I've set it up with PostgreSQL using the module for that.

WriteFreely

You're using it now visiting this site. I like it's super-simple writing UI, although I occasionally have thoughts of going back to a static site generator.

Deployed using the NixOS module which also takes care of the MySQL database, all in a NixOS container.

I started writing this post thinking "I'll just knock out a quick one to help get me back into the habit". LOL. Turns out it's easy to forget just how much stuff I'm operating!

The joy of code review

Fri, 21 Jun 2024 17:46:00 +0100

I've often heard code review spoken of as a chore, and have witnessed (and myself been guilty of) reviews that reflect this attitude. Cursory skims of the proposed changes, nitpicking small but obvious faults, rushing to get the review done to move on to matters more pressing.

But this attitude, or culture where it's pervasive, denies the reviewer a great opportunity to learn.

As someone a couple of decades in to my career, I've noticed that whilst I still get excited to learn a new tool, it can be easy to overlook how the tools I'm most comfortable and familiar with continue to evolve.

I've found myself learning new things about the tools I thought I knew best whilst reviewing the code of someone who's earlier in that journey and discovering all the features of those tools for the first time as they are today, and that's brilliant!

Code review is also a great opportunity to get some insight into how other folks think and solve problems. Doing my job well as a reviewer demands that I understand the problem being solved, and is often rewarded by discovering an approach or techniques I may never have considered.

Regardless of experience, we have so much to learn from each other. So make time for and give reverence to your code reviews. And if your organisation doesn't allow for that, that sucks! If you're able, find one that does.

Differently complicated hosting - The Before

Sun, 14 Jan 2024 10:17:00 +0000

After a few months of getting comfortable with NixOS as my desktop operating system, I decided it was time to try it out for servers. But first I wanted to write about the setup I had before.

Disclaimer: This post is likely full of bad ideas - you probably shouldn't setup anything you care about like this. It is my opinion that my most valuable learning is when I'm learning what not to do, and I know there are some gems lurking in here.

How it started

At home, I was running a 8th-gen i5 NUC doing double duty as my home router and server, called homegw. It was running Ubuntu 22.04 with LXD. Router stuff happened on the base OS, with dnsmasq, AdGuard Home and a bunch of ufw rules. It also ran xl2tpd to bring up my Andrews and Arnold LT2P connection so I can have real IPv6 here, which my cable ISP doesn't provide.¹

There was an LXD VM for Home Assistant, and LXD containers for a backup server (restic and Arq over SSH), for Plex, and one running various tools for... ahem downloading Linux ISOs.

There was a Wireguard tunnel from this machine out to Mullvad, and a static route with NAT to 10.64.0.1 on this link. Mullvad operates a SOCKS5 proxy on this address, so I configured all my Linux ISO downloading tools to use this proxy.

Another Wireguard tunnel connected to a dedicated server, hetty, running in Hetzner's Falkenstein data centre. It was nabbed from their server auction a little over a year ago. It was an 8-core 9th-gen i9, 128Gb RAM and 2x1Tb NVMe drives, for the bargain price of 54€ per month. In retrospect, way more computer than I needed, and more money than I should be spending.

This too ran Ubuntu 22.04, with LXD. In this case, a single LXD container called kubeservices running microk8s. I had Kubernetes setup with flux, a GitOps tool that keeps your cluster in sync with a bunch of YAML defined in Git. Hosted here were:

This blog (Writefreely with MySQL)
The CrunchyData Postgres Operator providing PostgreSQL instances for most of the other services mentioned here
My Mastodon instance, with Elasticsearch and Libretranslate
Vaultwarden, a full featured and self hosted Bitwarden server
Synapse, a Matrix chat server
Prometheus, for gather metrics on all the services, including from Home Assistant on the other end of the Wireguard tunnel, so I can get pretty graphs on my home electricity generation/usage, heating, etc.
Grafana for visualising those metrics
Keycloak for single sign-on to most of the above services

Kubernetes PersistentVolumeClaims provided all the stateful storage, using the microk8s host path provisioner, all ultimately ending up in a single directory on the host, an LXD custom volume with daily snapshots and backed up to OneDrive² and to homegw with restic.

I'd chosen to use LVM on top of a LUKS device made from an mdadm initialised RAID1 mirror. ZFS was tempting, but whilst this was easy to setup with Ubuntu Desktop, it was less straightforward with Ubuntu Server. So I took the easy road. Or so I thought.

LXD creates a ThinPool for it's storage when using LVM, which eventually bit me when it's space reserved for metadata filled up. Despite having plenty of free data space, I couldn't figure out how to reallocate more space for metadata (I don't think you can). So I ended up ejecting the second SSD from the RAID1 mirror, and adding it as a new disk to the volume group in order to expand the size of the thin pool in order to recover.

That was my first big "this was a mistake" moment, letting LXD allocate all the remaining volume group space for a single Thin Pool was a bad idea. Ultimately I think LVM was not a good choice for this kind of setup, and I wouldn't make it again.

Oh, and I'd made the same choice of LVM with LXD on homegw. Multiple times I filled the disk and had a bit of a nightmare recovering from it. One does not simply.

Another "this was a mistake" moment was from filling OneDrive with restic backups. It turned out the Postgres Operator by default creates a single initial backup using pgrestbackup, and then captures the write-ahead log forever. Eventually my daily restic snapshots, even with regular pruning, filled the dedicated OneDrive account I'd setup for the job. At that point, restic could simply not recover. Any kind of pruning operation needed some amount of storage space in OneDrive that impossible to provide. You can pay for extra storage, but only on the primary Microsoft 365 account, so I couldn't buy my way out of it. In the end I trashed the entire restic repository and started again.

A simple change to the PostgresCluster YAML switched PGO to giving daily backups, storing up to 3 in the cluster. My disk usage went down to 600Gb to 150Gb after 3 days.

    pgbackrest:
      global:
        repo1-retention-full: "3"
        repo1-retention-full-type: count
      repos:
      - name: repo1
        schedules:
          full: "0 4 * * *"

Another consequence of filling the metadata for the thin pool mentioned earlier was the disk switched to read-only. Upon recovery I had some corrupted Postgres DBs and needed to restore from backup. The Postgres Operator makes that possible with by poking at different parts of the YAML. It's an incredible tool, but not knowing it inside out, I spent a lot of time feeling frustrated that I couldn't just jump on the server and fix things. Instead, everything is orchestrated, and it's feels a bit like operating a light switch with a broom stick.

For my use, I don't need anything like the capabilities PGO offers, and I should KISS.

The big takeaway

Understand your tools. Read the docs. Be curious about what can go wrong. Test those scenarios at your leisure, not in production.

Next time

All of that is gone. I'll be back to describe what replaced it.

¹ Notably, I've found IPv4 performance is often better over this link too, with lower ping times to many sites with AA compared to Virgin Media, despite having to transit VM to get to AA first.

² You can pick up Microsoft 365 Family for under 50GBP per year if you watch out for offers on the "gift card" version of it. This gives you 6x 1Tb OneDrive accounts, which is some of the cheapest cloud storage out there. Encrypting what you put there is a good idea, so tools like restic with rclone are your friends.

Tunnelling VLANs over Wi-Fi with OpenWrt and GRE

Wed, 20 Dec 2023 19:19:00 +0000

My home network is effectively in two segments, with the cable modem, router/server and access point wired together downstairs and another access point acting as a bridge, connected to a wired network in my office upstairs.

I'd been wanting to up my security game and split out dedicated networks (VLANs) and SSIDs for trusted devices, guests and untrusted IoT things for a while. One of the frustrating things about using Wi-Fi to bridge the two wired networks is that this typically precludes VLAN functionality.

I could just do things right and run a cable between the two wired islands, but my willingness to venture into technical escapades vastly exceeds my willingness to drill holes in walls and ceilings. And so, I spent a full day of my 2023 Christmas holiday messing around with OpenWrt.

All my routing is performed by my NUC server, so I'm using both OpenWrt devices purely as access points. Downstairs, I have a Ubiquiti Unifi 6 Long Range which was easy to reflash with vanilla OpenWrt by following the instructions on that link. Upstairs I have a Belkin RT3200, also flashed with vanilla OpenWrt. It turns out both devices are very similar, running the same SoC and Wi-Fi 6 chipsets. The Belkin unit has a 4 x 1GbE port switch + WAN port, where the Unifi has a single 2.5GbE uplink.

It is possible to pass VLANs over Wi-Fi by tunnelling your L2 network over IP, with the caveat of needing a higher than typically MTU to allow for the overhead. I created a mesh connection between the APs, bringing up network interfaces on either side with static IPs. Over that connection GRETAP devices on either end are able to pass Ethernet frames, and bridge to interfaces exposing endpoints for each VLAN.

I'm indebted to oofnikj for his post and more especially his config repo for getting me closer to a working configuration than the official docs.

If you need or want a similar setup, read on.

Tips

tcpdump was my friend on this adventure, revealing a storm of DHCP, ARP and multicast traffic at one point that was escaping the VLANs due to the switch in the Belkin unit effectively munging everything together and confusing the rest of the network until I got the configuration correct.

I started the configuration with both APs on the same wired network, next to each other. There came a point, just after bringing up the tunnel, where I had to make sure to disconnect one of them to avoid loopbacks. STP should take care of this, but didn't seem to for me, though that may have been down to the switch issue above.

Several times along the way I was saved by the dedicated tunnel network, where one AP was accessible, I could typically SSH from it using the internal IP assigned to the other to fix things.

My setup involves the trusted network behaving as the primary, untagged VLAN on each other physical Ethernet ports, with the IoT and Guest VLANs tagged.

Any feedback on what I could have done better is most welcome!

AP 1 Configuration (Unifi 6 LR, downstairs, attached to router)

# /etc/config/network

config interface 'loopback'
    option device 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config globals 'globals'
    option ula_prefix 'fd92:bb11:b3e9::/48'

# I'm not clear on why this is a device rather than an interface
# but it was the default config, and it works, so I don't feel like
# experimenting with it further now!
config device
    option name 'br-lan'
    option type 'bridge'
    list ports 'eth0'
    # VLAN 1 on the interface connected to the GRE tunnel
    list ports '@trunk.1'
    option stp '1'
    option igmp_snooping '1'

# this interface holds the IP of the AP on the trusted network
config interface 'lan'
    option device 'br-lan'
    # I configure my IPs with fixed entries in DHCP
    option proto 'dhcp'
    # don't delegate IPv6, my router will take care of it
    option delegate '0'

# this interface is one end of the underlying IP network for the tunnel
# between the two APs
config interface 'wtun'
    option proto 'static'
    # any address outside our normal range is fine here
    option ipaddr '172.16.0.1'
    option netmask '255.255.255.0'
    option delegate '0'
    # it's here we need a larger-than-normal MTU to be able to pass
    # our 1500 MTU + overhead Ethernet frames
    option mtu '2048'

# the trunk bridge acts as the Ethernet endpoint of the GRE tunnel
# and we can create VLAN tagged versions of it to bridge to our
# wired and wireless interfaces
config interface 'trunk'
    option type 'bridge'
    option proto 'none'
    option bridge_empty '1'
    option delegate '0'
    option stp '1'
    option defaultroute '0'

# the GRE tunnel itself
config interface 'gre'
    option proto 'gretap'
    option ipaddr '172.16.0.1'
    option peeraddr '172.16.0.2'
    # transit over the dedicated network we created
    option tunlink 'wtun'
    # attach to our trunk bridge
    option network 'trunk'
    option df '0'
    option mtu '1500'
    option delegate '0'

# a bridge connecting the IoT (101) VLAN between the GRE
# tunnel and whatever we want to attach to it
config device
    option name 'br-iot'
    option type 'bridge'
    list ports 'br-lan.101'
    list ports '@trunk.101'
    option stp '1'
    option igmp_snooping '1'

# bring up an interface on the IoT VLAN for this AP
config interface 'iot'
    option proto 'dhcp'
    option device 'br-iot'
    option defaultroute '0'
    option peerdns '0'
    option delegate '0'

# a bridge connecting the Guest (102) VLAN between the
# GRE tunnel and whatever we want to attach to it
config device
    option name 'br-guest'
    option type 'bridge'
    list ports 'br-lan.102'
    list ports '@trunk.102'
    option stp '1'
    option igmp_snooping '1'

# bring up an interface on the Guest VLAN for this AP
config interface 'guest'
    option proto 'dhcp'
    option device 'br-guest'
    option defaultroute '0'
    option peerdns '0'
    option delegate '0'

AP2 Configuration (RT3200, upstairs)

# /etc/config/network

config interface 'loopback'
    option device 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config globals 'globals'
    option ula_prefix 'fddd:12af:6646::/48'

config interface 'wtun'
    option proto 'static'
    option ipaddr '172.16.0.2'
    option netmask '255.255.255.0'
    option delegate '0'
    option mtu '2048'

config interface 'trunk'
    option type 'bridge'
    option proto 'none'
    option bridge_empty '1'
    option delegate '0'
    option stp '1'
    option defaultroute '0'

config interface 'gre'
    option proto 'gretap'
    option ipaddr '172.16.0.2'
    option peeraddr '172.16.0.1'
    option tunlink 'wtun'
    option network 'trunk'
    option df '0'
    option mtu '1500'

# my lan bridge is a little different on this AP due to the switch
# ports available on this device, which allow me to dedicate specific
# ports to VLANs as needed. Here every port is configured with the
# trusted network (VLAN 1) as it's untagged, primary VLAN and the
# other networks are available with tags on each port.
config device
    option name 'br-lan'
    option type 'bridge'
    option stp '1'
    option igmp_snooping '1'
    option bridge_empty '1'
    # attach VLAN 1 of the tunnelled Ethernet to this bridge, doesn't
    # seem quite right, but breaks without it
    list ports 'br-trunk.1'
    list ports 'lan1'
    list ports 'lan2'
    list ports 'lan3'
    list ports 'lan4'
    list ports 'wan'

# the untagged primary trusted VLAN on all ports
config bridge-vlan
    option device 'br-lan'
    option vlan '1'
    list ports 'br-trunk.1:u*'
    list ports 'lan1:u*'
    list ports 'lan2:u*'
    list ports 'lan3:u*'
    list ports 'lan4:u*'
    list ports 'wan:u*'

# tag the IoT VLAN on all ports
config bridge-vlan
    option device 'br-lan'
    option vlan '101'
    list ports 'lan1:t'
    list ports 'lan2:t'
    list ports 'lan3:t'
    list ports 'lan4:t'
    list ports 'wan:t'
    option local '0'

# tag the Guest VLAN on all ports
config bridge-vlan
    option device 'br-lan'
    option vlan '102'
    list ports 'lan1:t'
    list ports 'lan2:t'
    list ports 'lan3:t'
    list ports 'lan4:t'
    list ports 'wan:t'

# bring up an interface for the AP on the trusted VLAN
config interface 'lan'
    option device 'br-lan.1'
    option proto 'dhcp'
    option delegate '0'

config device
    option name 'br-iot'
    option type 'bridge'
    option stp '1'
    option igmp_snooping '1'
    list ports 'br-lan.101'
    list ports 'br-trunk.101'

config interface 'iot'
    option proto 'dhcp'
    option ipv6 '0'
    option device 'br-iot'
    option defaultroute '0'
    option peerdns '0'
    option delegate '0'

config device
    option name 'br-guest'
    option type 'bridge'
    option stp '1'
    option igmp_snooping '1'
    list ports 'br-lan.102'
    list ports 'br-trunk.102'

config interface 'guest'
    option proto 'dhcp'
    option ipv6 '0'
    option device 'br-guest'
    option defaultroute '0'
    option peerdns '0'
    option delegate '0'

Common configuration

The wireless and firewall configurations are identical on both APs. If your APs have different hardware, you'll likely need to adjust the radio sections for each.

# /etc/config/wireless
config wifi-device 'radio0'
    option type 'mac80211'
    option path 'platform/18000000.wmac'
    option channel '6'
    option band '2g'
    option htmode 'HT20'
    option cell_density '0'

config wifi-device 'radio1'
    option type 'mac80211'
    option phy 'wl1'
    option country 'US'
    option cell_density '0'
    option htmode 'HE80'
    option band '5g'
    option channel '149'

# the mesh network underlying our GRE tunnel
# I'm only using the 5GHz radio for this, because the signal is strong
# and the access points are static
config wifi-iface 'wifinet7'
    option device 'radio1'
    option mode 'mesh'
    option encryption 'sae'
    option mesh_id 'upstream'
    option mesh_fwding '1'
    option mesh_rssi_threshold '0'
    option key 'SomeRandomString'
    option network 'wtun'
    option ifname 'wtun'

# the 2GHz version of my trusted network SSID
config wifi-iface 'wifinet2'
    option device 'radio0'
    option mode 'ap'
    option ssid 'Trusted'
    option encryption 'sae'
    option key 'SuperStrongPassphrase'
    option network 'lan'

# the 5GHz version of my trusted network SSID
config wifi-iface 'wifinet5'
    option device 'radio1'
    option mode 'ap'
    option ssid 'Trusted'
    option encryption 'sae-mixed'
    option key 'SuperStrongPassphrase'
    option network 'lan'

# the 2GHz version of my Guest network SSID
config wifi-iface 'wifinet9'
    option device 'radio0'
    option mode 'ap'
    option ssid 'Guest'
    option encryption 'sae-mixed'
    # setting isolate to prevent wireless devices on this
    # SSID from talking to each other (same on IoT)
    option isolate '1'
    option key 'welcomeguest'
    option network 'guest'

# the 5GHz version of my Guest network SSID
config wifi-iface 'wifinet8'
    option device 'radio1'
    option mode 'ap'
    option ssid 'Guest'
    option encryption 'sae-mixed'
    option key 'letmeinplease'
    option network 'welcomeguest'
    option isolate '1'

# the 2GHz version of my IoT network SSID
config wifi-iface 'wifinet10'
    option device 'radio0'
    option mode 'ap'
    option ssid 'IoT'
    option encryption 'psk2'
    option isolate '1'
    option key 'untrusted'
    option network 'iot'

# the 5GHz version of our my network SSID
config wifi-iface 'wifinet11'
    option device 'radio1'
    option mode 'ap'
    option ssid 'IoT'
    option encryption 'psk2'
    option isolate '1'
    option key 'untrusted'
    option network 'iot'

# /etc/config/firewall
config defaults
    option input 'REJECT'
    option output 'ACCEPT'
    option forward 'REJECT'
    option synflood_protect '1'

# I have a firewall zone for each network
config zone
    option name 'lan'
    # only the trusted network and underlying tunnel network
    # have "input 'ACCEPT'" which will allow access to the
    # AP admin interfaces
    option input 'ACCEPT'
    option output 'ACCEPT'
    option forward 'ACCEPT'
    list network 'lan'

config zone
    option name 'guest'
    option input 'REJECT'
    option output 'ACCEPT'
    option forward 'ACCEPT'
    list network 'guest'

config zone
    option name 'iot'
    option input 'REJECT'
    option output 'ACCEPT'
    option forward 'ACCEPT'
    list network 'iot'

config zone
    option name 'tunnel'
    option input 'ACCEPT'
    option output 'ACCEPT'
    option forward 'ACCEPT'
    list network 'wtun'

Moving from macOS to Linux: The unknowns

Sun, 12 Nov 2023 18:10:00 +0000

Updated with Preview.app features, prompted by Neil's helpful response Desktop Linux: the software I'm currently using

With my new laptop coming in a few days, I'm finally thinking through the implications of moving away from macOS whilst still having an iPhone. These are some questions I need to answer:

How will I listen to music?
Currently streaming with Apple Music, blended by the Music app with my local music collection
How will I manage photos?
Currently in Photos, synced with iCloud, mostly originating from my phone, with a collection of old stuff originally synced from the Mac
How will I do quick annotations on screenshots, markup PDFs, resize/crop/export images without Preview.app?
What Passkeys do I need to migrate out of iCloud (or the Mac specifically)?
What other credentials are in my Mac/iCloud keychain that aren't in 1Password for some reason?
Safari has been my primary browser and macOS, and will likely continue to be on iOS
I tend to throw stuff at Reading List to pick up later, which I won't have access to on my new computer, so what to do instead?
Should I keep an old Mac around, running?
Logged into iCloud, it could provide another authentication factor for iDevices, which assume you've got some other Apple device nearby to do authentication
There are bridges for iMessage that could let me continue to read/reply to iMessage and SMS from my phone
I have an old MacBook Pro with a non-functioning screen that's too expensive to fix that could do the job
I subscribe to Microsoft 365, mainly for cheap cloud storage (effectively 6Tb for around ~£45 per year when bought on semi-frequent offer), but do use Excel for a few tasks. Should I...
Migrate away to something else?
Use the web version?
Try Wine/Crossover/Windows VM?
I almost entirely interact with Mastodon through Ivory, on my phone and with the macOS app. I really like that it stays in sync across both with my read position. Am I just going to use the iOS app now, or is there some other solution?

I guess I'll find my answers in the coming weeks.

Non-specific nerd thoughts

Thu, 24 Aug 2023 10:17:00 +0100

That whole writing more thing went well, didn't it? ?

Let's try a little brain dump of the nerdy things I've assigned myself to do:

Replace macOS with NixOS as my primary computing experience.
Being a cloud infrastructure engineer, I love me some declarative configuration.
Apple annoyed me with the offer of a £700 fix for the most expensive computer I ever purchased that was just out of warranty and had display issues.
- The £700 was to replace a display that's screwed because the connecting cable has been damaged by their hinge design. There is a company in London that offer a £300 repair. Still quite ouchy.
- It's an Intel Mac, how long are they even likely to support it anyway?
- It was already passed on to Sean, replaced for me by a 14" M1 Pro.
Framework seems kinda great, so I'm eagerly awaiting my Ryzen-based Framework 13. At which point Sean can have the 14" and become untethered from a desk again.
I'll miss the screen of the MacBook, but I think even more I'll miss the speakers.
Capture my (non-phone) computing world into a Git repo.
Nix allows me to describe my systems and my user environments in code!
On that whole phone thing, wouldn't it be nice if there were some genuinely open-source phone ecosystem without Google that could actually run the apps I've come to depend on (banking, etc)?
I've sadly accepted my iPhone 12 mini will eventually be replaced with another iPhone.
Adopt a more keyboard-centric computing life, with a tiling window manager.
Endless configuration tweaking awaits.
Have a go at (neo)vim being my primary editor again, or otherwise invest properly in VSCod(e|ium).
I might as well go all in, eh?
Failing at this and continuing to use GoLand and PyCharm wouldn't be terrible.
Migrate my Hetzner dedicated server running this site, and my Masto instance to something else (self-host, Hetzner cloud).
It's way over-specced for my current use (but 50€ for 128GB RAM, 2x1Tb SSD and 8-core i9-9900K is amazing value).
I use microk8s on it as a single node instance, which is pretty inflexible, storage being a large part of sense of unease with that.
I made the mistake of choosing LVM and let LXD create a thin-pool consuming 100% of the remaining storage.
It ran out of metadata space, and you can't grow it into the unused data space, so I had to de-RAID1 the SSDs to make things work so that's a bit of a mess now.
Said mess returns errors when trying to do operations like take snapshots for backups, or even delete old snapshots, a reboot might fix it, or it might leave me with a complex failed boot to resolve.
Hetzner Cloud seems to be pretty darn cheap, and I can create a proper Kubernetes environment with separate control plane and real storage volumes and come in at similar or lower price than the dedicated server, albeit with less overall resources and with non-dedicated CPU.
I experimented briefly with Talos Linux on it yesterday, and it went pretty smoothly.
I can Terraform it (or OpenTF eventually, right?)
Or maybe I can just host it all at home on a single box?
Migrate my home server from Ubuntu with LXD to NixOS with something.
It's an 8th gen NUC in a fanless Asaka case.
It's my router and adds IPv6 to my home Internet with AAISP's L2TP service, because Virgin Media can't even, but I'm definitely taking their 1Gb/100Mb service over my next best option of 40Mb/10Mb DSL.
It's running an LXD-launched VM for Home Assistant, and LXD containers for Plex, some *aars, etc.
It's also built on the same LVM thin pool scheme as the Hetzner box, so is ultimately doomed.
I'm clearly going to NixOS it, and microvms looks super interesting.

I can't help think of Amelie's father and his toolbox.

Writing more

Sun, 19 Feb 2023 08:45:00 +0000

I want to improve my writing, so I need to do more of it.

Improving means that I can communicate effectively, efficiently and engagingly. It means you can understand me, that I'm not wasting your time, and that I can hold your attention long enough to impart what I wanted you to know.

Part of my job as a software engineer is to express ideas in a way that is understandable by machines. More importantly, those ideas can be understood by other engineers who work with me now and in the future, whether I'm present or not. That is what good code means to me.

When I write for a machine I'm able to get feedback near instantly. Frequently that feedback is unambiguous. I was either successful in communicating my intent, or I was not. Code reviews help ensure I'm successfully communicating with other engineers too.

Much of my work is useful to people who shouldn't need to understand the details of it’s construction to find it helpful. I should be able to describe it in a way that respects their time and attention. Documentation matters.

Career progression requires I am able to communicate ideas in a way that places them in a wider context to demonstrate their value. I distill the ideas of multiple people and in doing so am representing them as well as myself.

I need to write to justify advancements, to mediate conflict, to convince others to invest their resources, to give candid feedback kindly, to explain failure, and to celebrate success.

I need to be able to do all of that in a timely manner that leaves room for all the other things to be done. Improving also means writing more quickly when that's necessary.

Success will be hard to measure here. Improvement needs feedback as well as practice. That's why I'm writing this publicly.

The most uncomfortable part is convincing myself that this is good enough, when I know that it could be better. If the only outcome is that getting easier, then this will be worthwhile.