Runbook — "DNS is slow everywhere / DietPi keeps dying"¶

Four separate things were found and fixed. Keep this as the checklist for the next time DNS goes sideways.

Symptom¶

DietPi appears to die or spike CPU every few minutes, and DNS resolution across the whole homelab stalls with it.

Summary¶

	Fault	Real cause of the DNS symptom?
1	NetBird watchdog restarts the mesh every 5 min	No — measured, see the correction below. Fix anyway; it is pointless churn.
2	Stale Tailscale `/etc/resolv.conf` on vps-apps	No — misattributed; see the correction under Fault 2. Stale and worth removing regardless.
3	UFW blocks :853 (DoT) and NetBird's :36880 on the LAN	Yes — most likely what the LAN actually experiences
4	The Pi has 16 MB free RAM and 581 MB in SD-card swap	Yes, as a standing condition
5	LLMNR fan-out across 56 links on vps-apps	Yes — this is what actually cost ~8 s, on single-label names only

DietPi has no watchdog of its own — its timers are only phpsessionclean, fake-hwclock-save, tmpfiles-clean, dpkg-db-backup and fstrim, and its netbird had been up 16 h. The restart storm is one-sided, on vps-apps.

Note on "continuously dying": uptime on DietPi showed 16 h 22 m, so it had not rebooted. Do not try to confirm that from journald — /var/log is tmpfs, so journalctl --list-boots reports a single boot regardless of history, and its first entry was ~3 h after the real boot time.

Fault 1 — the NetBird watchdog restarted the mesh every 5 minutes¶

Root cause. /usr/local/sbin/check-netbird.sh health-checked 100.64.1.1 and ran systemctl restart netbird when the ping failed. vps-net is deliberately not a NetBird peer, so 100.64.1.1 is unreachable by design and the check could never pass.

Evidence.

ping 100.64.1.1  ->  sendmsg: Required key not available   (no WG peer exists)
timer fires      ->  15:47:51, 15:52:51, 15:57:51, 16:02:51  (every 5 min)
wt0 ifindex      ->  6013 -> 6014 -> 6015 -> 6016 in 15 minutes
16:03:00 wt0=GONE  16:03:02 netbird pid 1130941 -> 1134765  16:03:03 wt0=6016

~288 restarts/day, each forcing every peer through a fresh signal/ICE/WireGuard renegotiation.

Correction: this does NOT spike DietPi¶

The first version of this runbook claimed the restart storm was what spiked DietPi's CPU and stalled DNS. That was wrong, and the measurement is worth keeping so nobody re-derives it.

vmstat 5 30 was run on DietPi across a window (06:09:16–06:11:42) that contained a watchdog fire at 06:11:13. There was no spike:


CPU idle, whole window	88–97 % (never dipped at the fire)
swap-in / swap-out	1–76 / ~0 pages per second
load average	0.32 (the docs' 4.80 was measured with an agent session running)
DietPi's own netbird uptime	16 h — it never restarted
DNS latency, 25 queries from vps-apps	90–122 ms, no timeouts

DietPi absorbs a peer reconnect without noticing. The storm is still worth killing — it is ~288 pointless interface teardowns a day — but it is not the cause of the DNS symptom. Fix it because it is wrong, not because it will make DNS faster.

Fix. Replaced by the version in vps-apps/netbird-watchdog/ — checks the local client only, requires 3 consecutive failures, enforces a 30 min cooldown. Install steps in its README. Disabling the timer outright is also fine; netbird's own unit has Restart=always.

How to spot a recurrence. wt0's ifindex should be stable for as long as the host is up. If it is climbing, something is cycling the interface:

cat /sys/class/net/wt0/ifindex        # note it, wait 10 min, check again
systemctl list-timers --all | grep -i netbird
journalctl -t netbird-watchdog --since -2h --no-pager

Fault 2 — Tailscale's leftover `/etc/resolv.conf` on vps-apps¶

Root cause. Tailscale is installed but inactive/disabled, yet resolv.conf was still the file it wrote (frozen), listing two nameservers that no longer exist:

# resolv.conf(5) file generated by tailscale
nameserver 1.1.1.1
nameserver 45.90.28.105
nameserver 100.100.100.100      # dead - Tailscale MagicDNS, no tailscale0 iface
nameserver fd7a:115c:a1e0::53   # dead - Tailscale IPv6

Every name not answered from public DNS walked into the dead servers:

Lookup	via `resolv.conf`	via a live resolver
`pi-home`	8027 ms	44 ms
`vps-net`	7745 ms	47 ms
`nas`	7737 ms	24 ms

systemd-resolved was running the whole time, maintaining a correct stub at /run/systemd/resolve/stub-resolv.conf that nothing used.

Correction: the dead nameservers were NOT the cause¶

The ~8 s figure above is real but misattributed. Removing both dead entries changed nothing — getent hosts pi-home still took 7.887 s with a clean /etc/resolv.conf. It should have been obvious: an NXDOMAIN from the first nameserver ends the lookup, so entries three and four were never reached.

Two things were missed:

/etc/nsswitch.conf reads hosts: files myhostname resolve [!UNAVAIL=return] dns, so getent goes through nss-resolve to systemd-resolved and does not consult /etc/resolv.conf at all.
Every slow name was single-label. Measured after the cleanup:

Name	Type	Time
`pi-home`, `nas`, `printer`, `nosuchhost-xyz`	single-label	7.5–8.0 s
`nosuchhost-xyz.example.com`, `pi-home.nb.sh`	multi-label	9–25 ms

systemd-resolved routes single-label names to LLMNR, and 56 links on this host have +LLMNR — every docker bridge and veth, plus wt0 and wg-direct. Each is multicast and waited on until timeout.

Real fix:

sudo mkdir -p /etc/systemd/resolved.conf.d
printf '[Resolve]\nLLMNR=no\n' | sudo tee /etc/systemd/resolved.conf.d/10-no-llmnr.conf
sudo systemctl restart systemd-resolved

Safe on a VPS with no link-local name resolution needs; Windows/Samba name lookups happen on the home LAN against DietPi, not here.

Removing the Tailscale entries was still correct hygiene — they were stale, and because resolv.conf mode: foreign resolved had adopted them as its global DNS servers — but it was not the latency fix.

Lesson: a plausible cause plus a real measurement is not causation. The check that would have caught this in one step is comparing a single-label name against a multi-label one, rather than assuming the resolver list was at fault.

Fix (cleanup only — for the latency fix see the LLMNR drop-in above). Delete the two dead entries in place:

sudo sed -i -e '/^nameserver 100\.100\.100\.100$/d' -e '/^nameserver fd7a:115c:a1e0::53$/d' /etc/resolv.conf

systemd-resolved re-reads the file on its own, so this clears the adopted global list too — verify with resolvectl status.

Do NOT symlink /etc/resolv.conf to the stub on this host, which an earlier revision of this runbook recommended. The stub contains only nameserver 127.0.0.53, which bridge-network containers cannot reach, so Docker silently substitutes its own fallback (8.8.8.8) for them. With ~30 containers currently inheriting 1.1.1.1 + NextDNS (45.90.28.105), that would quietly move every one of them off NextDNS filtering — which is exactly the drift mesh-improvements item 13 already complains about. Doing it properly needs DNS= in resolved.conf and a dns: entry in /etc/docker/daemon.json, and the latter restarts every container. Not worth it.

If Tailscale is not coming back, remove it so it cannot reclaim the file:

sudo apt-get -y purge tailscale        # NOTE: not covered by the claude sudoers whitelist

Fault 3 — UFW is blocking two DNS paths on the LAN¶

This is the one most likely to be the actual "DNS is slow" experience, and it was found by counting [UFW BLOCK] lines in DietPi's kernel log:

     30 SRC=192.168.0.57  DPT=853     <- DNS-over-TLS, blocked
      6 SRC=192.168.0.55  DPT=853     <- DNS-over-TLS, blocked
     16 SRC=192.168.0.28  DPT=36880   <- NetBird direct LAN path, blocked
      1 SRC=192.168.0.29  DPT=36880   <- same

Port 853 (DNS-over-TLS) — opening the firewall is NOT the fix. Two LAN clients are configured to reach Pi-hole over DoT and UFW drops every attempt; the kernel log shows them retrying at 1 s intervals then backing off, so those devices experience DNS as timing out. The first version of this runbook said to ufw allow port 853. That was wrong. Checking what DietPi actually listens on:

udp  0.0.0.0:53      <- pihole-FTL
tcp  0.0.0.0:53      <- pihole-FTL
tcp  127.0.0.1:5335  <- unbound, loopback only
udp  0.0.0.0:36880   <- netbird

Nothing listens on 853. Opening the port only converts a silent drop into a connection refused — marginally better, because the client fails fast and can fall back instead of hanging on retries, but DoT still will not work. The real fix is one of:

reconfigure 192.168.0.57 and 192.168.0.55 onto plain DNS on :53, or
actually terminate DoT on the Pi (stunnel, nginx stream, or Unbound built with TLS) and then open 853.

docs/hosts/dietpi.md lists the UFW allowances as 53 from LAN+CGNAT with no mention of 853, which fits: this is client config that outran the server, not a firewall regression.

Port 36880 (DietPi's NetBird WireGuard port) — here the firewall rule IS the fix. Unlike 853, this port has a real listener (udp 0.0.0.0:36880). LAN peers trying to establish a direct path to DietPi's mesh endpoint are being dropped, which forces those tunnels onto srflx/relay — DNS queries from a device sitting on the same LAN hairpin out to the internet and back. Allowing 36880/udp from 192.168.0.0/24 lets ICE pick the host candidate:

sudo ufw allow from 192.168.0.0/24 to any port 36880 proto udp

Firewall changes are on the "ask Javier first" list, and the read-only claude account cannot make them by design.

Verifying afterwards. The read-only account cannot run ufw status, so the only signal available to it is the absence of new [UFW BLOCK] lines — which is weak, because a client that has backed off looks identical to a rule that worked. Confirm from the node itself:

sudo ufw status numbered | grep -E '853|36880'

Fault 4 — the Pi has no memory headroom¶

Not a fault with a fix so much as the standing condition that makes everything else feel like a DNS problem:


RAM	606 MB used of 917 MB, 16 MB free
swap	581 MB in use — a file on the SD card
PSI	unavailable — this kernel has no `/proc/pressure`
largest single process	`ccd-cli` (Claude Code) at 68 MB / 7.2 %, running 15 h

The biggest consumer is an agent session, ahead of dockerd (40 MB), netbird (40 MB) and pihole-FTL (39 MB). Any memory demand has to come from swap on an SD card, which is the documented 0.018 s → 12.8 s latency signature. Killing a stale agent session is the cheapest headroom on the box.

Also worth reconciling: docs/hosts/dietpi.md records NFS as "fully removed packages purged", but 16 nfsd kernel threads plus rpc.mountd, rpcbind, rpc.statd, lockd and blkmapd are running. Something brought NFS back and the doc is stale.

Still open after this fix¶

netbird status on vps-apps reports Nameservers: 0/0 Available, so vps-apps does not use Pi-hole at all and resolves via 1.1.1.1 + NextDNS. Consolidating is item 13 of the mesh-improvements backlog.
DietPi is genuinely oversubscribed independently of this bug (load ~4.8, 793 MiB of file-backed swap on the SD card). Removing the restart storm takes the recurring spike away but does not create headroom — the levers are moving services to vps-apps and using zram instead of a swapfile.