Runbook — "Netbird is down"¶

Netbird is the mesh that carries SSH, service access, and all agent traffic. If it goes down you lose access to everything unless you can fall back to the WireGuard tunnels. Read the topology in docs/networking.md first — which path you have depends on which node you're trying to reach.

First: figure out what "down" means¶

The mesh has three possible failure modes; recovery differs for each.

failure	symptom	still reachable backend
One peer dropped (e.g. vps-apps)	that host unreachable on `100.64.x.x`, others fine	the host's own tunnel only if it has one
Control plane (vps-net) down	everyone starts dropping as registration/signal fails	`wg-direct` (NFS), `wgr-vpsnet`
The whole mesh / vps-net + relay gone	nothing reachable on `100.64.x.x`	pi-home's LAN, static tunnels

Check from the laptop:

netbird status          # peer health; is the control-plane peer listed?
ping 100.64.3.46        # pi-home   (also runs DNS)
ping 100.64.0.100       # vps-apps

Recovery by path¶

A. Control plane (vps-net) is down → the critical one¶

Everything relies on the Netbird control plane on vps-net (vpn.jmrdez.com:443). If it's down, the best fallback is the static tunnels that don't need the mesh:

wg-direct (vps-apps ↔ pi-home, 10.99.2.1/24) — carries NFS. Still works; gives you a shell path only if you add SSH, but the NFS/backup leg survives.
wgr-vpsnet (pi-home ↔ vps-net, 10.99.1.10/32) — the only way into vps-net:
```
ssh -J DietPi jmrdez@10.99.1.1
```
Then bring the netbird compose back up:
```
docker compose -f /opt/homelab/vps-net/netbird/compose.yml up -d
```

Get to vps-net even without the mesh — the laptop reaches vps-net only through ssh -J DietPi (pi-home must be up and its side of wgr-vpsnet up). pi-home is on your LAN, so if pi-home itself is reachable over the LAN you have a path.

B. pi-home is down¶

pi-home runs the mesh DNS and the cloudflared tunnel, so its loss is wide. It's on your LAN first (home-assistant/local IP), then mesh 100.64.3.46. It also holds the restic backup repo ON ITS HDD (/mnt/pi-hdd) — a pi-home failure risks the only backup. Priorities: 1. Reboot/recover pi-home (RPi 3B, 1 GB — check it isn't swap-thrashing; see mesh-dns-flapping.md for the low-RAM footguns). 2. Confirm /mnt/pi-hdd mounts and the restic repo is intact.

C. Full mesh outage with no control plane¶

If vps-net is unreachable and Netbird can't re-register: - From the laptop you still have the LAN to pi-home. - From pi-home you have wg-direct → vps-apps (NFS only) and the wgr-vpsnet tunnel → vps-net. - Don't curl anything over the mesh expecting it to resolve — DNS (pi-hole on pi-home) is also mesh-dependent for some names.

Do not¶

Do not restart Netbird blindly on every node — that's what the old check-netbird.sh watchdog used to do and it caused a 288-restart/day storm (see mesh-dns-flapping.md, Fault 1). Restart one peer, watch it re-register.
Do not edit firewall rules on vps-net out of panic — a previous edit here caused a full lockout needing console recovery.
Do not touch /etc/exports or NFS options as a "fix" — the NFS leg is independent by design.

After it's back¶

netbird status on all peers; confirm the mesh IPs match docs/networking.md.
Confirm DNS resolves (pi-hole :53 on 100.64.3.46).
Run one backrest/restic backup to confirm the repo path survived.

Prevention¶

Keep the static tunnels (wg-direct, wgr-vpsnet) healthy — they are the no-mesh escape hatch. They get persistent keepalive and survive Netbird outages by design.
Periodically practice reaching vps-net via ssh -J DietPi without the mesh, and vps-apps via the NFS tunnel, so the fallback is muscle memory.