Skip to content

Runbook — node rebuild from scratch

Rebuilding a node is the truest test of "the repo + backups is the source of truth". Each node has a different runtime (Docker vs native systemd), so the procedure differs per node. vps-net is the highest-stakes — a mistake there locks the whole fleet out — and is called out separately.

Shared principle

Config comes from this repo (and the untracked /opt/docker trees). Data comes from the restic backups (see the restic-restore runbook). Never hand recreate a config on the fly; always re-declare it in the repo first.

vps-apps (Debian trixie, 8 GB, Docker Compose)

  1. Provision the host (Hetzer/OVH console): Debian 13 trixie, openssh, a non-root sudo user (jmrdez — is in the docker group). Install Docker Engine + Compose plugin.
  2. Recreate the data disk layout (there is none — single 75 GB /). If you resized the disk, extend /dev/sda1 and resize2fs.
  3. Clone the repo: 
    git clone git@github.com:ZeroBoust/homelab.git /opt/homelab
    Only vps-apps/ is used here.
  4. Mount the data root: mkdir -p /opt/homelab-data and either restore per app from restic (see that runbook) or re-create empty trees with the right ownership so the apps re-init.
  5. Recreate the untracked /opt/docker stacks — these are not in the repo but were snapshotted by backrest (/backup/opt-docker). Restore them: 
    restic -r /repos/pihdd/vps-apps/config restore --target /opt \
  --include '/opt/docker/**' latest
    Then docker compose -f /opt/docker/<app>/compose.yml up -d for each stack.
  6. Recreate the tracked stacks: 
    for app in romm dawarich ...; do
  docker compose -f /opt/homelab/vps-apps/$app/compose.yml up -d
done
  7. Reconnect the mesh: Netbird netbird up, join, register the peer. The mesh IP 100.64.0.100 will be re-assigned.
  8. Recreate the wg-direct tunnel (NFS client side, 10.99.2.1/24), peer to DietPi, so the backrest NFS repo mounts. Keep the nfs_backup_pihdd volume.
  9. Restore the restic repo mount into backrest and run one backup to confirm the loop is closed.

pi-home (DietPi, RPi 3B, 1 GB — native systemd, no Docker)

  1. Flash a new SD card with DietPi, boot, run setup (set hostname dietpi).
  2. Mount the HDD at /mnt/pi-hdd (687 GB — holds the restic repo under /mnt/pi-hdd/backups/vps-apps). The SD card is /.
  3. Recreate the native units. These are declared in docs/nodes.md (pi-home section) but the actual unit files live on the box; re-create them from /etc/systemd/system/. Restore the units / /etc drop-ins from the config backup branch or re-declare them.
  4. Recreate the data root /mnt/dietpi_userdata/<app> for each app and restore from the last restic snapshot of the pi (if one exists) or the full-system backup. Critical apps: Vaultwarden, Radicale, Pocket-ID, ntfy, syncthing, beszel.
  5. Recreate the network stack — this is the fiddly part:
  6. Netbird mesh peer 100.64.3.46 (also runs DNS).
  7. wg-direct server side 10.99.2.3/24 (NFS to vps-apps).
  8. wgr-vpsnet 10.99.1.10/32 (the only tunnel to vps-net).
  9. Recreate the shares: /etc/exports (restic repo + media), Samba, Pi-hole + unbound DNS, cloudflared token tunnel.
  10. pihole keeps LAN :53; everything else binds the mesh IP.

vps-net (control plane — highest blast radius)

Get explicit approval before touching anything here. A misconfiguration takes down the Netbird control plane and locks the whole fleet out. Do this from the VPS console (not over the tunnel) if possible.

  1. Provision: Debian 13, 1 GB, non-root jmrdez with the scoped sudo (systemctl/docker/rm only), Docker Engine.
  2. Clone repo and use only vps-net/: 
    git clone ... /opt/homelab
docker compose -f /opt/homelab/vps-net/netbird/compose.yml up -d
  3. WireGuard wgrescue: recreate the 10.99.1.1/24 interface with the peer that allows DietPi's 10.99.1.10. DNS/public point at vpn.jmrdez.com → 88.218.77.101.
  4. Verify the recovery path LAST: only connect over the tunnel (ssh -J DietPi jmrdez@10.99.1.1) after everything else is confirmed.
  5. Do not add any unrelated service. Keep the box minimal.

Verify the rebuild (any node)

  • df -h && free -m — sane usage.
  • All expected containers/units running: docker ps / systemctl --state=running.
  • Mesh reachable: netbird status.
  • A data app you restored actually works (e.g. log into Vaultwarden, open a RomM library item, play a synced file).
  • The restic repo mount works and a fresh backup succeeds.

The ugly truth

pi-home's own full-system backup path is gone; only the restic branches remain. "Rebuild from scratch" after a real fire is only as good as the most recent restic snapshot (pi-hdd local, gdrive off-site). The gdrive leg is the difference between "recoverable" and "lost" — keep it working.