Runbook — restore from backup (restic / Backrest)

Both hosts back up to two locations (pi-home HDD = physical, gdrive = off-site). Restic gives versioned, encrypted snapshots. This covers the common restore paths; for app-specific gotchas see the per-app runbooks.

Repo matrix

host backed up	branch	what's in it	pi-hdd (physical)	gdrive (off-site)
vps-apps	config	/opt/docker	/repos/pihdd/vps-apps/config (in Backrest)	gdrive:backups/vps-apps/config
vps-apps	data	/opt/homelab-data + romm-assets	/repos/pihdd/vps-apps/data (in Backrest)	gdrive:backups/vps-apps/data
pi-home	config	/etc	/mnt/pi-hdd/backups/pi-home/config	gdrive:backups/pi-home/config
pi-home	data	/mnt/dietpi_userdata	/mnt/pi-hdd/backups/pi-home/data	gdrive:backups/pi-home/data

• Engine: vps-apps flows run in Backrest (docker compose -f /opt/docker/backrest/compose.yml, UI at 100.64.0.100:9898). pi-home flows run as native timers (restic-backup@{data,config}.service, script /usr/local/bin/restic-pihome-backup).
• Passwords: one per branch, shared across both locations: vps-apps' in /opt/docker/backrest/repo-password-{config,data} (mounted in the container at /run/...), pi-home's in /etc/restic-backups/{config,data}.pass (root 600). Lost password = lost backups. These are also in the fleet's encrypted secrets store — do not commit them.
• Schedules: vps-apps data 03:30 pihdd / 04:30 gdrive, config 04:00/05:00 (Madrid). pi-home data 02:00, config 02:30 Madrid (node clock is UTC). Retention: config d14/w8/m12, data d7/w4/m6.

Verify a backup exists before trusting the rest of this

# vps-apps (Backrest container has the pihdd mount + RCLONE_CONFIG set):
docker exec backrest sh -c \
  'restic -r /repos/pihdd/vps-apps/data --password-file /run/repo-password-data snapshots --latest 5'

# pi-home:
sudo restic -r /mnt/pi-hdd/backups/pi-home/data \
  --password-file /etc/restic-backups/data.pass snapshots --latest 5

# gdrive leg (from the node that owns the flow):
sudo RCLONE_CONFIG=/home/dietpi/.config/rclone/rclone.conf \
  restic -r rclone:gdrive:backups/pi-home/data \
  --password-file /etc/restic-backups/data.pass snapshots --latest 5

If there is no recent snapshot, stop — re-run a backup first. Check journalctl -u restic-backup@data (pi-home) or the Backrest UI ops log (vps-apps) for why.

Restore a single file (fast path)

# Find which snapshot holds the path, then pull it to a scratch dir
restic -r <repo> --password-file <pass> find --latest '<path>'
restic -r <repo> --password-file <pass> restore --target /var/tmp/restore \
  --include '<path>' latest

Restore a whole app / branch (vps-apps example)

1. Stop the app so it cannot write over the restore: docker compose -f /opt/homelab/vps-apps/<app>/compose.yml down
2. Restore to a scratch location first, never directly onto the live path:

   docker exec backrest sh -c \
     'restic -r /repos/pihdd/vps-apps/data --password-file /run/repo-password-data \
      restore --target /var/tmp/restore --include "opt/homelab-data/<app>/**" latest'
   

   (backup paths are absolute, so they appear under <target>/opt/....)
3. Check ownership/permissions match the app's user:/group: before moving into place.
4. Swap, start, verify, keep <app>.pre-restore until confirmed healthy.

For pi-home apps (native units): stop the unit, restore the subtree of mnt/dietpi_userdata/<app> the same way, fix ownership, start the unit.

Choosing a source repo

• vps-apps died: restore from pi-hdd (fast, wg-direct NFS) or from gdrive (slower, survives everything).
• pi-home died: rebuild pi-home first (node-rebuild runbook). Its pihdd repos die with the HDD — use the gdrive leg for pi-home restores.
• gdrive unavailable (OAuth client retirement, account issue): pihdd leg still covers on-site recovery for both hosts.

Caveats

• Vaultwarden / pi-home data: covered by the pi-home timers. For restores use the vaultwarden-backup runbook — SQLite WAL handling matters.
• pCloud FUSE (romm-assets): data plans can end WARNING with xattr EIO notes on some .srm saves — content is archived. A hung pCloud mount can stall a run for up to 1 h/file; sudo systemctl restart rclone-pcloud.service releases it.
• Test the restore periodically. A backup nobody has restored from is a hope, not a plan.