Backups | Jerry Kroskinski

Backups only count if you'd actually trust them mid-emergency. Every production dataset in the homelab is covered by restic, running on staggered schedules across local and offsite destinations, with failure reporting that assumes silence is dangerous, not reassuring.

3-2-1, Exceeded

Production data plus a local restic repo, an on-demand sync to a genuinely separate on-site machine, and two independent Backblaze B2 buckets -- one for general app data, one dedicated to Immich's photo library -- adds up to four copies across three physically independent locations. That clears the standard 3-2-1 minimum (3 copies, 2 media, 1 offsite) with room to spare.

Each B2 bucket has its own scoped Application Key -- read/write to that bucket only, never the account's master key.
Databases are stopped before every backup that touches them, restarted immediately after, so a snapshot never catches Postgres mid-write. A trap ... EXIT guarantees the restart happens even if the backup itself fails partway through.
Schedules are deliberately staggered across hosts and destinations, so nothing saturates the same disk or uplink at the same time.

Dead Man's Switch

Every backup script reports its own exit status to Uptime Kuma on every run -- success or failure -- rather than only pinging when things go well. A cron job that silently dies to a syntax error or a full disk gets caught by Kuma's own heartbeat timeout instead of going unnoticed until someone happens to check. Status now pushes to two independent Uptime Kuma instances (one local, one external), so an outage of either monitor doesn't blind the other to a real failure.

Gotcha: a "successful" push isn't proof of anything. An early version of the reporting script used curl -f, which only treats 4xx/5xx as failure. When an auth gate silently redirected a push request to a login page instead of erroring, curl -f saw a normal HTTP response and reported success -- while the monitor it was supposed to update never actually moved. A cron job's own exit code isn't proof the thing it was reporting on actually happened; the fix was checking the real response, not just the script's happy path.

Immich's Storage Layout

Immich v3+ shards files into content-hash-based folders, with originals and derived thumbnails/transcodes mixed together inside the same buckets -- there's no clean path-based way to back up "just the originals" the way older guides describe. Given how cheap B2 storage is at this data size, backing up the whole directory is the simpler and more honestly-correct choice over trying to exclude derivatives via the database.

Known gap: every backup completes cleanly and passes its own integrity check, but a full end-to-end restore hasn't actually been tested on either offsite repo yet. A clean check proves the archive isn't corrupted -- it doesn't prove a real restore would go smoothly. On the list before this counts as fully trusted.