Velero Backs Up to the Cluster It's Backing Up: Breaking the Backup Circularity

Originally published at woitzik.dev

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

The circularity: Velero backs up into Garage S3, which runs inside the Kubernetes cluster it's backing up. Cluster lost → Garage PVC lost → Velero backup bucket gone → nothing to restore from.

This is the backup equivalent of storing your offsite backup key in the safe you're backing up. The offsite target exists in config but isn't actually reachable — and you don't realize the gap until you need it.

View the complete homelab infrastructure source on GitHub 🐙

The Two-Problem Stack

Two issues compound into structural fragility:

Problem 1: Backup Circularity

Velero's backup storage location points to Garage, an S3-compatible object store running as a StatefulSet inside the cluster. Every Velero backup — daily snapshots of Vault, Authelia, databases, ArgoCD configs — lands in a bucket that depends on the cluster surviving.

Cluster lost → Garage PVC lost → Velero backup bucket gone → nothing to restore from

PBS backs up the VM/LXC disks to a Seagate 2TB External HDD*, but PBS runs on the same physical host — a BMAX Mini PC*. Simultaneous loss of host + USB HDD means total unrecoverability at the VM layer. Velero was the last safety net — except it was circular.

Problem 2: Torn-State Hazard

Two PVCs with different storage backends, different failure modes:

On 2026-07-12, an unclean Garage shutdown (OOM kill) left db.sqlite corrupted — the same NFS/SQLite locking incompatibility that eventually forced every embedded database off NFS. The metadata — SQLite, bucket/key ownership, access grants — and the actual object data were on different volumes with different rollback fates. A partial restore or node loss would leave one without the other, with no clean way to reconcile them.

The Four Options

Option A: Consolidate meta + data on one volume

Move garage-meta from local-path to NFS (same backend as garage-data). Both PVCs on the same storage class → same snapshot/restore fate.

Verdict: Fixes torn-state but not circularity. NFS for SQLite is proven fragile — the 07-12 corruption happened on NFS-backed data. Moving data to local-path requires hardware you don't have yet.

Option B: External off-cluster target (Cloudflare R2)

Activate the existing r2-offsite backup storage location. Velero dual-targets: in-cluster Garage for speed + off-cluster Cloudflare R2 for bootstrap.

Cluster lost → Garage gone → Velero still has offsite copy on R2
Bootstrap: rebuild cluster → ArgoCD → Velero → restore from R2

Cost: ~$0.015/GB/month on R2 (egress free). At current backup size (~5-10Gi), negligible. Risk: Near-zero — additive only, no existing backup path changes.

Verdict: Directly solves circularity. The existing R2 config is ready — needs credentials and a "go".

Option C: PBS as the primary recovery layer

Lean into PBS as the from-scratch recovery path. PBS already backs up all VMs/LXCs (including the k3s VMs that carry Garage's data). Velero becomes a convenience layer for single-app restore, not the bootstrap path.

Verdict: Already the real bootstrap path (DR Tier 1-4). Velero is Tier 5, only if Garage survived. PBS is the actual safety net. The circularity is a Velero-level problem, not a PBS-level one.

Option D: Combine B + C (what I chose)

Activate R2 offsite for Velero (breaks circularity) AND acknowledge PBS as the primary VM-level recovery layer. Velero becomes the single-app/namespace convenience tool with a real offsite target.

Verdict: Covers both problems. R2 breaks circularity (zero risk, additive). PBS handles VM-level recovery (already there). Garage rebuild from scratch is the fallback if PVCs are lost — better than a partial restore that could leave metadata/data inconsistent.

The Recovery Model

Three independent layers, each covering a different failure scenario:

Scenario                          → Recovery path
─────────────────────────────────────────────────────
Single app crash                  → Kubernetes restart (self-heals)
Single PVC lost                   → Velero restore from R2 offsite
Full cluster lost (host survives) → PBS restores k3s VMs → Velero from R2
Host + USB HDD lost               → Insurance (nothing saves this)

The key insight: no single layer has to be perfect. Each layer covers the failure modes the layer below misses. Garage's torn-state problem becomes irrelevant when the recovery procedure is "rebuild from scratch" rather than "restore from backup."

Implementation (Zero Downtime)

The R2 activation was purely additive — no existing backup path changed:

1. Create Cloudflare R2 bucket (free tier)
2. Populate r2-backuplocation.yml with real R2 credentials via External Secrets Operator
3. Apply the daily-offsite Velero Schedule (already committed)
4. Run one test backup, verify with velero backup describe
5. Confirm PodVolumeBackup entries present for PVCs

The in-cluster Garage target stays. Velero now writes to both: Garage for fast local restores, R2 for bootstrap-from-scratch. If Garage is gone, R2 has everything.

What I'd Change

1. Implement R2 sooner. The config existed for weeks before activation. The circularity was a known risk — the "I'll do it later" mindset is what creates disaster recovery gaps.

2. Add a Velero backup age alert. Alert when the most recent successful backup is older than 48 hours. If backups silently stop, you want to know before you need one.

3. Test the full recovery path. PBS restore of k3s VMs → cluster boot → ArgoCD reconciliation → Velero restore from R2. An untested recovery path is a theory, not a plan.

Backup circularity is the most common structural flaw in self-hosted Kubernetes: the tool that protects you depends on the thing you're protecting. The fix is always the same — add an independent offsite layer. Cloudflare R2 costs nothing at homelab scale, and the R2 egress-free model means you're not penalized for actually using the restore. The hardest part isn't the implementation. It's admitting your current backup doesn't survive the scenario you built it for.

Further reading: For understanding the full recovery architecture, Designing Data-Intensive Applications by Martin Kleppmann* covers replication, backup strategies, and the failure modes that make torn-state hazards so dangerous. The Velero circularity is a specific case of the general problem Kleppmann describes: systems that depend on their own recovery path.