📜 HomeLab Chronicles: Episode 6 - Source of Truth

Hey all 👋

Last episode a power cut exposed an uncomfortable fact: my cluster's entire memory lived in one SQLite-flavored database, on one laptop, bound to one Wi-Fi address, guarded by one aging battery. Four single points of failure in a trench coat.

The fix isn't making that database unkillable. The fix is making it unimportant. If every manifest lives in git and something reconciles the cluster against git continuously, then "the datastore died" stops being a tragedy and becomes a reboot with extra steps.

So: Flux. Here's the setup, and the four ways I face-planted installing it.

🗂️ The Shape of the Repo

clusters/homelab/
  flux-system/          <- Flux writes this at bootstrap; hands off
  infrastructure.yaml   <- points at infrastructure/
  apps.yaml             <- points at apps/
infrastructure/
  controllers/          <- Longhorn + Envoy Gateway HelmReleases
  configs/              <- GatewayClass, Gateway, StorageClasses
apps/homelab/
  airflow/              <- the actual point of all this

Three Flux Kustomizations, chained: infra-controllers → infra-configs → apps, via dependsOn. That chaining is not decoration. My GatewayClass can't exist until Envoy Gateway's CRDs exist, and the CRDs arrive with the controller's Helm chart. Without dependsOn, Flux sprints ahead, tries to create a GatewayClass into a cluster that's never heard of GatewayClasses, and fails with the enthusiasm of a golden retriever running into a glass door.

dependsOn plus wait: true turns that into: install controllers, wait until healthy, then configs, then apps. Boring. Sequential. Correct. The three great virtues.

🔑 Sidequest 1: The Token Bureaucracy

flux bootstrap github needs a GitHub token, and the docs-diving summary is:

• Classic PAT: repo scope. Needed if Flux should create the repo.
• Fine-grained PAT (pre-created repo): Contents read/write, Metadata read, and — the one everyone misses — Administration read/write, because Flux installs an SSH deploy key on the repo, and deploy keys are an admin operation.

Here's the nice part: the deploy key is what Flux uses forever after. The PAT is scaffolding. Bootstrap, confirm flux get kustomizations goes green, then revoke the token. Give it a 7-day expiry at creation and let it die of natural causes. Also: I renamed the repo mid-project and exactly zero manifests cared — the repo URL lives only in the bootstrap-generated gotk-sync.yaml. Rename after bootstrapping though, and Flux keeps pulling the old URL. Ask me how I know I almost knew that.

💥 Sidequest 2: The Secret That Must Come First

Some things cannot live in a public-ish git repo: the git deploy key for DAG sync, the Airflow admin password, the JWT signing key. Those get created by hand (well — by script), out-of-band.

And order matters more than I expected. My Airflow HelmRelease pulls its password via valuesFrom with optional: false — no secret, no install. I bootstrapped first and created secrets second, like a person with hope in their heart. Flux tried to install Airflow, found no secret, retried three times, exhausted its retry budget, and parked the release in a failed state.

Then I ran the secrets script and hit this beauty:

✗ Failed to perform remediation: missing target release for rollback:
  cannot remediate failed release

Translation: the first ever install failed, so there's no previous good release to roll back to, so Flux's remediation has nothing to grab. A plain flux reconcile won't retry it. You need --force, or a suspend/resume, or — the actual lesson — create the secrets before you bootstrap. The cluster exists after MicroK8s comes up; Flux doesn't need to be watching yet for you to kubectl create secret.

📌 Sidequest 3: Pin Your Charts Like You Mean It

First draft, every HelmRelease said version: "1.x". Feels flexible. Is flexible. Flexible means "a major chart upgrade can land at 3am while a reconcile loop runs unattended," and for a storage system that's not a version policy, that's a séance.

So I went and found the real versions — including the fun discovery that Airflow's chart 1.22.0 has appVersion: 3.2.2, exactly the Airflow I was running. Now the repo says:

airflow:      version: "1.22.0"
longhorn:     version: "1.12.1"
gateway-helm: version: "v1.9.1"

Upgrades still happen. They just happen when I commit them, on purpose, with coffee.

🫥 Sidequest 4: The Dot of Shame

One evening a reconcile hung with context deadline exceeded and zero pods appearing. Twenty minutes of cluster spelunking later, I looked at my own shell prompt:

╭─sunny@sunny ~/github/lab ‹main●›

That little dot. Uncommitted changes. I'd committed the kustomization entry for a new directory but not the directory. Flux was faithfully applying exactly what I'd pushed, which was a reference to a folder that existed only on my laptop. Flux cannot read your working tree. Flux cannot read your intentions. Flux reads origin/main, and origin/main reads whatever you actually pushed.

The dot was there the whole time. Judging me.

⚖️ What Flux Is Not

Worth saying plainly, because I briefly believed otherwise: Flux is not a backup system. It restores what's declared — Deployments, HelmReleases, routes. It knows nothing about what accumulated at runtime: your Postgres data, DAG run history, Longhorn volume contents, the secrets you created by hand.

So the full recovery story is a two-piece set: Flux rebuilds the skeleton from git, and last episode's nightly dbctl snapshots plus (eventually) Longhorn backup targets restore the organs. Either one alone is half a cluster.

The payoff, though, is real: the rebuild I did mid-way through all this — full microk8s reset, rejoin the Dell, bootstrap, secrets script — took about twenty minutes, most of it image pulls. Episode 5 me would have wept. Episode 6 me made tea.

🧠 What This Taught Me

• dependsOn is load-bearing. CRDs before custom resources, controllers before configs, or Flux runs face-first into glass.
• Secrets before bootstrap. An optional: false reference plus a missing secret burns the retry budget and needs --force to resurrect.
• A first-install failure has no rollback target. "Cannot remediate failed release" means delete-and-retry, not reconcile-harder.
• Version ranges are unattended upgrades wearing a disguise. Pin, then upgrade on purpose.
• Flux reads the remote, not your working tree. The ● in your prompt is a Flux error message you're rendering locally.
• GitOps restores manifests, backups restore data. You need both, and neither is impressed by the other.

📋 Quick Reference (For Skimmers)

🚀 What's Next

With the repo in charge, I did the responsible thing and enabled RBAC on the fresh cluster — you know, best practice — and the control plane immediately locked itself out of its own house. Next episode is short, painful, and has a body count of one coredns.

💬 Final Thoughts

Git is now the cluster. The cluster is now a cache. The dot in my prompt is now a trauma response.

Popcorn 🍿, coffee ☕, UPS 🔌, and a pre-push git status — the ritual expands.