My self-hosted CI runner was the least trusted machine I owned. Now every job runs in a disposable Firecracker microVM on Krova Cloud — here's the setup, the scripts, and the trade-offs.
Earlier this year a boring dependency-bump PR came in to one of my open-source projects. One hunk touched the CI config, just slightly. I almost merged it with one eye on the diff.
Then I actually read it: one extra step on every build. And my self-hosted runner — a cheap VPS I'd set up to save CI minutes — ran jobs from forks. It also had cloud credentials sitting in a config file some past version of me had left there "for convenience."
A malicious PR wouldn't have needed to escape anything. It could have curled my environment to a pastebin and the runner would have obliged.
The real lesson wasn't about that PR. It's that a self-hosted runner is a machine that executes strangers' code, and then keeps living. My cheapest machine was my most dangerous one.
Why I didn't just containerize the runner
containers are the standard answer, and they're not nothing. But on a runner they're a screen door where you need a wall:
shared kernel with the host
mounted caches that outlive the job (a poisoned cache hands itself to the next build)
often a Docker socket
a filesystem that persists between builds
I didn't want politeness. I wanted a boundary. So now every CI job gets a fresh Firecracker microVM — own kernel, own rootfs — and the machine dies when the job ends.
The per-job lifecycle
On Krova Cloud each microVM is a Cube. Provisioning is one command and boots in under a second:
npm i -g @krovacloud/cli
krova cubes create ci-job-812 --cpu 2 --ram 4 --disk 40 --image ubuntu-24.04
# ✓ Cube provisioned · booted in 0.9s
The wrapper my pipeline calls per job:
#!/bin/bash
set -euo pipefail
CUBE="ci-job-$BUILD_ID"
# fresh box per job — nothing carries over, ever
krova cubes create "$CUBE" --cpu 2 --ram 4 --disk 40 --image ubuntu-24.04
# secrets injected at runtime as short-lived, scoped tokens.
# never in the image, never on disk after teardown
krova ssh "$CUBE" "DEPLOY_TOKEN=$DEPLOY_TOKEN ./run-build.sh"
# teardown is the security feature
krova cubes delete "$CUBE"
When the orchestrator does it directly (retries included — note the idempotency key so a retried request doesn't double-provision):
curl -X POST https://krova.cloud/api/v1/spaces/$SPACE/cubes \
-H "X-API-KEY: $KROVA_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{
"name": "ci-job-812",
"image": "ubuntu-24.04",
"resources": { "vcpu": 2, "ramGb": 4, "diskGb": 40 }
}'
What the runner looks like from inside
This is the part that changed my threat model. The job's machine has no public IP — it lives on a private NAT'd network:
root@ci-job-812:~# ip -brief addr
lo UNKNOWN 127.0.0.1/8
eth0 UP 10.0.x.x/24 # private. that's it.
root@ci-job-812:~# ss -tlnp
# nothing listening that you didn't start.
# no SSH on the internet, no scannable surface.
Inbound is default-deny; only ports you explicitly open are reachable, and those can be IP-allowlisted. A CI box opens none.
So the three properties that matter:
- Nothing persists. No poisoned cache, no leftover cron, no creds on disk for the next build.
- The boundary is real. Own kernel per Cube (Firecracker — same tech behind AWS Lambda). A kernel bug popped in a build lands in a box that's seconds from deletion.
- No address to attack. The runner isn't a scannable VPS with port 22 on the internet.
Cost: billed by the minute, so a 12-minute job on a 2 vCPU / 4 GB Cube is a few cents. Ephemeral only works when teardown is cheap — that's the whole reason this architecture is affordable.
What this doesn't fix (read this part)
Isolation ≠ safety. If you hand a job a long-lived production credential, a fresh VM just means the credential gets stolen from a brand-new machine. So on top of the VM:
- short-lived, scoped tokens per job — injected at runtime, gone at teardown
- no shared volumes between jobs
- egress restricted where it matters
Krova gives you the blast radius. Secret hygiene is still your job, on any platform. Anyone selling "secure CI" as a checkbox is selling something.
The reframe
That PR moved CI from "infrastructure" to "supply chain" in my head. The code your pipeline runs is code you didn't write; the machine that runs it should be disposable, bounded, and unreachable.
Give every job its own box. Kill the box when it's done.
Has a PR ever targeted your CI instead of your code? And if you run ephemeral runners — microVMs, containers, or bare-metal chaos — what does your teardown look like? Genuinely comparing notes, comments open.
