Vault Auto-Unseal Without Cloud KMS: The Polling Sidecar Pattern

Originally published at woitzik.dev

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

HashiCorp Vault Enterprise has auto-unseal: a sealed Vault automatically unseals using a cloud KMS (AWS KMS, Azure Key Vault, GCP Cloud KMS). Vault OSS doesn't. Every time a Vault pod restarts — node failure, OOMKill, Kubernetes rescheduling — someone has to manually unseal it with vault operator unseal using the unseal keys.

On a homelab cluster where Vault backs ExternalSecrets for 25+ services, a sealed Vault means every ExternalSecret refresh fails. Authelia can't start (no hmac-secret), Open WebUI can't start (no WEBUI_SECRET_KEY), and half the cluster sits in init-container loops waiting for secrets that Vault can't provide.

The fix: a polling sidecar that auto-unseals Vault OSS without a KMS. The trade-off is a collapsed security boundary — but for a homelab, it's the right trade-off.

View the complete homelab infrastructure source on GitHub 🐙

The Sidecar

The vault-unseal Deployment runs as a separate pod in the vault namespace, polling every 5 seconds:

# kubernetes/system/vault/unseal.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-unseal
  namespace: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault-unseal
  template:
    spec:
      serviceAccountName: vault-unseal
      containers:
        - name: vault-unseal
          image: hashicorp/vault:1.21.4
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                # Check if Vault is sealed
                SEALED=$(vault status -format=json | jq -r .sealed)
                if [ "$SEALED" = "true" ]; then
                  # Read unseal keys from the Kubernetes Secret
                  for i in 1 2 3; do
                    KEY=$(kubectl get secret vault-unseal-keys \
                      -n vault -o jsonpath="{.data.key$i}" | base64 -d)
                    vault operator unseal "$KEY"
                  done
                  echo "Vault unsealed at $(date)"
                fi
                sleep 5
              done
          env:
            - name: VAULT_ADDR
              value: "http://vault.vault.svc.cluster.local:8200"
          resources:
            requests:
              cpu: 10m
              memory: 32Mi
            limits:
              cpu: 50m
              memory: 64Mi

The pod needs kubectl access to read the unseal keys from a Kubernetes Secret. The vault-unseal ServiceAccount has a Role that grants get on the vault-unseal-keys Secret only.

The Security Trade-off

In a KMS-based auto-unseal setup, the unseal keys are never stored anywhere accessible to the cluster. The KMS holds the master key, and Vault uses it to decrypt the master key that seals the storage. An attacker who compromises the cluster can't unseal Vault because the KMS key is outside the cluster boundary.

In the polling sidecar pattern, the unseal keys are stored in a Kubernetes Secret (vault-unseal-keys). Anyone who can read that Secret — through kubectl, through a compromised pod with the right ServiceAccount, through an etcd backup — can unseal Vault.

This collapses the security boundary: Vault's unseal protection becomes "Kubernetes RBAC on one Secret" instead of "cloud KMS with its own IAM policy." For a homelab, this is acceptable because:

1. The cluster is not exposed to untrusted users
2. The threat model is "protect against accidental unseal, not nation-state attacker"
3. Manual unseal after every restart is operationally unsustainable

For production, use KMS-based auto-unseal. For a homelab where the alternative is "Vault stays sealed until I notice," the sidecar is the pragmatic choice.

Why 5 Seconds

The initial implementation polled every 30 seconds. This created a window where Vault was sealed but the sidecar hadn't tried to unseal it yet. During that window:

• ExternalSecret refresh requests failed
• Authelia's init container couldn't read hmac-secret
• Any service that depends on Vault-backed secrets was stuck

30 seconds of cluster-wide secret unavailability on every Vault restart. Tightening to 5 seconds reduced the seal window to an acceptable range — most Vault restarts complete unseal within 5 seconds, and the downstream impact is minimal.

The cost: the sidecar makes one vault status call and potentially three vault operator unseal calls every 5 seconds. On Vault's API, this is negligible — it's health-check-level traffic.

The ExternalSecret Dependency Chain

The real reason Vault auto-unseal matters: every ExternalSecret in the cluster depends on Vault being unsealed.

Vault sealed
  → ExternalSecret refresh fails
    → Kubernetes Secrets not updated
      → Pods using those secrets start with stale/missing data
        → Authelia can't start (no hmac-secret)
        → Open WebUI can't start (no WEBUI_SECRET_KEY)
        → Paperless can't start (no database password)

The chain reaction is invisible until you check pod logs and see secret "authelia-secrets" not found or connection refused to Postgres (because the password never synced from Vault).

Before the sidecar, I'd come back to a sealed Vault after a node restart and spend 15 minutes manually unsealing with three key shares while half the cluster sat in CrashLoopBackOff. The sidecar turned a 15-minute manual operation into a 5-second automatic one.

The Network Policy

Vault's ingress is locked down:

# kubernetes/system/vault/network-policies.yml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vault-allow-intra-namespace
  namespace: vault
spec:
  podSelector: {}
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector: {}
          podSelector: {}
    - from:
        - namespaceSelector:
            matchLabels: {}
          podSelector:
            matchLabels:
              app.kubernetes.io/name: external-secrets
      ports:
        - port: 8200

Default-deny ingress on the vault namespace. Only two sources can reach Vault's API port 8200: other pods in the vault namespace (the unseal sidecar), and External Secrets Operator pods. Everything else is blocked.

This is the minimum viable network segmentation for Vault: it needs to be reachable by ESO for secret syncing and by the unseal sidecar for auto-unseal, but nothing else needs direct Vault API access.

What I'd Change

1. Use Vault's built-in auto-unseal with a cloud KMS if available. The sidecar is a workaround for Vault OSS limitations. If you're running Vault Enterprise or can tolerate the cost of a cloud KMS, use it. The sidecar exists because my homelab doesn't have a KMS. Vault's configuration (policies, auth roles) is a separate concern from unsealing — see the staged Terraform migration for how that part is managed without touching unseal keys.

2. Store unseal keys in a more secure backend. The Kubernetes Secret is the weakest link. An improvement would be to store the keys in an HSM or a separate, more restricted secret backend. But at that point, you've basically built KMS-based auto-unseal from scratch.

3. Add monitoring on the sidecar. An alert when Vault transitions from sealed to unseal would provide visibility into restart frequency and sidecar health.

Vault auto-unseal without KMS is the same problem as managing encryption keys in environments without HSMs: you're trading security boundary strength for operational practicality. Azure Key Vault Managed HSM provides FIPS 140-2 Level 3 key protection — but it costs money and adds complexity. For non-production environments, the polling sidecar gives you 90% of the operational benefit at 10% of the security cost. The key is knowing which trade-off you're making.