I Learned Go by Hacking Kubernetes RBAC Security (And Nearly Quit 5 Times)

From zero Go experience to shipping kube-radar — a Kubernetes security CLI that detects RBAC wildcard violations. The story, the architecture, the mistakes, and why I did it the hard way.

The Setup

I manage Kubernetes clusters at Siemens. Not "I run kubectl apply from my laptop" manage — I mean Talos Linux on bare metal, Cilium replacing kube-proxy, eBPF policies doing deep packet inspection, ArgoCD running GitOps end-to-end. The kind of infra where one bad RoleBinding can cost you the crown jewels.

But here's the problem: I had never written production Go.

My code was Python — automation, Lambdas, Flask services, AI pipelines. My Go experience was exactly one Pulumi IaC program where I mostly copy-pasted from examples and hoped go build didn't yell at me.

I wanted to become what I call a "backend + cloud-native mega engineer" — someone who can architect a distributed system and ship the code that runs inside it. Not just configure the cluster, but write the controller. Not just deploy the service, but design the API.

So I made a deal with myself:

Build one real, public, useful project in Go. Ship it. Then use it as proof.

That's how kube-radar started.

Why This Project?

Kubernetes RBAC is a nightmare to audit.

If you've ever done a security review on a real cluster, you know the pain:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

This is a cluster-admin equivalent disguised as a single rule. Anyone bound to this can do anything in your cluster. And unlike cluster-admin, it doesn't show up in obvious places like kubectl get clusterrolebindings when you're scanning for privilege escalation.

Wildcard rules (verbs: ["*"], resources: ["*"], apiGroups: ["*"]) are the silent killers of K8s security posture. They slip past most CNAPP tools because they're technically valid — they just give away the kingdom.

I wanted a tool that:

1. Scans any cluster's RBAC in seconds
2. Flags wildcard over-permissions with severity
3. Tracks findings across scans (is this new, or did someone just accept it?)
4. Gates CI/CD with exit codes — fail the build when someone commits a wildcard rule
5. Reports in SARIF for GitHub Advanced Security integration

And I wanted to build it from scratch in a language I'd never shipped in. Because if I can ship this, I can ship anything.

The Learning Curve (Or: How I Nearly Gave Up 5 Times)

Attempt 1: The Cathedral Plan

I sat down and wrote a 5,000-word design document.

I'm not kidding:

• Full SQLite schema with WAL mode, migrations, and ON DELETE CASCADE
• Fingerprinting strategy with SHA-256 canonical hashing
• Cobra CLI with 6 subcommands
• Rule engine with Go interfaces and a plugin registry
• Diff engine with set operations across snapshots
• .goreleaser cross-compilation for 4 platforms
• SARIF 2.1.0 schema validation

It was beautiful. It was also unbuildable for someone learning Go from zero. After 3 weeks of wrestling with database/sql semantics and SQLite driver imports, I had a schema but no working code.

Lesson 1: A perfect design you can't ship is worthless.

Attempt 2: Copy-Paste Panic

I found a GitHub repo that did something vaguely similar. I copy-pasted Go client-go List calls, cobra wiring, even error wrapping. It compiled! It ran! It... panic'd on startup because I shadowed err in an if block and the outer function didn't know.

Go's error-as-values model broke my brain. In Python, I'd raise an exception and catch it somewhere. In Go, the error is just a return value you need to handle every single time. My code was 40% error handling by line count.

Lesson 2: You can't paste your way to understanding.

Attempt 3: The Interface Rabbit Hole

The design doc said "Rule interface type." I spent a full day understanding why interface{} isn't the same as a typed interface, why empty interfaces are a smell unless you need them, and why Go composition beats inheritance so completely.

It clicked when I realized: a Rule in my analyzer isn't a class. It's a contract.

type Rule interface {
    ID() string
    Describe() Description
    Evaluate(s *model.Snapshot) []model.Finding
}

KR001 (wildcard verbs), KR002 (wildcard resources), KR003 (wildcard apiGroups) — all three are just structs that implement those three methods. No inheritance tree. No base class. Just behavior.

Lesson 3: Go interfaces are small, implicit, and compositional. Stop thinking in classes.

Attempt 4: client-go vs. Kubernetes API

Here's something nobody tells you: kubectl does a lot of heavy lifting that client-go doesn't.

kubectl config current-context uses shell-level kubeconfig resolution. client-go's clientcmd.BuildConfigFromFlags doesn't. It took me 2 hours to figure out why my code was connecting to the wrong cluster — client-go reads the kubeconfig file but doesn't use the current-context field unless you explicitly load it.

// This does NOT use your shell's "kubectl config current-context"
config, err := clientcmd.BuildConfigFromFlags("", kubeconfigPath)

// This DOES — it loads the kubeconfig and resolves context, cluster, user
config, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
    &clientcmd.ClientConfigLoadingRules{ExplicitPath: kubeconfigPath},
    &clientcmd.ConfigOverrides{CurrentContext: contextName},
).ClientConfig()

That single discovery changed how I build the kube-radar scan command.

Lesson 4: client-go is lower-level than kubectl in ways that matter.

Attempt 5: Determinism

My first version printed findings in random order every time. Tests flaked. Diff was impossible. Fingerprints changed because I included rule index in the hash.

The fix was a lesson in Go's core philosophy: if the language doesn't guarantee something, you must enforce it yourself. Go map iteration is deliberately randomized. So you sort. Always.

sort.Slice(findings, func(i, j int) bool {
    if findings[i].Severity != findings[j].Severity {
        return findings[i].Severity > findings[j].Severity // desc
    }
    return findings[i].Fingerprint < findings[j].Fingerprint
})

Lesson 5: Randomness is a feature of the language. Determinism is your job.

The Architecture

Here's what I shipped — and more importantly, why the architecture looks like this:

Design Decisions

The Fingerprint System (The Most Underrated Part)

This is where most security tools fail: they generate "new findings" every scan because of random IDs or timestamps. My fingering system:

// v1|<rule_id>|<kind>|<namespace>|<name>|<canonical_rule>
canonical_rule = sort-and-join(apiGroups) + "|" + sort-and-join(resources) + "|" + sort-and-join(verbs)
fp = sha256(v1|KR001|ClusterRole||admin|*|*|*)[:16]

Key properties:

• Stable across rescans: Same cluster state → same fingerprint
• Survives reordering: If someone reorders rules in a Role, the fingerprint doesn't change
• Detects narrowing: verbs: ["*"] → verbs: ["*"] + resources: ["pods"] is a different finding (content changed)

This makes the diff feature in v0.2 trivial: it's just set arithmetic on fingerprint maps.

Exit Codes as CI Contract

0 — Clean scan, no new findings ≥ --fail-on
1 — New unacknowledged findings ≥ --fail-on severity threshold
2 — Operational error (unreachable cluster, bad flags, DB failure)

Why only 3 codes? Because CI pipelines need simple logic: kube-radar scan && deploy || fail works with zero tooling.

What It Looks Like

$ kube-radar scan --context homelab --fail-on medium

RULE    | KIND        | NAMESPACE | NAME          | DETAIL
--------|-------------|-----------|---------------|------------------
KR001   | ClusterRole | -         | admin-legacy  | Wildcard verbs: [*]
KR002   | Role        | default   | app-service   | Wildcard resources: [*]
KR003   | ClusterRole | -         | cluster-admin | Wildcard apiGroups: [*]

3 findings (1 Critical, 1 Medium, 1 Low)  exit code: 1

That exit code: 1 means CI fails. That means the wildcard rule does not get deployed.

3 Lessons That Transfer Beyond This Project

1. Design the hard thing in writing first

My 5,000-word design doc wasn't a waste. It was a map. When I cut the scope to v0.1 (no SQLite, no diff, no SARIF), I knew exactly what each deferred feature looked like because I'd already designed it. The doc became the roadmap.

2. Build for yourself, then write about the pain

Every feature in kube-radar solves a problem I actually have:

• Can't find wildcard rules quickly → the scan command
• Can't tell if a finding is new → the diff feature (v0.2)
• Can't baseline accepted risk → acks (v0.2)
• Can't integrate with GitHub Security → SARIF (v0.2)

Tooling built from real pain is better than tooling built for a portfolio.

3. A shipped v0.1 beats a perfect v2.0

I had to fight my own instinct to build everything at once. v0.1 is:

• 1 command (scan)
• 3 rules (KR001-003)
• Table output only
• Fingerprints computed but not yet persisted

But it works. It already catches real violations. It already fails CI builds. Everything else is additive.

The Roadmap

Try It

git clone https://github.com/beltagyy/kube-radar.git
cd kube-radar
go build -o kube-radar ./cmd/kube-radar
./kube-radar scan --kubeconfig ~/.kube/config --fail-on medium

Or grab a release binary (when I cut v0.1.0 — soon).

What I need from you:

• Does it work on your cluster?
• What's missing for your use case?
• Are there wildcard variants I'm not catching?

Every issue helps me learn.

About This Series

This is post 1 in the "kube-radar: Building a Security CLI in Go" series. Next up:

• Deep-dive: Why I chose SQLite over in-memory state
• Deep-dive: Designing deterministic fingerprints for security findings
• Deep-dive: The Go interfaces that made the rule engine testable

If you're learning Go from a cloud background like I am — or if you're a backend engineer who wants to understand K8s security internals — follow along.

*Questions? I'm @beltagyy on GitHub and beltagyy on DEV.