Kafka as Code: Beyond Topics

Someone on your team asked "which Terraform provider do we use for Kafka?". Your answer is probably between Mongey/kafka and your vendor's provider like Aiven, MSK or Confluent.

Good, but the question can actually be broader. There are three separate layers hiding in there:

Creating the cluster (brokers, VPC, MSK Serverless)
Creating the topics (partitions, retention, ACLs)
Governing the platform (who owns what, which rules apply, what a team can ship without asking)

Both 1 and 2 have had providers for years. Layer 3 is just getting started with solutions like Conduktor. Let me show you why you should care and why it exists now.

The Kafka protocol has no idea who owns what

No provider that speaks the Kafka protocol can express ownership, because the protocol has no concept of it.

A topic in the Kafka protocol is a name, a partition count, a replication factor, and a config map. That's it. There is nowhere to put "the payments team owns this", "this contains PII," or "C1 criticality". Not because providers are lazy but because there's no field.

So what happens instead? You build a central Terraform repo. The platform team owns it. Every topic request becomes a PR into a codebase the requester has never seen. And then:

"They'll go in and have to make a PR to that repo. It's not super intuitive on how to do that. So they'll end up pinging our team anyway." — platform engineer at a consumer-lending company

Self-service that routes through a human is just a ticket queue with extra YAML.

Two other things that bite

Your CI becomes the most privileged Kafka client in the company. A provider that dials brokers needs admin credentials and a network route to every broker of every cluster, with TLS/SASL/IAM wired up in the runner. Every new cluster is a new networking problem and a new credentials problem in your pipeline.

Terraform will happily delete your data. AWS's own MSK Terraform tutorial walks you through changing a topic from 50 partitions to 10. Kafka can't shrink partitions. So Terraform destroys and recreates the topic, data included. The plan says:

Plan: 1 to add, 1 to change, 1 to destroy.

Which is technically accurate and completely fails to communicate "you are about to lose weeks of events." and make your consumers very unhappy. (without mentioning business impact)

Notice the trend AWS itself is following: the newer aws_msk_topic resource takes a cluster_arn, not bootstrap servers. Don't dial brokers from CI. Talk to a control plane over HTTPS.

What a control-plane provider looks like

I've been using the Conduktor provider. It doesn't create clusters, it registers ones that already exist and governs what runs on them. MSK, Confluent, Aiven, Redpanda, self-managed, Gateway virtual clusters: same model regardless.

Registering an existing MSK cluster:

resource "conduktor_console_kafka_cluster_v2" "msk" {
  name = "payments-msk"
  spec = {
    display_name      = "Payments (MSK, eu-west-1)"
    bootstrap_servers = "b-1.xxxxx.kafka.eu-west-1.amazonaws.com:9198"
    properties = {
      "sasl.jaas.config"  = "software.amazon.msk.auth.iam.IAMLoginModule required awsRoleArn='arn:aws:iam::123456789123:role/MSK-role';"
      "security.protocol" = "SASL_SSL"
      "sasl.mechanism"    = "AWS_MSK_IAM"
    }
  }
}

A Gateway virtual cluster registers exactly the same way, so your non-prod isolation is governed by the same code path as prod:

resource "conduktor_console_kafka_cluster_v2" "payments_vc" {
  name = "payments-vc"
  spec = {
    display_name      = "Payments (virtual cluster)"
    bootstrap_servers = "gateway.internal:6969"
    properties = {
      "security.protocol" = "SASL_SSL"
      "sasl.mechanism"    = "PLAIN"
    }
    kafka_flavor = {
      gateway = {
        url             = "https://gateway.internal:8888"
        user            = "admin"
        password        = var.gateway_admin_password
        virtual_cluster = "payments"
      }
    }
  }
}

The topic carries its own metadata

Same topic you'd declare anywhere, except now the fields that matter to humans are first-class:

resource "conduktor_console_topic_v2" "orders" {
  name    = "click.orders.events"
  cluster = conduktor_console_kafka_cluster_v2.msk.name

  labels = {
    domain      = "orders"
    criticality = "C1"
  }
  description = "Order lifecycle events, one message per state change."

  spec = {
    partitions         = 6
    replication_factor = 3
    configs = {
      "cleanup.policy" = "delete"
      "retention.ms"   = "604800000"
    }
  }

  lifecycle {
    prevent_destroy = true
  }
}

That prevent_destroy is not optional, see the partition-shrink story above.

Policies are Terraform resources

The rules a topic must satisfy are themselves Terraform resources:

resource "conduktor_console_topic_policy_v1" "orders_rules" {
  name = "orders-topic-rules"
  spec = {
    policies = {
      "metadata.name" = {
        match = { pattern = "^click\\.(?<event>[a-z0-9-]+)\\.(avro|json)$" }
      }
      "metadata.labels.criticality" = {
        one_of = { values = ["C0", "C1", "C2"] }
      }
      "spec.configs.retention.ms" = {
        range = { min = 3600000, max = 604800000 }
      }
    }
  }
}

Now compare this to the usual approach: an OPA or Sentinel check on the Terraform plan. That check only fires for topics created through that repo. Anyone who opens the UI, runs a CLI, or curls the API will just ignore it.

A control-plane policy object is evaluated on every creation path, because it lives where the creation is admitted, not where one particular pipeline runs.

Attach the policy to an application, hand the team an API key scoped to their own application instance instead of an admin key, and the loop closes: the dev applies Terraform in their own repo, with their own credentials, and gets

Error: topic name "orders-events" does not match ^click\.(?<event>[a-z0-9-]+)\.(avro|json)$

...which they can fix in thirty seconds without pinging anyone. That's the difference between self-service and a ticket with YAML.

Where this stops

To clarify, it's not one provider to rule them all:

It does not create clusters. No brokers, no VPC, no MSK Serverless. Keep hashicorp/aws or confluentinc/confluent for the infra layer. The pairing is: cloud provider creates the cluster, control-plane provider governs what runs on it.
It requires Conduktor Console. It's the IaC surface of a platform, not a standalone Kafka client.