I built an MCP server for Kafka. It is a backend that connects to Kafka brokers and exposes Kafka operations as tools, defined through the Model Context Protocol, so a Copilot agent can call them. An operator asks a plain-English operational question — a consumer group falling behind, a topic that needs a closer look — and the agent answers from the live cluster instead of from memory.
The interesting part was never the model. The interesting part was everything around it: the tools, the permissions, the rules. This is the design, layer by layer.
Layer 1: You and the copilot
The interaction starts in plain English. The operator states the problem in ordinary language. The agent interprets the intent behind the ask, then picks the right tool from the set it has been given.
That last clause carries the whole design. The agent chooses from a fixed toolset. It does not get a shell, a Kafka CLI, or a blank connection string. Its world is exactly the tools on the server, nothing wider.
Layer 2: The MCP server
The server defines each operation as a tool with a typed schema. Inputs are precise: a topic name is a string with constraints, a group ID is a group ID. There is no free-form parameter for the agent to improvise into.
Two tools show the pattern:
- Describe topic returns partitions, configs, ISR, and offsets for a topic.
- Consumer lag returns group lag read from live metadata — committed offsets compared against the latest offsets, per partition.
Each tool is a narrow, named operation against the cluster. The server, not the agent, owns the broker connection. That connection authenticates with SASL or OAuth and runs with least-privilege permissions: the credentials can do what the tools need and nothing beyond that.
Layer 3: The Kafka cluster
On the other side sits the cluster itself — brokers, topics and partitions, consumer groups with their membership, offsets, and lag, and the health signals: under-replicated partitions, offline partitions, alerts.
The cluster is the live source of truth. Metadata is fetched fresh on every tool call. The agent never answers from a cached snapshot or from patterns in its training data. A lag number in the response is a lag number the cluster reported a second ago.
Design rule 1: Tools are the contract
A vague schema produces improvised inputs, and improvised inputs against a production cluster are a liability. Typed tools keep the agent honest. The contract is written once, by a person, in the schema — and every call the agent makes is forced through it.
This is the same lesson APIs taught a decade ago, applied to a new kind of caller. The caller is probabilistic. The contract must not be.
Design rule 2: Live state, not guesses
An LLM can produce a fluent explanation of consumer lag with no cluster access at all. It will sound right and it may be wrong. The fix is retrieval from the system itself: answers grounded in metadata the cluster just returned.
Every operational claim the agent makes traces back to a tool result. Strip the tool results away and the agent has nothing to say about this cluster — which is exactly the correct behavior.
Design rule 3: Destructive means a human gate
Read tools can run freely. Anything that changes the cluster stops for a person. Always.
This rule is enforced in the server, not requested in a prompt. Prompts are suggestions. Server-side gating is a mechanism. An agent that cannot reach a destructive operation without human approval is an agent that can be left running next to production.
The RAG side
Tools cover live state. They do not cover knowledge: runbooks, design documents, postmortems, the accumulated written memory of a platform team. That is the retrieval side of the system.
Documents are embedded and stored in a vector store. At ask time, the relevant passages are retrieved and handed to the model — Claude, running on Amazon Bedrock — which answers grounded in the retrieved text. The agent ends up with both halves of an operator's brain: the current state of the cluster from tools, and the team's documented experience from retrieval.
Inference where the data already flows
One more piece fits naturally into this stack. Flink UDFs can run LLM inference in-stream, enriching events as they pass through the pipeline. Not every record needs a model call, and the stream is the right place to decide which ones do. Enrichment happens where the data already flows, instead of in a separate service the data has to be shipped to.
Takeaways
- An agent operating infrastructure should work through a fixed set of typed tools, never through open-ended access.
- Typed schemas are the contract, and the contract is what keeps a probabilistic caller predictable.
- Operational answers must come from live system state, retrieved at call time — never from model memory.
- Destructive operations belong behind a human gate enforced by the server, not by the prompt.
- Tools give an agent the present. Retrieval gives it the past. A useful operator needs both.
- The model is the easy part. The system around it is the job.
