A customer got charged twice for one order. The payment provider's dashboard showed two authorizations, same amount, eleven minutes apart. Our Kafka consumer group had enable.auto.commit=false and isolation.level=read_committed. We had turned on the transaction flag. The logs showed the same order_id processed twice, and the second run sailed straight through our handler.
What the broker actually guarantees
Kafka's "exactly-once semantics" means: when you consume from topic A and produce to topic B inside the same transaction, the offsets you commit and the records you write land atomically. Consumer group metadata, produced records, committed offsets. One atomic unit, inside Kafka.
That is a real guarantee. It is also narrower than the name suggests. It says nothing about the HTTP call your handler makes to Stripe, the row it inserts into Postgres, or the email it hands to SendGrid. Those are outside the transaction. Kafka cannot roll them back, and it cannot know they happened.
The failure is not exotic. Your handler charges the card. Then the process is killed before commitSync runs — OOM killer, deploy, network partition, a SIGKILL from the orchestrator. The broker never saw an offset commit. The partition is reassigned. A new consumer picks up at the last committed offset and runs your handler again. The card is charged twice. No error was thrown anywhere.
This is the two generals problem wearing a config flag. Two parties need to agree on one bit — did the side effect happen? — over a channel that can lose the acknowledgment. The sender cannot distinguish "the message never arrived" from "the ack never came back". No timeout, retry count, or protocol change fixes it. It is a proof, not an engineering gap.
The ack can die after the side effect
The asymmetry matters. Losing a message before the side effect is safe: retry, and nothing duplicated. Losing the ack after the side effect is the dangerous case, because the retry is indistinguishable from a first attempt.
So you cannot make delivery exactly-once. You can make processing idempotent, which is a property of your handler, not the broker. That is the actual answer, and it is your code's job.
Deduplication with a uniqueness constraint
Do not dedup in application memory. A Set of seen IDs dies with the process and grows without bound. Put the constraint where it can be enforced atomically: in the database, next to the data you are writing.
CREATE TABLE processed_messages (
consumer text NOT NULL,
message_id uuid NOT NULL,
processed_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (consumer, message_id)
);
Then write the side effect and the dedup record in one transaction:
def handle(msg):
with db.transaction():
try:
db.execute(
"INSERT INTO processed_messages (consumer, message_id) VALUES (%s, %s)",
("billing", msg.headers["message_id"]),
)
except UniqueViolation:
return # already done; ack and move on
db.execute("UPDATE orders SET charged = true WHERE id = %s", (msg.key,))
# commit the offset only after the transaction above returns
consumer.commit(msg)
Two rules make this work. The dedup key must come from the producer and be stable across redeliveries — a UUID generated inside the consumer is useless, because the retry makes a new one. And the offset commit must happen after the transaction commits, so a crash between them replays the message into a handler that will reject it cleanly.
Your database is now the arbiter. Kafka's transaction is a nice optimization; it is not the guarantee.
The produce side needs an outbox
The same problem faces writes. If your service updates a row and then publishes an event, a crash between the two leaves the database and the topic disagreeing. Publishing first and writing second has the mirror failure.
The transactional outbox closes it. Write the event into an outbox table in the same transaction as the business row. A separate relay reads unpublished rows and publishes them. The transaction gives you atomicity because both writes share one database. The relay gives you delivery, and since it may publish a row twice, it stamps each event with a stable ID that the consuming handler dedups — the same constraint as above.
Stop calling it a flag
"We enabled exactly-once" is not a design. It answers a question nobody asked: whether the broker's internal bookkeeping is atomic. It says nothing about your handler's idempotency, your dedup key's origin, or whether the offset commits after the side effect.
Pick message IDs at the producer. Enforce uniqueness in the store that holds the side effect. Commit offsets last. Then measure the failure rate you actually care about: count duplicate message_id values that reach the handler and compare it to the number of successful side effects. That ratio tells you whether your dedup is doing the work the flag pretended to do.
The two generals will still not agree. Your database can.
I write about production failures in Postgres, queues, and distributed systems.
Subscribe by email · RSS · Bluesky
