Architecture walkthroughs tend to show you the polished version — the diagram that survived a whiteboard, cleaned up after the fact. What they skip is the sequence of decisions, tradeoffs, and dead ends that got there. That's what I want to share here, based on having built and operated Kafka-based payment pipelines at production scale in Brazilian fintech.

Start with Requirements, Not Technology

The first mistake teams make is starting with "we're going to use Kafka" before defining what correctness means for their payment events.

Key questions before drawing a single box:

What is the ordering requirement? Within a Kafka partition, ordering is guaranteed. Across partitions, it is not. If you need per-payment ordering (all events for a given payment ID in sequence), partition by payment ID and you get it for free.

What is your durability requirement? Kafka with acks=all and min.insync.replicas=2 is durable. For payment events, this is non-negotiable.

What is your latency requirement? Kafka adds latency — typically single-digit milliseconds for the publish path. If you need sub-millisecond authorization decisions, use Kafka for audit, reconciliation, and notifications; use a direct synchronous call for the authorization itself.

Topic Structure

Design topics around event lifecycle stages, not consumers:

payment.initiated       — customer submits payment intent
payment.authorized      — auth decision from card network / PIX
payment.settled         — funds confirmed settled
payment.failed          — any terminal failure
payment.refund.initiated
payment.refund.settled

Why separate topics? Consumer groups subscribe at topic granularity. A reconciliation service only cares about settled events — it shouldn't filter through initiated events at volume. Separate topics allow independent scaling, retention policies, and consumer group management.

What I avoid: a single payment.events topic with a type field that consumers filter on. The fan-out cost is real at scale.

Partitioning Strategy

Partition by payment ID. Payment reconciliation requires that all events for a given payment arrive in order at the consumer. Partitioning by customer ID creates hot partitions for high-volume customers (merchants, corporate accounts).

Use hash(paymentId) % partitionCount to compute partition number.

Consumer Design

Every consumer must be idempotent. This needs to be stated explicitly because Kafka's exactly-once semantics does not make your consumers idempotent for external writes.

@KafkaListener(topics = "payment.authorized")
@Transactional
public void handleAuthorized(PaymentAuthorizedEvent event) {
    // Idempotency check — unique constraint on event_id handles races
    if (paymentRepository.existsByEventId(event.getEventId())) {
        return;
    }

    Payment payment = paymentRepository.findByPaymentId(event.getPaymentId())
        .orElseThrow();
    payment.authorize(event.getAuthCode(), event.getTimestamp());
    paymentRepository.save(payment);

    // Outbox entry in same transaction for downstream event
    outboxRepository.save(new OutboxEntry(event.getEventId(), "payment.settled", ...));
}

The outbox entry ensures that the downstream event and the database state change are atomic.

Dead Letter Queue

Every consumer topic needs a corresponding dead letter topic:

payment.authorized.DLT
payment.settled.DLT

When a consumer throws an unrecoverable exception, the message routes to the DLT with error metadata appended. Do not retry poison messages indefinitely on the main topic — they stall the entire partition.

Instrument DLT queue depth as a critical metric. A growing DLT is an incident.

Schema Evolution

Payment event schemas change. Card networks add fields. PIX adds event types. Use Avro or Protobuf with a schema registry — both enforce backward/forward compatibility at publish time.

Mark all new fields as optional with defaults. Never remove fields from published schemas without a full migration plan for all consumers.

Monitoring That Matters

• Consumer lag per partition — not just the sum, the distribution
• Processing latency histogram (p50, p99, p999)
• DLT queue depth per topic
• Rebalance frequency and duration

Treat DLT depth and processing latency as paging metrics.

The Part Most Design Docs Skip

The hardest design decision isn't the technology choices — it's defining your payment state machine.

A payment can be initiated, pending authorization, authorized, settling, settled, failed, refunded, partially refunded, disputed. Not all transitions are valid. If you don't model this explicitly before building consumers, you'll encode the state machine implicitly in consumer logic — scattered, inconsistent, hard to audit.

The event pipeline is infrastructure. The state machine is the business logic. Get the business logic right first.

If you found this useful, I run 1:1 mentoring sessions for Java/backend engineers at topmate.io/aliasgar_kantawala

My Java interview guides and system design resources are at aliasgarmk.gumroad.com