How Kafka Saved a Failing System -The Story of Arjun & Meera

Modern systems run on data in the form of fast, real-time, continuous streams.

But what happens when your system grows faster than your architecture can handle?

This is the story of Arjun, a backend engineer from Indore, and Meera, a DevOps engineer — and how Kafka rescued their collapsing system while teaching them how real-time streaming actually works.

It's a story, but it's also a complete walkthrough: why queues break, what Kafka replaces them with, the code, the bugs, and the cases where Kafka is genuinely the wrong choice.

Chapter 1: The Breaking Point

SwiftKart, a fast-growing hyperlocal delivery startup, was booming.

Orders jumped from 4,000 to 40,000 per day in just two months.

But growth brought chaos. Arjun, the Node.js backend engineer, was drowning:

• Notifications delayed by 15 minutes
• Redis queue overflowing
• Analytics dashboard freezing
• Duplicate messages
• Missing events
• Workers crashing randomly

Meanwhile, Meera's monitoring showed red spikes everywhere.

Arjun said:

"Meera, the system can't breathe. The Redis queue is suffocating!"

Their stack was the classic startup design:

• Node.js microservices
• MySQL
• Redis job queues
• Background workers

Perfect at small scale. Completely helpless for high-volume real-time streaming.

Chapter 2: Why Their Old System Failed

As traffic increased, the cracks became craters.

1. Queue overflow. Redis had a single-lane queue → bottlenecks everywhere.

2. No durability. If Redis restarts, messages disappear.

3. No replay. Consumed once = gone forever. If analytics breaks, that data is lost permanently.

4. Slow consumers collapsed the system. Events arrived faster than workers could drain them, and back-pressure took everything down with it.

5. No multi-consumer support. Analytics, notifications, fraud detection — sabko same event chahiye tha. Redis sirf ek consumer ko deta tha. Baaki sab bhookhe.

Arjun finally said:

"We need a real streaming system. Not this jugaad."

Chapter 3: The Monday Meeting — Kafka Enters

The CTO walked into the meeting room and declared:

"We are moving to Kafka."

Arjun frowned. "Isn't Kafka too complex? Why do we need it?"

Meera smiled. "Kafka isn't overkill. It's exactly what we need."

What Kafka gives you:

And the cons, honestly:

1. Complex to operate
2. Resource-heavy
3. Genuine overkill for small apps

But SwiftKart had moved well beyond "small."

Chapter 4: Kafka Explained Like Arjun Understood It

Meera drew a diagram on the whiteboard and started with the sentence that reframed everything:

Kafka is not a queue. Kafka is a distributed, durable, replayable event log.

A queue deletes a message when someone reads it. A log doesn't. Readers just move a bookmark forward, and every reader has their own bookmark.

Then the vocabulary:

The one that took Arjun longest: consumer group.

Consumers in the same group split partitions between them — that's how you scale. Consumers in different groups each get their own copy of every message — that's how notifications, analytics and fraud detection all read the same event stream without fighting over it.

That single distinction is what Redis could never do.

Chapter 5: The Migration Plan — Nobody Turns Redis Off on a Friday

Arjun's first instinct was to rip out Redis in one weekend.

Meera stopped him.

"Agar tumne ek hi raat mein switch kiya, aur Kafka mein koi bug nikla, toh 40,000 orders ka data kahan jayega? Migration is not a deploy. It's a slow, boring, reversible process."

They agreed on a three-week plan.

Week 1 — Dual write. Every service that pushed to Redis now also published the same event to Kafka. Redis stayed the source of truth; Kafka just quietly collected events. Zero risk — if Kafka broke, nothing broke.

Week 2 — Shadow consumers. New Kafka consumers ran alongside the old Redis workers, doing the same work but writing to a separate analytics table. Every morning Meera compared both tables. When the counts matched for seven days straight, they had proof.

Week 3 — Cutover, one consumer at a time. Notifications first (least damage if it broke). Analytics second. Fraud detection last. The Redis workers were disabled, not deleted, for another two weeks.

Then came the part Arjun underestimated: designing the topics.

His first draft had one topic called events. Meera rejected it immediately.

"One topic for everything means one retention policy for everything, one partition count for everything, and no way to give a team access to only what it needs."

The final design:

Two rules drove every choice.

1. The key decides the order.

Kafka guarantees ordering inside a partition, not across a topic. Keying by orderId means every event for order #5512 — CREATED, PACKED, PICKED_UP, DELIVERED — lands in the same partition and is read in sequence. Key it by something random and you will eventually deliver an order before you accept it.

2. Partition count is your maximum parallelism.

A consumer group can never have more active consumers than partitions. 12 partitions = up to 12 notification workers, and the 13th sits idle. You can add partitions later, but adding them rehashes the keys, so old and new events for the same order can land in different partitions. Over-provision a little on day one.

Chapter 6: The Code Arjun Actually Wrote

They used KafkaJS, since the whole backend was Node.js.

The event envelope

Before writing a single producer, Meera made him standardise the shape of every event:

// events/envelope.js
import { randomUUID } from "crypto";

export function buildEvent({ type, version = 1, orderId, payload }) {
  return {
    eventId: randomUUID(),   // for consumer-side deduplication
    type,                    // "ORDER_CREATED"
    version,                 // schema version — you WILL need this
    orderId,
    occurredAt: new Date().toISOString(),
    payload,
  };
}

"Version field abhi useless lagta hai," she said. "Six months baad jab tum payload change karoge aur purane consumers crash honge, tab samajh aayega."

The producer

// kafka/producer.js
import { Kafka, Partitioners, CompressionTypes } from "kafkajs";

const kafka = new Kafka({
  clientId: "order-service",
  brokers: process.env.KAFKA_BROKERS.split(","),
  ssl: true,
  sasl: {
    mechanism: "scram-sha-512",
    username: process.env.KAFKA_USER,
    password: process.env.KAFKA_PASS,
  },
  retry: { retries: 8, initialRetryTime: 300 },
});

const producer = kafka.producer({
  idempotent: true,              // no duplicates on internal retries
  maxInFlightRequests: 5,
  createPartitioner: Partitioners.DefaultPartitioner,
});

await producer.connect();

export async function publishOrderEvent(event) {
  await producer.send({
    topic: "order.events",
    acks: -1,                    // wait for all in-sync replicas
    compression: CompressionTypes.GZIP,
    messages: [
      {
        key: event.orderId,      // ← this is what guarantees ordering
        value: JSON.stringify(event),
        headers: {
          eventType: event.type,
          version: String(event.version),
        },
      },
    ],
  });
}

Three lines do the heavy lifting:

• idempotent: true — when the producer retries after a network blip, the broker recognises the duplicate and drops it. Without this, retries create duplicate events.
• acks: -1 — the send only succeeds once every in-sync replica has the message. acks: 1 is faster and loses data when a leader dies mid-write.
• key: event.orderId — ordering, as above. Omit it and Kafka round-robins your events across partitions.

The consumer

// consumers/notification.consumer.js
const consumer = kafka.consumer({
  groupId: "notification-service",  // each service = its own group
  sessionTimeout: 30000,
  heartbeatInterval: 3000,
});

await consumer.connect();
await consumer.subscribe({ topic: "order.events", fromBeginning: false });

await consumer.run({
  autoCommit: false,                // commit only after real success
  eachMessage: async ({ topic, partition, message, heartbeat }) => {
    const event = JSON.parse(message.value.toString());
    const nextOffset = (BigInt(message.offset) + 1n).toString();

    try {
      if (await alreadyProcessed(event.eventId)) return;  // dedupe
      await sendNotification(event);
      await markProcessed(event.eventId);
    } catch (err) {
      await sendToDLQ(event, err);  // park it, don't block the partition
    }

    await consumer.commitOffsets([{ topic, partition, offset: nextOffset }]);
    await heartbeat();              // long jobs: prove you're still alive
  },
});

process.on("SIGTERM", async () => {
  await consumer.disconnect();      // commit in-flight work, leave the group cleanly
  process.exit(0);
});

Two details worth pausing on. You commit the next offset to read, not the one you just processed — hence the + 1. And offsets are int64, which is why KafkaJS hands them to you as strings; BigInt keeps that safe once your topics get busy.

And here's the part that made Arjun finally get it:

Notifications, analytics and fraud each ran the same subscribe call — same topic, different groupId.

All three received every single event, independently, at their own pace. Analytics could be an hour behind and notifications wouldn't care. Redis could never do this. That was the whole point.

Chapter 7: The Four Bugs Nobody Warns You About

Kafka went live. Then Kafka started teaching.

Bug 1 — Duplicates came back anyway

"Meera, maine idempotent producer lagaya tha. Phir bhi ek user ko do SMS gaye!"

Producer idempotence only prevents duplicates on the write side. On the read side Kafka is at-least-once: if a consumer processes a message and dies before committing the offset, the next consumer reads it again.

The fix isn't in Kafka — it's in your consumer. Give every event a unique eventId, store processed IDs behind a unique constraint, and skip anything you've seen. That's the alreadyProcessed() call above.

Assume every message will arrive twice. Design for it.

Bug 2 — The rebalancing death spiral

One notification took 45 seconds (a slow third-party SMS gateway). sessionTimeout was 30 seconds. Kafka assumed the consumer was dead, evicted it, and triggered a rebalance. The rebalance paused every other consumer. Those consumers then timed out too.

Three fixes, in order of preference: call heartbeat() during long work, move the slow call off the consumer path entirely, or raise sessionTimeout as a last resort.

A consumer that goes quiet is a consumer Kafka will evict.

Bug 3 — One bad message froze an entire partition

A malformed payload threw on every retry. Kafka doesn't skip it — it can't, because skipping would break ordering. So the consumer retried forever and every message behind it waited.

That's what order.events.dlq is for: catch, park the poison message in a dead-letter topic with the error attached, commit the offset, move on. Fix it offline.

Without a DLQ, one bad record can stall a partition indefinitely.

Bug 4 — Consumer lag was invisible until it wasn't

Meera added lag monitoring on day three:

kafka-consumer-groups.sh --bootstrap-server $BROKER \
  --describe --group notification-service

The LAG column is the single most important number in a Kafka system — it tells you how far behind reality your consumers are. They alerted at lag > 10,000 and scaled consumers (never beyond the partition count) when it fired.

And the security bits Arjun almost shipped without

• The brokers were exposed with PLAINTEXT initially. They moved to SASL/SCRAM over TLS — the config in the producer above.
• Every service got its own credentials and ACLs. The notification service can read order.events and nothing else; it has no business writing to payment.events.
• No raw PII in event payloads. Kafka retains data for days and every consumer group reads everything. Send userId and let the consumer look up the phone number. A topic is not a place to store card numbers.

Chapter 8: Two Months Later

Arjun opened the dashboard on a Monday morning. Orders had crossed 90,000/day.

Nothing was on fire.

The moment that convinced the CTO wasn't the throughput number.

Analytics had a bug for six days — a wrong tax calculation on every order. Under Redis, that data was consumed and gone; six days of reporting would have been permanently wrong. With Kafka, Meera reset the analytics consumer group's offset to a week earlier and reprocessed everything in forty minutes.

"Kafka isn't just faster," she said. "It gave you a time machine for your data."

But please don't copy this blindly

Kafka is the wrong answer for a lot of teams, and Arjun is very clear about that now:

• Under ~10,000 events/day? Redis, BullMQ or a database-backed queue will serve you better at a fraction of the operational effort.
• Need per-message delay, priority or scheduling? That's a job queue's job. Kafka has no concept of "run this one first" or "run this in 10 minutes."
• Nobody to own it? Kafka isn't a library you install, it's infrastructure someone maintains — partitions, retention, rebalances, lag, upgrades. (Modern Kafka runs on KRaft, so at least ZooKeeper is no longer part of the deal.)
• One consumer and no replay requirement? Then you're paying Kafka's complexity tax for features you don't use.

Kafka solved SwiftKart's problem because SwiftKart had exactly the problem Kafka is built for: high volume, multiple independent consumers, and a hard need for durability and replay.

If that's not your problem, it's not your tool.

If you're running Kafka in production, I'd love to hear how you sized your partitions — it's the one decision that's genuinely hard to undo.