Every Kafka consumer application calls poll(). Very few developers can list what happens inside it.

That gap shows up at the worst time: a consumer that hangs, a group that rebalances in a loop, a commit that keeps failing. Each of those failures lives in a specific step of the flow below. The flow is seven steps. The first is setup, the next four are a handshake, and the last two are the steady state the consumer spends its life in.

Step 1: Create consumer and subscribe

Application code builds a KafkaConsumer with bootstrap.servers and a group.id, then calls subscribe() with the topic list.

subscribe() only registers interest. Nothing is fetched, nothing is assigned, no broker knows the consumer exists yet. Every step that follows is driven by poll() — it is the engine of the entire client, and the handshake below unfolds across its first calls.

Step 2: FindCoordinator

The consumer asks any bootstrap broker a simple question: which broker coordinates this group.

The answer is the broker that leads the __consumer_offsets partition assigned to the group. In this walkthrough that is Broker 3, coordinating the group payments-group. From this point, all group traffic — joins, heartbeats, offset commits — goes to Broker 3. None of it goes to the brokers holding the actual data.

Step 3: JoinGroup

The consumer sends a JoinGroup request carrying its protocol preferences and timeouts:

group.id=payments-group
session.timeout.ms=45000
rebalance.timeout.ms=300000

assignment protocols offered: range, cooperative-sticky

The coordinator starts a rebalance and elects a group leader from the members. It issues a generation number — generation 17 in this walkthrough — plus a member ID for each consumer. The generation matters later: it is how the coordinator detects members acting on a stale view of the group.

Step 4: SyncGroup

The elected leader computes the partition assignment for every member and sends it back through the coordinator. SyncGroup returns this consumer's share: partitions payments-0 and payments-1.

Only now does the consumer know what it owns. Everything before this step was negotiation.

Step 5: OffsetFetch

The consumer asks the coordinator — not the partition leaders — where the group left off. The committed offsets come back: payments-0 → 100, payments-1 → 250.

Position starts from the committed offset. If a partition has no committed offset at all, auto.offset.reset decides where to start. That one configuration quietly determines whether a brand-new group reads a topic from the beginning or only sees new records.

Step 6: Fetch

Fetch requests go to each partition's leader broker — this is the data path, separate from the coordinator path. payments-0 is fetched from Broker 1, payments-1 from Broker 2.

Each fetch starts at the consumer's position, and record batches stream back per poll(). Fetch is a long poll: the broker holds the request, waiting up to fetch.max.wait.ms to accumulate at least fetch.min.bytes of data. An idle topic does not mean a stuck consumer. It usually means the broker is patiently waiting to send a batch worth sending.

Step 7: Process and commit

Application code processes the records, then commits. The commit carries the next offset: after processing through offset 109 on payments-0, the committed offset is 110. The coordinator writes it into __consumer_offsets, and that write is the group's durable progress.

A commit can fail on a stale generation. That failure means a rebalance happened mid-processing — the correct response is to rejoin and retry, not to hammer the commit. After a successful commit, the flow loops back to Fetch. Steps 6 and 7 repeat for the life of the consumer. The handshake in steps 2 through 5 only reruns when a rebalance forces it.

Under the hood

The Java client splits this flow across a few key classes:

• KafkaConsumer — the API application code touches.
• ConsumerCoordinator — group management: FindCoordinator, JoinGroup, SyncGroup, OffsetFetch, OffsetCommit, and heartbeats.
• Fetcher — records from the partition leaders.
• SubscriptionState — the current assignment and position.
• NetworkClient — everything on the wire.

The split explains the two network paths. Group management runs through the coordinator, Broker 3. The data path runs to the leaders, Broker 1 and Broker 2. Committed progress at the end of this walkthrough: payments-0 → 110, payments-1 → 260, stored in __consumer_offsets where the next generation of the group will find it.

Rule of thumb

When a consumer hangs, the useful move is naming the step that broke: the handshake (steps 2–5), the fetch (step 6), or the commit (step 7).

It is almost never the process() code.

Takeaways

• subscribe() registers interest and nothing more. poll() drives every step that follows.
• Group traffic and data traffic take different paths: coordination to one broker, fetches to the partition leaders.
• A rebalance is the handshake rerunning. Generation numbers are how stale members get caught.
• A commit stores the next offset to read, and committed offsets in __consumer_offsets are the group's only durable memory.
• Consumer problems debug fastest by step: handshake, fetch, or commit.