Most engineers don't think seriously about Kafka partitioning until something breaks in production. A topic that worked fine at low volume starts falling behind. Events that should be in order aren't. All of it traces back to a partitioning decision that was made quickly and never revisited.
Why Partitioning Actually Matters
Partitions are the unit of parallelism in Kafka. Every consumer in a group is assigned one or more partitions, and it processes those partitions alone. No two consumers in the same group share a partition. That means your partition count sets a hard ceiling on how many consumers can work in parallel: if you have 6 partitions, the 7th consumer in your group sits idle no matter how much load you're under.
Partitioning also controls ordering. Within a single partition, events are strictly ordered. Across partitions, there are no guarantees. So how you distribute events across partitions determines what ordering guarantees your consumers can actually rely on. Get this wrong and you'll spend a long time debugging why events from the same user are being processed out of sequence.
The partition key controls both of these things. It determines which partition an event lands in, and that decision has consequences that are expensive to reverse.
Partitioning Strategies
Partition by Key
This is the most common strategy and the right default when ordering matters. You supply a key when producing an event, Kafka hashes it using the murmur2 algorithm, and takes the modulo against the partition count to decide where it lands.
producer.send('orders', key=b'user_4821', value=event)
Every event with the same key always lands in the same partition. That's what guarantees ordering within a key. All events for user_4821 go to partition 3 (or wherever the hash resolves), and your consumer reads them in the exact sequence they were produced. I default to this for almost everything I build now and only go keyless when I have a specific reason to.
Use key-based partitioning when the sequence of events for a specific entity matters: order lifecycle events, user activity streams, account transactions, IoT device readings. Basically any time you need to reconstruct what happened to a thing, in order.
Where this goes wrong is key distribution. If you use country_code as a key and 80% of your traffic comes from one country, 80% of your events land in the same partition. That partition becomes a bottleneck; the consumer handling it is overwhelmed while others sit mostly idle. This is a hot partition, and it's one of the most common Kafka performance problems I see on teams that are new to it. Keys with high cardinality and relatively even distribution are what you want: user_id, order_id, device_id. The reason those work is simple: they have millions of possible values, so traffic spreads naturally across partitions. Keys like status, region, and event_type tend to cause skew because they only have a handful of possible values, which means a handful of partitions carry almost everything.
No Key, No Order
Skip the key and Kafka spreads events evenly across partitions on its own, no configuration needed. Throughput is good, distribution is even, and you've traded away ordering entirely. That's a fair trade for logs, metrics, anything you're aggregating where sequence doesn't matter. It's a bad trade the moment you need to reconstruct a sequence: an order moving from pending to confirmed to shipped needs those three events read in order, and round-robin gives you no such guarantee.
The Default Kafka Doesn't Advertise
This is what Kafka does by default since version 2.4, and almost nobody knows it until they're confused by something in their metrics. Instead of switching partitions with every record, Kafka batches records to the same partition until the batch fills or the linger.ms timeout hits, then moves to the next one. It's more efficient than scattering tiny batches across brokers, and it costs you nothing to configure.
The only reason to care: if you're watching a keyless topic and partition distribution looks lumpy in the short term, that's sticky partitioning, not a sign of skew. Give it time and it evens out.
Rolling Your Own
When the built-in strategies don't fit, you write your own. A custom partitioner takes the key, value, and topic, and returns a partition number directly.
from kafka import KafkaProducer
def custom_partitioner(key, all_partitions, available_partitions):
region = key.decode().split(':')[0] # key format: "region:entity_id"
if region == 'EU':
return all_partitions[0] # partitions 0-2 for EU
elif region == 'US':
return all_partitions[3] # partitions 3-5 for US
return all_partitions[0]
producer = KafkaProducer(
bootstrap_servers='localhost:9092',
partitioner=custom_partitioner
)
The use cases are narrow: geographic routing so region-specific consumers don't have to filter, customer tiering to give enterprise accounts dedicated partitions, compliance rules that pin certain event types to specific partitions for audit purposes. I've built exactly one of these in production, for a compliance requirement nobody was willing to negotiate on, and I wouldn't reach for it again unless forced to. Your routing logic now lives in producer code, every change means a redeploy, and the next engineer debugging a partition imbalance has to learn your custom logic before they can learn anything else. A well-chosen key solves almost everything else.
Hot Partitions: What Key Skew Actually Looks Like
A hot partition is one partition taking significantly more traffic than the rest. The consumer assigned to it is CPU-bound and falling behind; everyone else is comfortable. Latency spikes for whatever's stuck in that partition while the rest of the pipeline looks completely healthy on every dashboard that matters.
I watched a team chase a phantom performance bug for two days because of exactly this. Their aggregate consumer lag sat at a few hundred messages, well within their alert threshold. CPU across the consumer group looked fine. On paper, everything was healthy. But one partition out of twelve was carrying sixty percent of the volume, and the consumer assigned to it was quietly drowning. The key was region_code, and most of their customer base happened to be in one country. Nobody noticed because nobody was looking at lag per partition, only the average. Once we split it out, the problem was obvious in about thirty seconds.
The root cause is nearly always the same: a key whose values don't spread evenly across your traffic. Geography does this constantly. So do status fields, where most events sit in one or two states, and timestamps, which cluster during business hours and go quiet overnight.
The fix depends on how bad the skew is. If it's moderate, adding a random suffix to the key helps: user_id becomes user_id_0, user_id_1, or user_id_2 assigned randomly at write time. This redistributes load but breaks per-key ordering, so your consumer logic needs to handle events from the same user arriving out of sequence. If skew is severe and you need ordering, the options get harder: a custom partitioner that maps high-volume keys to multiple partitions, reordering logic in your consumer that buffers and sequences events before processing them, or rethinking the data model so the skewed key isn't the partition key at all. None of those are fun to retrofit. That's why catching it in pre-production matters.
The best place to catch this is before you go to production. Sample your actual key distribution and simulate what percentage of traffic would land in each partition. If any partition would receive more than 3-4x the average, change the key before you deploy.
Changing Partition Count After the Fact
You can increase the number of partitions on an existing topic. You can't decrease it. And increasing it is less safe than it sounds the moment you're using key-based partitioning.
The partition a key lands in is determined by hash(key) % partition_count. Change the partition count and the mapping changes. A user_id that was routing to partition 3 with 6 partitions now routes to partition 9 with 12 partitions. Any stateful consumer that was building per-user state by reading partition 3 now has an incomplete picture.
I've watched a team burn the better part of a week on exactly this. They went from 6 partitions to 12 on a Friday afternoon to handle a traffic spike, called it a routine scaling change, and went home. By Monday, a stateful aggregation job that built per-user session windows was producing numbers that didn't reconcile with anything, and nobody connected it to the partition change for three more days, because on paper, adding partitions is supposed to be safe. It wasn't, because the moment the partition count changed, every key's hash landed somewhere new, and the consumer's in-memory state for partition 3 was now watching a completely different slice of users.
The safer path when you genuinely need more partitions: create a new topic with the target count, run dual-write from your producer during a migration window, let consumers drain the old topic, cut over, then retire the old topic completely. That last step gets skipped more often than it should. A zombie topic with stale data sitting around is its own category of future confusion. More work upfront, but silent data integrity problems are worse than loud failures, and this approach avoids both.
If your topic is stateless and consumers don't maintain any per-partition state, increasing partition count is lower risk. But write that assumption down somewhere. It's easy to forget, and the next person to touch the pipeline may not know it matters.
Monitoring Partition Health
Aggregate consumer lag hides imbalances by design. It's an average, and averages smooth out exactly the spikes you need to see. If one partition out of six is 200,000 events behind, the aggregate number can still look fine while that partition is functionally broken. Watch lag per partition, not the total, and watch partition size on disk too. Uneven sizes across partitions of the same topic is key skew showing up in a different form, often before lag becomes visible at all.
Rebalance frequency is worth tracking too. A consumer stuck on a hot partition will sometimes start timing out, which triggers rebalances across the whole group. If you're seeing frequent rebalances and can't explain why, check whether one partition is running a consumer ragged. Both lag and size metrics live in Kafka's JMX interface and surface cleanly in Prometheus with the Kafka exporter. Confluent's built-in monitoring shows per-partition lag directly, no extra setup required.
Before You Create the Topic
Get this right before the topic exists, not after. Does ordering matter for this data? If yes, you need a key. What does your actual key distribution look like in production traffic, not the sample data on your laptop? And what's the largest number of consumers you'll realistically want running in parallel against this topic? That last answer is your partition count.
Set it to 2-3x your initial consumer count with room for traffic growth, not just where you are today but where you expect to be in a year. Write down why you chose the number. Treat the decision the way you'd treat a schema change. You can walk it back later if you really have to. You don't want to.
The reason this section exists at the end of an article full of incidents is that every single one of those incidents had a cleaner solution available before the topic was created than after. The region_code hot partition was fixable with a better key upfront. The Friday partition count disaster was avoidable with a higher initial count or a documented assumption about statefulness. The problems aren't hard to prevent. They're just easy to deprioritize when you're moving fast and the topic is one line of setup in a larger ticket. It takes about ten minutes to think through these questions properly. The incidents in this article took days.
Conclusion
I've never seen a team get burned by picking the wrong partitioning strategy. I've seen plenty get burned by picking a key without checking how it actually distributes, or bumping a partition count without asking what depends on the old mapping. Ask the key distribution question before you write the first producer, and you'll skip most of the incidents in this article entirely.
