Kafka, Inside-Out

Ask anyone what Kafka promises and you get the same sentence: ordering is guaranteed within a partition. It is in the documentation, it is in every tutorial, and it is true.

You already have the parts. Queue Models: Broker vs Log gave you the offset and the difference between competing for a message and reading your own copy. The Kafka codelab had you create a partitioned topic, watch keys pick partitions, run a consumer group and rewind. Delivery Semantics settled at-most, at-least and exactly-once. Assume all of it.

This lesson is about the four words nobody finishes the sentence with: within a partition — and the partition is not a stable place.

When you send a record with a key, Kafka does not look up where that key went last time. There is no directory. It computes:

Read the right-hand end of that expression. The partition a key lands in depends on the number of partitions the topic has at the moment you send it. Nothing about the key changed; nothing about the record changed. Change the divisor and you change the answer.

And the number of partitions is the one knob everybody eventually turns, because it is the only way to add consumers.

Sixty keys — user-1 through user-60 — produced to a topic with three partitions, so every key has a history. Then the topic is grown to six partitions, exactly as a team would do it on a Tuesday afternoon to keep up with load. The same sixty keys are produced again.

30 of the 60 keys — exactly half — now land in a different partition from the one holding their own earlier events.

Look at what that means for user-3. Everything that account did before Tuesday is in partition 2. Everything it does after is in partition 5. Those are two independent logs with no ordering relationship whatsoever — not a weak one, none. A consumer reading both cannot tell you which came first, and no amount of care on the consumer side can recover it, because the information was never recorded.

Nothing failed. No alert fired. The producer kept returning success, the lag stayed at zero, and the per-key ordering the whole design rested on quietly stopped being true for half the keys.

Kafka is entirely upfront about this. From its own documentation:

"this partitioning will potentially be shuffled by adding partitions but Kafka will not attempt to automatically redistribute data in any way"

and the remedy it offers is not a repair — it is a rebuild: "you will have to manually copy data from the old low partition topic to a new higher partition topic."

The obvious instinct is to undo it. You cannot.

The partition count is a ratchet. It goes up, it never comes down, and going up is the operation that breaks per-key ordering for every key already in the topic. That is a genuinely unusual property for a number in a config file, and it is why the partition count deserves the same care as a shard key rather than the casual treatment it usually gets.

It also explains why the standard advice is to over-provision partitions at creation. That advice is real, and this is the reason for it — but it is not free either, which is the next measurement.

Nobody adds partitions for fun. They add them because a consumer group cannot keep up, and the partition count is the only lever that exists.

Five consumers in one group, on a topic with three partitions. This is the broker's own view, not the clients' opinion of themselves:

Two of the five consumers were assigned nothing at all. A partition is never shared between members of a group, so the partition count is a hard ceiling on how far that group can scale. Adding machines past it changes nothing; they join, they are given no work, and they sit there heartbeating.

So the two facts sit on top of each other, and this is the trap the lesson exists for:

The only lever that raises your consumer ceiling is the same lever that permanently breaks per-key ordering.

Which is why the answer to we need more throughput is never just add partitions. It is add partitions, and here is what happens to every key that has a history.

This runs Kafka's actual partitioner — the same murmur2 arithmetic the broker runs — so the keys move exactly where a real cluster would put them. Type your own keys if you like.

Three things worth doing on purpose. Run the default 3 to 6 and confirm you get 30 of 60, which is the number measured on the live broker. Then try 3 to 4, or 5 to 7, and watch that there is no safe multiple — doubling is not gentler than any other jump. Then set them equal and watch nothing move at all, which is the only configuration that preserves every key's history: the one where you did not turn the knob.

It is worth being exact about what survives, because the useful version of this knowledge is a precise claim rather than a worry. Three keys, five events each, one topic of four partitions:

Two things in that output are worth pausing on. First, bob and carol share a partition — you do not get one partition per key, you get a hash, and unrelated keys collide constantly. Their events interleave, and that is fine, because they were never related.

Second, the guarantee is real and it is narrow: per key, per partition, for as long as that key keeps hashing to the same partition. Every one of those three conditions is load-bearing, and the third one is the one nobody says out loud.

The partition count is the surface that moves under your feet on purpose. There is a second one that moves on its own, and it decides durability rather than ordering.

acks=all sounds like all replicas. It means all replicas currently in sync — and the in-sync set is a quantity the cluster maintains by itself, shrinking whenever a follower falls behind. The number that actually decides whether your write is safe is min.insync.replicas.

A note on where this number comes from. Everything above was measured on a live broker. This part is documented rather than measured here, and it is worth saying why: the experiment was attempted on a single broker and abandoned. A topic was configured with min.insync.replicas=2 against one replica — the setting applied, confirmed with kafka-configs.sh — and yet both a console producer with acks=all and a client with acks=-1 were accepted. A single-node cluster will not reproduce the refusal, so rather than publish a number that did not happen, here is what Kafka documents:

• Below the threshold with acks=all, the producer raises NotEnoughReplicas or NotEnoughReplicasAfterAppend.
• The recommended pairing is replication factor 3 with min.insync.replicas 2, which leaves exactly one spare.
• unclean.leader.election.enable allows a replica from outside the in-sync set to become leader as a last resort, and Kafka states plainly that it may result in data loss.

The consequence is worth having ready, because it is a favourite interview question and it sounds like a paradox. Set replication factor 3 and min.insync.replicas 3, lose one broker out of three, and the cluster is healthy while the topic is read-only — every durable write refused, nothing actually broken. Set it to 2 and the same failure is a shrug. One number, and it decides whether a single node leaving is an incident.

Reach for it when you need a durable, replayable, ordered-per-key record that several independent consumers read at their own pace — the shape Queue Models: Broker vs Log argued for. Do not when what you actually want is a work queue with per-message acknowledgement and redelivery, when the consumer count needs to scale freely with load, or when strict global ordering matters, because Kafka has never offered that and one partition is not an answer at scale.

The thing that separates a candidate who has read about Kafka from one who has run it is not knowing that ordering is per-partition. Everybody says that. It is knowing that the partition is a function of a number that changes.

And the sentence to keep if you keep only one: Kafka's guarantees are scoped to a partition, and the partition is a function of a number that moves. Ordering, consumer scale and durability are three faces of that one fact — which is why they are all decided long before the first message is produced, in the arguments to the command that creates the topic.