Queue Models: Broker (RabbitMQ) vs Log (Kafka)

It's Monday morning and the loyalty team has bad news: a bug shipped last week has been double-counting points on every order. Seven days of customers now have the wrong balance. The fix is obvious in principle — take the last week of order events and reprocess them correctly. The question is whether you can.

If those order events flowed through a work queue — the RabbitMQ kind — the answer is no. Each event was handed to a consumer, processed, acknowledged, and then deleted. That's how the queue is supposed to work: once someone's done with a message, it's gone to make room. The events you need to replay don't exist anymore. You're now reconstructing history from database snapshots and hoping.

If those same events flowed through a log — the Kafka kind — the answer is yes, and it's almost boring. The log kept every record for its retention window, say thirty days. You reset the loyalty service's read position back seven days, and the same events stream through again, in order, and you recompute. Every other consumer — fulfillment, analytics, fraud — never notices, because they have their own read positions.

Same events. Two completely different fates. One model crossed messages off a list; the other kept the tape. That difference — delete-on-ack versus replayable offsets — is what this lesson is about, and it decides far more than whether you can fix a bug on a Monday.

In this lesson: the two shapes a queue can take, why one can rewind and the other can't, how each spreads work across consumers, the one design choice (smart broker vs dumb broker) that explains every other difference, and how to pick.

Scope: broadcasting one event to many independent listeners — the pub/sub pattern — is the next lesson, Pub/Sub: Broadcasting Events. The delivery guarantees (at-least-once, exactly-once, dead-letter queues) are Delivery Semantics. Here we stay on the two underlying models and what each one is good at.

Both a broker and a log sit between a producer and a consumer and let them run at their own pace — that part you already know from the last lesson. The difference is what happens to a message after it's delivered, and it splits messaging into two families.

A broker is a to-do list you cross items off. A producer puts a message on a queue; the broker holds it, hands it to a consumer, and the moment the consumer acknowledges it, the broker deletes it. The queue is always trying to reach empty — an empty queue means all the work is done. Messages are transient by design; they exist to be processed once and then forgotten. RabbitMQ is the classic example. This is exactly what you want for work: send this email, resize this image, charge this card. Once it's done, why keep it?

A log is a ledger you read at your own position. A producer appends a record to the end of an ordered, append-only log — like adding a line to a file that only ever grows. Reading a record does not remove it. The record sits there, at a fixed numbered position called an offset, until a retention policy (say, seven days) eventually ages it out. Consumers don't drain the log; they walk along it, each keeping track of how far they've read. Kafka is the classic example. This is what you want for a stream of facts you might need to read more than once, by more than one reader, possibly starting from the past.

That single design decision — delete on acknowledgement versus keep-and-track-a-position — is the root of every other difference between the two. Get this picture right and the rest of the lesson is just consequences.

Broker: “process it, then it's gone.” Log: “it stays; you remember where you are.”

Here's a small detail with enormous consequences: who keeps track of your position.

With a broker, the broker does. It knows which messages it has handed out and which have been acknowledged, and it deletes them as they're confirmed. You, the application, never think about position — you just process what you're given. Convenient. But it also means you can never go back, because there's nothing to go back to; the messages are deleted.

With a log, you do. Your position is a single integer — the offset — pointing at the next record you haven't read yet. Kafka stores that offset for you (the “last one I finished, plus one”), but the crucial thing is that the record itself is still on the log. And if you own a cursor over data that still exists, you can move the cursor backward. Reset the offset to zero and you reprocess everything from the beginning of the retained window. Reset it to yesterday's position and you replay the last day. That's replay, and it falls out for free from “keep the data, track a position.”

Replay is not a party trick. It's how you reprocess after a bug (our Monday morning), how a brand-new consumer catches up on history it was never around for, how you backfill a new database from the stream, and how audit and debugging work when you can re-run the exact sequence of events. None of it is possible once a broker has acked-and-deleted the message.

Retention is the catch, and the honest limit: a log doesn't keep everything forever. Records age out by time or size — Kafka's default is 7 days. You can only replay as far back as your retention window reaches. (There's a clever variant, log compaction, that keeps at least the latest record for each key and drops older versions — turning the log into a compact “current state per key” table. That trick is the seed of change-data-capture and event sourcing, which you'll meet later; for now, just know retention isn't always “delete the oldest.”)

Now the part people get wrong in interviews. What happens when you add a second consumer?

On a broker, consumers on the same queue compete. Put two workers on one queue and the broker hands each message to exactly one of them — the work is split, which is how you scale a task queue. Simple and effective. The catch: because messages are being pulled by whichever worker is free, two related messages can end up processed out of order. And if a different team wants those same messages — say fraud detection wants the order events that fulfillment already consumes — they can't just also listen, because fulfillment's workers would steal the messages. You have to set up a fan-out that copies each message into a second queue.

On a log, it depends on the group — and this is the elegant part. Consumers are organized into consumer groups. Within a group, the log's partitions are divided up so each partition is read by exactly one consumer in that group — that's competing consumers, giving you parallelism and keeping per-partition order. Across groups, every group reads the entire stream independently, each with its own offset. So fulfillment is one group, fraud is another, analytics is a third, and all three read every order event without stepping on each other — no copies, just three cursors over one log. Adding a new reader is free.

But there's a ceiling that trips everyone up. In a group, parallelism is capped by the number of partitions. A partition is read by only one consumer in the group, so if a topic has 4 partitions and you start 6 consumers in one group, two of them sit idle — there's no partition left to give them. “Just add more consumers” does nothing once you've hit the partition count; to go faster you have to repartition, which isn't free. A broker's competing consumers have no such limit (add as many as you like), but they'll trade away ordering to get it. Parallelism on a log is something you plan for up front, by choosing the partition count.

The difference between “deleted” and “retained” is easy to nod along to and surprisingly hard to feel until you watch it. Here is one stream of events fed into both models at once.

Try this. Produce a handful of events — they appear in both the broker's queue and the log's tape. Now, on the broker, deliver a few: each acknowledged message is crossed off and drops into the “gone” bin, and the queue shrinks toward empty. Try to replay — you can't; there's nothing left. On the log, notice the records never leave. Advance a consumer group's cursor along the tape, then hit replay to rewind it to the start and watch it re-read the exact same records. Add a second consumer group and see it read the whole history from offset 0, completely independent of the first. Finally, add consumers to a group past the partition count and watch the extras go idle — the ceiling from the last section, made real.

The thing to internalize: the broker forgets, the log remembers. Everything else follows from that.

Under all of this is one philosophical choice about where the intelligence lives, and once you see it, every other difference clicks into place.

RabbitMQ is a smart broker with a dumb consumer. The broker does the thinking. It decides routing (which queue a message belongs in, via exchanges that can match on topic or content), it tracks per-message delivery and acknowledgements, it handles retries and dead-lettering, it enforces priorities and time-to-live. It pushes messages to consumers the instant they arrive, which keeps per-message latency very low. The consumer's job is simple: process what you're handed. If your problem is “route these messages by complex rules, some are more urgent than others, expire the stale ones, and answer me fast,” a smart broker was built for you.

Kafka is a dumb broker with a smart consumer. The broker is deliberately, almost aggressively simple: it appends records to a log and serves ranges of bytes. It does not route by content, it has no per-message priority, it does not track what each consumer has seen. All of that intelligence moves to the consumer, which pulls batches when it's ready and remembers its own offset. Doing less is exactly why the broker can be so fast and so replayable — there's almost nothing between the producer and the disk. The cost is latency (pull adds a little) and that you own more of the logic.

That one axis — smart broker versus smart consumer — predicts the rest. Push vs pull. Rich routing vs a partition key. Per-message priority vs strict per-partition order. Modest throughput vs enormous throughput. It's not that one team was cleverer than the other; they optimized for opposite jobs.

So which do you reach for? The honest tagline holds up better than most: Kafka for streams, RabbitMQ for queues — a broker distributes work, a log remembers a stream of facts. In practice, pick by what your problem actually needs.

Reach for a broker (RabbitMQ) when you need:

• Complex, broker-side routing — send messages to different consumers by topic or content, without baking the logic into every consumer.
• Per-message priority or TTL — this one is urgent; that one should expire if not handled in 30 seconds.
• Request/reply (RPC) or the lowest per-message latency — push delivery answers fast.
• A straightforward task/work queue — do this job once, then forget it. Operationally it's simpler.

Reach for a log (Kafka) when you need:

• Replay or event sourcing — reprocess history, recover from bugs, rebuild state from the stream.
• Many independent consumers of the same data — separate teams and systems each reading every event, no copies.
• Very high throughput and long retention — a million messages a second is a different order of magnitude from a broker's tens of thousands, and you can keep data for weeks.
• Stream processing or a feed into analytics / a data lake — the log is the backbone of a data platform.

The numbers make the shape concrete: a broker like RabbitMQ handles on the order of tens of thousands of messages per second and shines on routing and latency; a log like Kafka pushes toward a million per second and shines on throughput, retention, and replay. Neither is “better.” They're tuned for opposite problems, and a lot of real systems run both — a broker for the intricate task routing, a log for the event backbone.

One honest complication before you file this away as “RabbitMQ deletes, Kafka replays.” The two products have been quietly borrowing each other's best ideas.

RabbitMQ now offers Streams: an append-only, replayable log type living right inside RabbitMQ, with non-destructive reads so several consumers can re-read the same messages from any position — a log, under the RabbitMQ roof. And Kafka is growing queue-like semantics (share groups, per-message acknowledgement) so that a partition's messages can be split across more consumers than there are partitions — chipping away at the very ceiling we just described.

So the neat product boundary is softening. But notice what isn't changing: you are still choosing between two models — a transient work-distribution queue versus a durable, replayable stream. That decision is about your problem's shape (do you need to forget, or to remember?), and it comes first. The product is the second question, and increasingly either vendor can be bent toward either model. Reason about the model; treat the brand as an implementation detail.

This topic is a minefield of half-truths. Trade these in:

• “Kafka is just a faster message queue.” → It's a log, not a faster queue. Treat it as a drop-in work queue and you'll hit the partition-parallelism ceiling and miss per-message routing, priority, and acks. The speed is a consequence of being a dumb append-only log, not the point of it.
• “A log means I can always replay anything.” → Only within the retention window (Kafka defaults to 7 days). Older records age out; compaction keeps the latest per key, not the full history. Replay is bounded by what you chose to keep.
• “More consumers always means more parallelism.” → In a Kafka consumer group, parallelism is capped by the partition count — extra consumers sit idle. On a broker, more competing consumers do parallelize, but they can process related messages out of order. Neither gives you free speed.
• “A broker can replay if I really need it.” → A classic queue can't — acknowledged means deleted means gone. You'd need RabbitMQ Streams or an external store you wrote to on the side. “We'll just replay later” is a promise a plain work queue cannot keep.

• Messaging comes in two shapes. A broker (RabbitMQ) is a to-do list: messages are deleted on acknowledgement, the queue drains toward empty, and work is distributed to competing consumers. A log (Kafka) is a tape: records are appended and retained, and consumers read at their own offset.
• The offset is the whole trick. Because you own a position over data that still exists, you can rewind and replay — reprocess after a bug, onboard a new consumer on history, backfill. A broker can't; the messages are gone.
• Consuming differs. Broker consumers compete (split work, can reorder). Log consumers form groups: within a group they split partitions (capped by partition count — extra consumers idle); across groups each reads the whole stream independently.
• One axis explains it all: smart broker, dumb consumer (RabbitMQ — routing, priority, TTL, push, low latency) versus dumb broker, smart consumer (Kafka — a plain append-only log, pull, huge throughput, replay).
• Choose by the problem: queues for work (routing, priority, RPC, simplicity), logs for streams (replay, many readers, throughput, retention). The products are converging, but the model choice comes first.

Next — Pub/Sub: Broadcasting Events: we saw that a log lets many consumer groups read the same stream independently. Generalize that into a first-class pattern — one event, many listeners — and you get publish/subscribe, the backbone of event-driven systems, which is next.