Codelab: Produce & Consume with Kafka

You've read about the log model, offsets, consumer groups, and replay. Now you're going to run them. In the next fifteen minutes you'll stand up a real single-node Kafka broker, create a partitioned topic, produce a handful of order events, and then watch — for real — the exact behaviors the last few lessons described: how a key decides a message's partition and keeps it ordered, how the log keeps everything instead of deleting on read, how two consumer groups read the same stream independently, how you rewind an offset to replay, and how the whole thing survives a hard crash because the log is on disk.

No Docker, no Zookeeper, no cloud account. Modern Kafka (4.x) runs itself in KRaft mode — the broker is its own coordinator — so it's a single process you start from a folder. Everything below was run exactly as written; the output you see is the real output.

This codelab makes concrete: the broker-vs-log model (Queue Models), independent consumers (Pub/Sub), and the delivery mechanics (Delivery Semantics) — the whole of this section, in one terminal.

You need exactly two things: a Java runtime (17 or newer — Kafka runs on the JVM) and a terminal. Check Java, then download and unpack Kafka 4.3.1:

That folder is a complete Kafka: the broker, and a bin/ full of command-line tools (kafka-topics.sh, kafka-console-producer.sh, and friends) that talk to it. Everything from here runs inside this directory.

A Kafka broker stores its data in a log directory, and before first use you format that directory with a unique cluster id — the same way you format a disk. Generate an id, format, then start the server:

When you see Kafka Server started, you have a live broker listening on localhost:9092. That one process is both the broker (it stores messages) and the controller (it manages metadata) — that's what KRaft means, and it's why there's no separate Zookeeper to babysit. Open a second terminal in the same folder for everything below; leave this one running the server.

A topic is a named stream. Its parallelism and ordering come from partitions — each partition is one append-only log. Create an orders topic with three partitions, then describe it:

Three partitions, each with Leader: 1 (our single broker owns all of them) and Replicas: 1 (no copies — this is a one-node toy; in production you'd set --replication-factor 3 so a dead broker loses nothing). Those three partitions are the unit of parallelism you met in the log lesson: at most three consumers in a group can read this topic at once.

Now send six order events. We'll give each a key — the customer — because Kafka routes a message to a partition by hashing its key. Same key, same partition, guaranteed order for that customer:

The producer prints nothing and exits — the six events are now durably on the log. The interesting part is where they went, which the next step reveals.

Read the topic from the start, printing each message's partition and key:

Two things just happened that the log lesson promised. First, --from-beginning replayed the whole topic — the messages were never consumed away; reading doesn't delete them. Second, look at the partitions: every one of alice's and bob's orders landed in Partition 0, carol's in Partition 2, none in Partition 1 — because partition = hash(key) % 3 is deterministic. Within a partition the order is exact (1001 before 1003 before 1006); across partitions there's no global order. That's the ordering guarantee, and its limit, in one screen. (The timeout-ms just tells this console consumer to quit after 4 seconds of silence instead of waiting forever.)

So far we read anonymously. Real consumers join a group, and Kafka tracks each group's position — its offset — for every partition. Read the topic as a group called fulfillment, then ask Kafka where that group is:

This little table is the whole model on one screen. CURRENT-OFFSET is how far fulfillment has read; LOG-END-OFFSET is how many records exist; LAG is the gap — how far behind the group is. Partition 0 has 5 records and the group has read all 5 (offset 5, lag 0); partition 2 has 1; partition 1 is empty. Now the payoff — start a different group and it reads the entire stream on its own, because its offsets are separate:

analytics read all 6, even though fulfillment had already "consumed" them. One log, two independent readers, no copies — exactly the fan-out from the pub/sub lesson, and the reason a log can feed your fulfillment pipeline and your data warehouse at the same time without either noticing the other.

Because a group's position is just a number over data that still exists, you can move it backward. This is replay — the thing a delete-on-ack queue can never do. Rewind fulfillment to the very start and read again:

Every partition's offset went back to 0, and re-consuming streamed all six events again — no re-producing, nothing restored from a backup. This is the exact move you'd make to reprocess after a bug: fix the consumer, reset its offset to just before the bad data, and let the retained log replay history through the corrected code. (--to-earliest rewinds to the start; you can also reset --to-datetime yesterday, or --to-offset a specific number.)

One more promise to check: the log is durable, on disk, not just in memory. Let's prove it the rude way — find the broker and hard-kill it, no graceful shutdown:

The broker died mid-life and came back with the topic and all six events intact — they were written to the log directory on disk, so a crash costs nothing. (You also just spotted __consumer_offsets in the list: that's the internal topic where Kafka itself stores every group's offsets — the log is even used to remember where everyone is reading.) On a single node a lost disk would still lose data, which is exactly why production runs replication factor 3 across brokers — but the log itself is durable, and you just watched it survive.

When you're done, stop the broker and remove its data directory:

You didn't just run commands — you performed the entire section. Map it back:

• --create --partitions 3 — a topic is a named log split into partitions; partitions are the unit of parallelism and the boundary of ordering. (Queue Models)
• keyed produce → Partition: 0 for alice — hash(key) % partitions places a message, so a key pins all its events to one partition and keeps them ordered. (Queue Models)
• --from-beginning re-reads everything — a log retains records; consuming doesn't delete them. (Queue Models)
• --describe offsets and lag — each group tracks its own position; lag is how far behind it is. This is the number you alarm on in production. (Message Queues)
• a second group reads all 6 independently — many groups fan out over one log, no copies. (Pub/Sub)
• --reset-offsets --to-earliest then re-read — replay is just moving your cursor back over retained data; it's how you reprocess after a bug. (Queue Models / Delivery Semantics)
• kill -9 then restart, data intact — the log is durable on disk; a crash is survivable, which is what makes at-least-once delivery possible in the first place. (Delivery Semantics)

• Modern Kafka is one process. KRaft mode makes the broker its own controller — no Zookeeper, no Docker, just kafka-storage format then kafka-server-start.
• A topic is a partitioned, replayable log. Partitions give parallelism and per-key ordering; offsets give each consumer group its own independent position; retention lets you rewind and replay.
• The CLI is the real thing. kafka-topics, kafka-console-producer/consumer, and kafka-consumer-groups are the same tools you'll use to inspect a production cluster — reading offsets and lag with --describe and replaying with --reset-offsets are day-job skills, not toys.
• Everything you read in this section is observable in a terminal. Broker vs log, fan-out to groups, replay, durability — you can always drop down to the CLI and watch the guarantee instead of trusting it.

That's the end of Real-time & Async Communication. Next comes the section checkpoint: a few product features where you'll choose the communication style — poll, stream, webhook, queue, log, or pub/sub — and the delivery guarantee, and defend the call.