Pub/Sub: Broadcasting Events

An order gets placed on your marketplace. That single fact matters to a surprising number of people. Email needs to send a receipt. Inventory needs to decrement stock. Analytics wants to count it. Fraud wants to score it. Recommendations wants to learn from it. Loyalty wants to award points. Six teams, six things that must happen — from one click.

The obvious way is for checkout to call all six. And it works, right up until it doesn't scale as a design. Checkout now has to know about all six services, make six calls, and handle six ways they can fail. Worse, the list keeps growing: next week the growth team wants to trigger a "first order" campaign, so someone opens the checkout code — the most critical, most carefully-guarded code in the company — and adds a seventh call. Every time anyone anywhere wants to react to an order, they have to modify checkout. The busiest code in your system has become a switchboard that everyone has to rewire.

There's a better shape. What if checkout just announced what happened — "an order was placed" — and didn't care who was listening? Each interested team raises its hand once and reacts on its own. Checkout publishes a single event and moves on; the growth team's seventh reaction is something they add on their side, without touching checkout at all. That's publish/subscribe, and it's how event-driven systems stay sane as they grow.

In this lesson: how one published event fans out to many independent subscribers, the topic-and-subscription model that unifies "broadcast" and "load-balance," what the fan-out costs, whether a subscriber that's offline actually gets the message, and when broadcasting is the wrong idea.

Scope: the delivery guarantees themselves — at-least-once, exactly-once, dead-letter queues — are the next lesson, Delivery Semantics. Here we focus on the pattern: one event, many listeners.

The move is to put a broker between the publisher and the consumers and change what gets sent. Instead of checkout calling services, checkout publishes an event to a named channel — a topic — like order.placed. It has no idea who, if anyone, is on the other end. Separately, each interested service subscribes to that topic. When an event arrives, the messaging system delivers a copy to every subscriber. One publish becomes many deliveries. That multiplication — one to many — is the fan-out, and it's the whole idea.

Notice what just happened to the coupling. In the last lessons we decoupled producers and consumers in time — a queue let them run at different speeds. Pub/sub adds a second, deeper kind: decoupling in space. The publisher doesn't know the subscribers exist. It doesn't know how many there are, what they're called, or whether any of them are even running. It announces a fact to the void and trusts the broker to deliver it. This is the loosest coupling any messaging pattern offers, and it's what makes the seventh consumer free.

That's the superpower, stated plainly: you extend the system by adding subscribers, not by editing the publisher. The order.placed event is published in exactly one place. Want a new reaction to it — a loyalty campaign, a data-warehouse feed, a Slack alert? You stand up a new subscriber and point it at the topic. The publisher, and everyone else already subscribed, carries on untouched. A system built this way is open for extension — new behavior is new subscribers — which is the heart of event-driven architecture.

People often frame this as "queue versus topic," as if they're opposites. They're really two settings of the same machine, and once you see the machine you'll never be confused about which you're using. Three nouns:

• A topic is the named channel a publisher sends to. It knows nothing about consumers.
• A subscription is an independent buffer attached to a topic. Each subscription gets its own copy of every event on the topic.
• A subscriber is a client reading from a subscription.

Now the two knobs:

1. How many subscriptions does the topic have? Each subscription is an independent reader that gets all the events. Two subscriptions — say billing and shipping — each see every order, on their own. More subscriptions = wider broadcast. This is the pub/sub, one-to-many fan-out.
2. How many subscribers share one subscription? Put three workers on the billing subscription and the events are split among them — each event goes to only one of the three. More subscribers on one subscription = deeper load-balancing. This is the work-queue, competing-consumers, one-to-one behavior.

So it was never queue versus topic. A single subscription with a single subscriber is a plain queue. A single subscription with many subscribers is a load-balanced worker pool. Many subscriptions is a broadcast. Many subscriptions each with many subscribers is a broadcast to several teams, load-balanced within each team — the shape most real systems actually run. And here's the payoff: this exact model is Amazon SNS fanning out to SQS queues, Kafka's consumer groups, and Google Pub/Sub's subscriptions. One idea, three brand names. (A subscription is essentially a Kafka consumer group from the last lesson.)

One more knob worth knowing: subscribers don't have to take everything on a topic. With filtering, a subscription can say "only events where type = order.paid." Billing takes the paid events, shipping takes the shipped events, both from one order-events topic — and, true to form, you add either without touching the publisher. Coarse routing by topic, fine routing by filter.

Fan-out is one of those ideas that's obvious in a sentence and slippery in practice. Here it is live: a publisher, a topic, and a set of subscribers you control.

Try this. Publish an event and watch the single message fan out along live arrows to every subscriber at once — each one's counter ticks up together. Now add a subscriber and publish again: the fan-out just got wider, and the publisher did nothing differently. Flip the delivery mode from broadcast to point-to-point and publish: now each event goes to only one subscriber, round-robin — that's the work-queue knob. Finally, take a subscriber offline and publish a few events, then bring it back: under ephemeral delivery it simply missed them, while under durable delivery they were buffered and it catches right up.

Keep an eye on the amplification number — deliveries per event. It's just the count of live subscribers, and it's the reason fan-out is both powerful and, at scale, expensive.

Fan-out gives you leverage: one publish, N reactions, for free at the publisher. But that N is doing real work somewhere, and the multiplication that makes pub/sub elegant is the same multiplication that makes it expensive at scale. One event to six internal services is nothing. One event to millions of subscribers is a different animal.

The classic version is the celebrity problem. On a social network, "user posted" is an event, and the subscribers are the followers who should see it in their feed. For someone with 300 followers, fanning that post out to 300 feeds is trivial. For someone with 10 million followers, that single post becomes 10 million writes, all at once — and if they're popular enough that reposts cascade, one action can turn into hundreds of millions of deliveries. The fan-out that was free for normal users becomes a stampede for stars.

This forces a genuine design choice about when you pay the cost:

• Fan out on write: when the event happens, immediately deliver a copy to all N subscribers. Reads are then cheap (everything's already in your feed), but a celebrity's write explodes into millions of deliveries.
• Fan out on read: store the event once, and assemble each subscriber's view when they ask. The celebrity's write is now a single cheap operation, but every read has to merge across everyone you follow.

Real systems go hybrid — fan out on write for ordinary accounts, fall back to fan-out-on-read for the handful of celebrities whose write amplification would otherwise melt the system. You don't need to design a news feed today; you just need the reflex: fan-out cost scales with the subscriber count, and extreme fan-out needs a strategy, not just a topic.

Here's the question that separates a demo from production: what happens to an event for a subscriber that isn't listening at that exact moment — because it crashed, or it's mid-deploy, or it's just slow? The answer depends entirely on the implementation, and getting it wrong quietly loses data.

The naive kind of pub/sub is fire-and-forget. Redis Pub/Sub is the poster child: it delivers each message to whoever is currently connected and then throws it away. No storage, no replay, no memory of who got what. It's blazing fast, and it's perfect for things you don't mind dropping — a live "someone is typing…" indicator, a cache-invalidation ping. But if a subscriber is offline for even a second, the events published in that second are gone, with no way to get them back. The rule is blunt: if you cannot afford to lose messages, do not use fire-and-forget pub/sub.

The robust kind gives every subscriber its own durable buffer — and you've already seen the mechanism. The topic fans out into a per-subscriber queue (Amazon SNS into an SQS queue per subscriber) or a retained log with per-consumer offsets (Kafka). Now a subscriber that's down doesn't miss anything; its messages pile up in its own queue and it works through the backlog when it recovers — exactly the burst-absorbing queue from two lessons ago. This also solves the slow-subscriber problem: because each subscriber has its own buffer, a sluggish analytics consumer can fall behind without slowing down email or inventory. One bad subscriber is isolated to its own lane instead of jamming the whole fan-out.

So "pub/sub" alone doesn't promise delivery — the durability of the subscription does. When someone says "we'll just publish an event," the right follow-up is: and if a subscriber is down when you publish, does it get the event later, or never? That one question tells you whether you have a notification toy or a reliable backbone.

Loose coupling is the whole point of pub/sub, and it's also the source of every one of its problems. The same property — nobody is wired to anybody — cuts both ways:

• "Who's even listening?" Because the publisher names no consumer, no one has the full list of who reacts to an event. Tracing a single order through six decoupled handlers is real detective work, and a message that silently gets dropped has no obvious owner. You buy extensibility with observability — budget for good tracing and monitoring, or the loose coupling will hide your bugs.
• The event shape is a contract you can't see the callers of. Every subscriber depends on the fields in order.placed. Rename one and you might break a consumer you forgot existed — the same schema-evolution problem from the API lessons, except now the "callers" are anonymous. Version your events and add fields rather than changing them.
• Event cascades. A subscriber reacts to an event by publishing more events, which trigger more subscribers… A tidy chain on the whiteboard can become a hard-to-follow web in production, and a storm in one place can ripple outward.
• No ordering, no answer. Pub/sub generally won't guarantee that subscribers see events in the same order, and it's one-way — fire and forget. If you need a response to keep going, or strict global ordering, this is the wrong tool.

So don't reach for a topic when the work is genuinely one job for one worker (that's a plain queue), when you need a reply to continue (that's request/reply), or when you need guaranteed delivery and you're about to use an ephemeral system to get it (use a durable one). Use pub/sub for what it's uniquely good at: announcing that something happened, to an unknown and growing set of things that care.

A few tempting half-truths to trade in:

• "Pub/sub guarantees my event gets delivered." → Only if the subscription is durable. Fire-and-forget pub/sub (Redis) drops anything published while a subscriber is offline. Delivery is a property of the subscription's durability, not of "pub/sub" as a word.
• "A topic is just a fancy queue." → No. A queue hands each message to exactly one consumer (work is split); a topic hands each message to every subscriber (events are broadcast). They're the two knobs of the same model — but confusing them means either duplicated work or dropped events.
• "The publisher knows its events were handled." → It knows nothing. It fires an event and moves on; it can't even tell whether anyone is subscribed, let alone whether they succeeded. If you need to know, you need a different pattern (request/reply, or a status event published back).
• "More subscribers is free." → Free for the publisher; not for the system. Every subscriber multiplies the delivery work. At small N you'll never notice; at celebrity N it's the whole engineering problem.

• Publish/subscribe decouples in space: a publisher announces an event to a topic and never names a consumer; the system delivers a copy to every subscriber. You extend the system by adding subscribers, not by editing the publisher.
• It's not queue versus topic — it's two knobs. Number of subscriptions on a topic sets broadcast breadth (each gets all events); number of subscribers on one subscription sets load-balancing depth (events split among them). SNS→SQS, Kafka consumer groups, and Google Pub/Sub are the same model.
• Fan-out amplification: one publish becomes N deliveries. Free at small N; at celebrity scale it needs a strategy (fan-out on write vs on read, usually hybrid).
• Delivery depends on durability, not on the word "pub/sub." Fire-and-forget (Redis) drops events for anyone offline; a durable per-subscriber queue or log buffers them and lets a slow or crashed subscriber catch up — and isolates one bad subscriber from the rest.
• The cost of loose coupling is visibility: who's listening, event-schema contracts, cascades, and no ordering or reply. Use pub/sub to announce that something happened to an unknown, growing set of listeners — not for point-to-point work or request/reply.

Next — Delivery Semantics: we've been hand-waving "the system delivers a copy" and "it catches up later." But exactly how many times? At-most-once, at-least-once, exactly-once — each is a different promise with a different price, and picking the wrong one means duplicate charges or lost orders. That, plus dead-letter queues, is next.