Message Queues: Absorbing the Burst

Your marketplace runs a flash promo: half off, tonight at six. You've spent real money driving people to the app, and it works. At 6:00:00 the order rate jumps roughly twentyfold and holds there for three minutes while everyone races to grab the deal.

Here's the problem. Placing an order isn't cheap work. Behind that one tap the system charges a card, reserves the item, pings the restaurant, and starts hunting for a driver — call it 200 milliseconds of honest effort per order. Your order pipeline was sized for a normal Tuesday, not for twenty times a normal Tuesday. So at 6:00:01 the pipeline saturates. Requests pile up on threads, connections time out, and checkouts start failing — at the exact minute you are paying to send customers to the door.

The instinct is to make the pipeline bigger. But you don't need twenty times the capacity for three minutes a week — you'd be paying for idle machines the other ten thousand minutes. There's a cheaper, sturdier move: stop making checkout wait for the pipeline at all. Put a queue between them. Checkout does one tiny thing — drop the order into the queue and say "order placed!" — and the pipeline pulls orders out and works through them at whatever pace it can sustain. The three-minute spike in arrivals becomes a few minutes of backlog that quietly drains. Nobody's checkout fails. That is a queue absorbing a burst, and it's one of the most reliably useful moves in all of system design.

In this lesson: what a message queue actually buys you, the single equation that governs every queue, why queue depth is really latency in disguise, how to size the thing (and the expensive trap in the obvious answer), what to do when the queue fills up, and when a queue is the wrong tool.

Scope: the two queue models — a broker that deletes on ack versus a replayable log — are the next lesson, Queue Models: Broker vs Log. Broadcasting one event to many listeners is Pub/Sub. The delivery guarantees (at-least-once, exactly-once, dead-letter queues) get their own lesson, Delivery Semantics. Here we stay on the one job every queue shares: absorbing the burst.

Strip a queue down and it's almost embarrassingly simple: a durable list that one side appends to and the other side reads from. The side that writes is the producer (checkout). The side that reads is the consumer (the order pipeline). The magic isn't in the data structure — it's in what the data structure decouples.

Without the queue, checkout and the pipeline are locked together in time. Checkout can only go as fast as the pipeline accepts work, and if the pipeline is slow or briefly down, checkout is slow or briefly down too. They share a fate. With the queue in the middle, that link is cut. Appending to a queue is cheap and nearly constant-time, so the producer can dump a burst in as fast as it arrives. The consumer, meanwhile, reads at its own comfortable rate, ignoring how frantic the producer is. As Microsoft's own pattern catalog puts it, the queue lets the service "handle the messages at its own pace even when concurrent tasks generate a high volume of requests."

That single move buys three things at once:

• Availability. A hiccup in the consumer no longer takes down the producer. If the pipeline stalls for thirty seconds, checkout keeps accepting orders — they just wait in line. The application "can continue to post messages to the queue even when the service isn't available."
• Burst tolerance. A spike in arrivals is absorbed by the buffer instead of slamming into the consumer. The consumer feels a steady stream; the queue eats the jaggedness.
• Cost control. And here's the one that pays the bills: you only need enough consumers to handle the average load, not the peak. The queue holds the difference during spikes. You stop paying for twenty times the machines to survive three minutes a week.

Think of it exactly like the shock absorber on a car. The road throws a sharp bump at the wheel; the shock absorber turns that violent spike into a gentle, survivable motion for the passengers. The bump's energy doesn't vanish — it's spread out over time. A queue does the same thing to a spike in traffic.

Everything a queue does — absorbing, backing up, draining, overflowing — falls out of one tiny relationship. Let the arrival rate (messages coming in per second) be λ-in, and the service rate (messages the consumers can finish per second) be λ-out. The queue depth L changes at the difference between them:

dL/dt = λ-in − λ-out

In plain words: the queue grows at exactly the rate that arrivals outpace service. That's the whole engine. Three cases, and only three:

• λ-in < λ-out — the consumers are faster than the work coming in, so the queue drains toward empty. This is the healthy resting state. A well-run queue sits near zero almost all the time.
• λ-in = λ-out — perfectly balanced; depth holds steady at whatever it is. Real systems never sit here for long, but it's the knife-edge between the other two.
• λ-in > λ-out — arrivals outrun service, so the queue grows. And it keeps growing for as long as that stays true.

Now the crucial distinction, the one that separates people who understand queues from people who just use them. A burst is a temporary spell of λ-in > λ-out — the flash promo, the morning rush, a retry blip. The depth climbs while it lasts, then, the moment arrivals fall back below the service rate, the queue drains back down. That's absorption working exactly as designed. A sustained overload is λ-in > λ-out that doesn't stop — you're structurally taking in more than you can process. Here the depth doesn't hump and recede; it climbs, and climbs, and never comes back. No queue survives this, because a queue is a buffer, not a bucket with infinite room and no bottom.

This is the first thing to burn in: a queue absorbs bursts; it does not create capacity. If your consumers genuinely cannot keep up with your average load, a queue will not save you — it will just change where the failure shows up, from fast errors at the front door to a silently swelling backlog you won't notice until the latency is already ruined.

It's tempting to treat queue depth as a harmless number — so what if ten thousand messages are waiting, they'll get processed eventually. But depth isn't harmless, because depth is delay. Every message sitting in the queue is a customer waiting, and the deeper the line, the longer the wait.

There's a law that makes this exact. Little's Law says the average number of items in a system equals the arrival rate times the average time each item spends there: L = λ · W. Rearrange it for a queue that's draining at throughput λ-out and you get the number that matters:

W = L / λ-out — average wait ≈ queue depth ÷ throughput

Read that off a real dashboard. Say 10,000 messages are backed up and your consumers are clearing 500 per second. The message at the back of the line waits W = 10,000 / 500 = 20 seconds before anyone touches it. Let the depth double to 20,000 at the same throughput and the wait doubles to 40 seconds. Depth and latency are the same fact told two ways.

Two consequences follow, and both are things senior engineers watch like hawks:

• Queue depth is your primary health metric — not CPU. Your consumers can be at 30% CPU, perfectly healthy, while the queue quietly grows and every order silently gets slower. CPU won't warn you; depth will. So will the age of the oldest message, which is the same signal from the other end.
• A growing queue is a latency bomb with a lit fuse. Nothing looks broken — no errors, no crashes — right up until the wait crosses the point where the work is useless. A driver-assignment message that surfaces twenty minutes late is worse than no message; the customer already cancelled. This is why queues carry a time-to-live: better to drop a stale order than to process a pointless one.

So when you look at a queue, don't ask "how many are waiting." Ask "how long is the wait, and is it growing." That question is the whole game.

Reading dL/dt = λ-in − λ-out is one thing. Feeling a burst swell the queue and drain away — or run away — is another. This is a live queue. Producers on the left pour messages into the buffer in the middle; workers on the right pull them out at their own rate.

Try this in order. Start it running and let the queue sit near empty — arrivals and service are balanced. Now hit Fire a burst and watch the depth graph hump up and then melt back to zero: that's absorption, the whole point. Next, drag the producer rate up past what your workers can handle and leave it there — watch the depth climb and keep climbing, and watch the oldest-message wait (straight from Little's Law) blow past anything acceptable. Now rescue it: add workers until the service rate crosses back above arrivals, and watch the backlog drain. Finally, cap the queue and choose what happens when it fills — push back on the producer, drop messages, or let it grow unbounded — and see who pays: the customer who's blocked, the order that's lost, or the latency that runs to the moon.

The number to keep your eye on isn't how many dots are in the queue. It's the wait and whether it's growing. That's the signal a real on-call engineer stares at.

We said the queue lets you provision for the average instead of the peak, and that's the cost win that sells the whole pattern. If your traffic averages 200 orders a second but spikes to 4,000 for three minutes at dinner, you don't buy 4,000-orders-a-second worth of pipeline. You buy something near the average, and the queue holds the overflow during the spike and hands it to the workers over the following minutes. You just turned a hardware problem into a patience problem, and patience is free.

Except it isn't quite free, and the bill is subtle enough that it takes down real systems. Here's the trap. If you size your consumers for exactly the average, then in normal times your service rate equals your arrival rate — surplus zero. That's fine until the day something knocks your capacity down for a while and a backlog builds. Now you need to drain that backlog, and draining requires spare capacity. The math is unforgiving:

drain time = backlog ÷ surplus, where surplus = total service rate − arrival rate

If your surplus is zero, drain time is infinite. You will never catch up. As one capacity-planning writeup put it bluntly: "systems provisioned exactly for steady-state traffic have zero recovery capacity and will never drain a backlog without intervention." Every consumer is green, nothing is broken, and the backlog just sits there forever because you have no headroom to eat into it.

Watch how fast this gets ugly. A service takes 10,000 messages a second, handled by 25 workers doing 400 each — capacity exactly 10,000, surplus zero. An incident kills 15 of those workers for ten minutes. During those ten minutes you fall behind by 6,000 a second, so when the workers come back you're staring at 6,000 × 600 = 3.6 million messages of backlog. Capacity is restored to 10,000, arrivals are 10,000, surplus is zero — and that 3.6 million never drains. An incident that lasted ten minutes has become permanent, not because anything is still broken, but because there's no room to recover.

The fix is the real lesson: size consumers for the average, but keep headroom to drain — and make that headroom automatic. Add workers on the queue backlog itself, using the metric AWS recommends for exactly this: backlog per instance. Actual backlog-per-instance is queue depth ÷ number of consumers. Your acceptable backlog-per-instance is the latency you'll tolerate ÷ the time to process one message — if a message takes 0.1 seconds and you'll accept 10 seconds of wait, that's 10 / 0.1 = 100 messages per instance. When actual exceeds acceptable, scale out until it doesn't. That's the competing-consumers pattern: many workers racing on one queue, spun up and down by how deep the line is. Queue depth, not CPU, is the throttle.

A queue is a buffer, and every buffer has a bottom. Sooner or later — a long enough burst, a slow enough consumer — the queue fills. What happens next is a decision you have to make on purpose, because the default is usually the worst option.

There are really only two honest answers when arrivals won't stop and the buffer is full, and they trade against each other:

• Push back (backpressure). Refuse the enqueue and make the producer wait. No data is lost, but the slowness now travels upstream — the producer stalls, and if the producer is a user's checkout, they see a spinner. You've chosen to spread the pain back to the source rather than pretend it isn't there. Telling producers to slow down is what true backpressure means.
• Throw away (load shedding). Drop messages — ideally the low-priority ones first — to keep the system alive. You've chosen survival over completeness: the service stays up, but some work is simply gone. For a like-count update that's fine; for a payment it is not.

The one option that feels like a third answer but isn't is "just let the queue grow." An unbounded queue doesn't avoid the problem — it hides it. It quietly absorbs the backpressure instead of enforcing it, swelling in the dark while latency and freshness rot, until you discover the mess long after your SLA is blown. A bounded queue that loudly rejects is almost always healthier than an unbounded one that silently bloats. So bound your queues, and decide in advance whether a full queue pushes back or sheds — don't let the memory limit decide for you at 3 a.m.

There's a nasty interaction worth flagging, because it connects straight back to the previous lesson. When a downstream is struggling and callers retry, every retry is another arrival — so retries inflate λ-in at the worst possible moment, and a durable queue faithfully stores every one of them. In one real incident a payment service was down for about eight minutes, which piled up roughly 200,000 messages; even after every consumer was healthy again, the queue kept growing for another forty minutes as the retry storm played out, turning an eight-minute outage into nearly an hour of pain. A queue plus careless retries doesn't just fail to help — it can stretch a short outage into a long one.

A queue is a genuinely powerful tool, and like every powerful tool it charges rent. Before you reach for one, be honest about what you're paying:

• Latency. The work happens later, so the result isn't ready when you return. Anything the caller needs an answer to right now — did my payment go through? show me the page I just saved — is a bad fit. Queues are ill-suited to real-time, low-latency request/response.
• Eventual consistency. For a stretch of time the order exists but hasn't been processed. Your system is correct eventually, not immediately, and the rest of your design has to be okay with that window.
• Operational weight. You've just added a broker to run, messages to persist, acknowledgements to track, retries to tune, dead-letter queues to watch, and a new dashboard to stare at. Those are real moving parts that need real care.
• Harder debugging. A request used to be a stack trace. Now it's a distributed trace that hops producer → broker → consumer, and "where did my order go" is a genuine investigation.

So the rule is a shape, not a slogan. Reach for a queue when the work can happen a little later and you're buying real burst-tolerance, decoupling, or cost savings — order fulfillment, sending emails and push notifications, encoding video, generating reports, anything write-heavy and spiky and tolerant of a short delay. Skip it when the caller must have the answer now, when the work is trivially fast, or when the traffic is low and steady — there, the broker and its whole entourage cost you more than the coupling ever did. As one well-worn blog title warns, you probably don't need a message queue as often as you think. Use it where its strengths are the exact thing the problem demands, and nowhere else.

Let's make the sizing trap concrete with the dinner rush. Your order pipeline handles a steady 200 orders per second. Each worker fully processes one order in 200 ms, so a worker clears 5 orders/s, and you run 40 workers — capacity 200/s, sized exactly for the average. Predict before you read on:

1. An availability-zone failure kills half your workers for 5 minutes. How big is the backlog when the zone comes back?
2. Your fleet is back to its original 40 workers. How long until the backlog drains?
3. If you want it drained within 5 minutes, how many workers do you actually need?

Work it with dL/dt and drain-time = backlog ÷ surplus, then check against a tiny script:

Running it prints:

backlog when AZ returns : 30,000 orders
drain with 40 workers  : Infinity (Infinity = never)
workers to drain in 5min: 60  -> drains in 300 s

So: five minutes at half capacity buries you under 30,000 orders. Bring the fleet back to its original 40 and — because you sized for the average — the surplus is zero and that backlog never clears. To actually recover you need 60 workers (your 40 plus 20 of headroom), which drains it in exactly 5 minutes; and by Little's Law the unlucky order at the very back waited 30,000 ÷ 300 = 100 seconds. The lesson in three numbers: sizing for the average is cheap until you have to recover, and recovery is something you must deliberately buy — usually as autoscaling that watches the backlog and adds workers before the wait gets ugly.

A few beliefs about queues feel true and quietly cause outages. Trade them in:

• "A queue makes my system faster." → It doesn't. It makes intake more available and smoother, but each individual request usually gets slower end-to-end, because now it waits in line before anything happens. You trade per-request latency for burst-tolerance and decoupling. That's a good trade — as long as you know you're making it.
• "With a queue I'll never drop or lose work." → Only if the queue is durable and bounded with a plan. An unbounded queue doesn't lose messages, it loses their usefulness — they go stale. A bounded queue must block or drop when full. "Never lose work" is a design you build, not a gift the queue hands you.
• "A queue lets me handle any amount of load." → It absorbs transient bursts. Sustained arrivals above your service rate overflow any queue ever made; you still need enough consumers for the average plus headroom to drain. Buffers buy time, not capacity.
• "Add a queue and the slow downstream stops mattering." → The downstream's throughput is still your ceiling. The queue only decides whether that overload shows up as growing depth (everyone waits) or as errors (some get dropped). It relocates the pain; it doesn't delete it.

• A message queue sits between a producer and a consumer and lets each run at its own pace. That decoupling buys availability, burst-tolerance, and the cost win of sizing for the average instead of the peak.
• Every queue is governed by one equation: dL/dt = λ-in − λ-out. A burst (temporary λ-in > λ-out) is absorbed and drains; a sustained overload grows without bound. A queue absorbs bursts — it does not create capacity.
• Depth is latency. By Little's Law, wait ≈ depth ÷ throughput, so a growing queue is a silent latency bomb. Watch queue depth and oldest-message age, not CPU.
• Sizing for the average is cheap until you must recover: drain time = backlog ÷ surplus, and zero surplus means never. Keep headroom — ideally autoscaling on backlog-per-instance — so a backlog can actually drain.
• A full queue forces a choice: push back (backpressure, slowness travels upstream) or shed (drop work). Bound your queues and decide in advance; an unbounded queue only hides the overload.
• Queues charge rent — latency, eventual consistency, operational weight, harder debugging. Use one when the work can happen later and the burst/decoupling/cost win is real; skip it when the caller needs the answer now.

Next — Queue Models: Broker vs Log: we've treated the queue as one generic buffer, but there are two very different machines behind that idea. A broker like RabbitMQ deletes a message once it's acknowledged; a log like Kafka keeps messages and lets consumers replay from an offset. That one difference reshapes how you fan out, recover, and reprocess — and it's next.