Capstone Pass 3: The Marketplace Survives

You've built this marketplace twice already. In the first pass you sketched the food-delivery MVP and sized its load; in the second you designed its data layer, the sharded orders, the replicated menus, the geo store tracking every driver. It works. Diners order, restaurants cook, drivers deliver, and on a normal evening it hums.

This pass is about the evenings that aren't normal. Everything you learned across this whole part of the course was building to a single question: can this system survive a bad day? Not a hypothetical one, the real kinds you just studied at Facebook, Amazon, and half the internet. So we're going to run a game day on your marketplace, the same drill real teams use to prove their systems are resilient: we throw genuine failures at it and see whether dinner keeps flowing.

Three failures, and they're not random. A whole region goes dark in the middle of the dinner rush. A retry storm ignites and feeds on itself. A flash crowd hits when a promo goes viral and traffic jumps tenfold. Your marketplace has to keep taking orders through all three, inside the service levels you promised. This is where the reading stops and the doing starts. Design it, then run the drill and see if it holds.

A design without numbers is a wish, so here's what you're defending. At the dinner peak your marketplace takes about 50,000 orders a minute — a little over 800 a second — across two active regions, backed by the sharded database and replicated menus from the last pass. A normal dinner rush is your daily flash crowd; a viral promo or a big-game night is the 5-to-10-times spike you have to plan for.

And here's the promise you've made, written as service-level objectives from SLIs, SLOs & Error Budgets: checkout stays available 99.9% of the time — that's a whole error budget of only about 43 minutes a month to spend on all your bad days combined — and checkout stays fast, under two seconds at the 99th percentile. Those two numbers are the finish line. Every decision in this pass is really the same question asked in different clothes: does this move protect the budget, or spend it?

One more given, the most important one, and it's the trap the whole drill is built around. All three failures look the same on your dashboard — orders falling, errors climbing, latency spiking. Overload is overload. But they are three different animals underneath, and the move that saves you from one will do nothing for another. Reading the symptom is easy. Naming the failure is the skill.

Before you read the reference design, do the drill yourself. The console below is your marketplace, live, with orders flowing and your service level green. It gives you three failures to trigger and a small palette of defenses to reach for. Your one job is the reflex this whole part of the course built toward: match the defense to the failure.

Reach for the obvious lever first, the one every tired engineer reaches for at 3 a.m. — just add more servers — and watch what it does to each failure. Then find the move that actually matches. A lost region doesn't care how many servers you add in it; it's gone, and you need to be somewhere else. A retry storm feeds on load, so adding capacity feeds it; you have to starve the loop. A flash crowd is real demand, so you absorb what you can and gracefully drop what you can't. Get all three right and your marketplace survives its worst night. The sections after this are the worked reference for each move, and a rubric to grade the design you'd actually build.

Here's the worked answer to the drill, one failure at a time. Notice that the naive fix, adding capacity, is wrong for all three — and wrong for a different reason each time.

A region goes dark. More servers in a dead region is nothing times a bigger number. The move is failover to your other region, and the way you learned to do it right in us-east-1: The Region the Internet Leans On: make it statically stable. The standby region is already provisioned to carry full load, and a health check flips traffic to it automatically, on the data plane, with no control-plane call in the hot path — so your failover doesn't depend on the region that's on fire. Size each region for the whole load, replicate across them, and accept a small data-loss window as your recovery point, the trade from Disaster Recovery: RTO & RPO and Multi-Region Patterns.

A retry storm ignites. This is the trap, and it's a real one — the engineers at delivery companies your size will tell you most of their worst outages are exactly this shape. A service slows, callers retry, the retries pile on more load, the service slows more, and the storm feeds itself long after the first hiccup is gone. It's a metastable failure from Anatomy of a Cascading Failure, and adding servers throws them straight into the fire. The move is to break the loop: a retry budget that caps retries at a small fraction of normal traffic and fails fast when it's spent, from Retries, Budgets & Retry Storms; circuit breakers that stop calling a drowning service from Circuit Breakers; and load shedding to drop the retries themselves, from Load Shedding & Graceful Degradation. Once the loop starves, your normal capacity is plenty.

A flash crowd hits. This one looks identical to the retry storm on the dashboard, but it's the opposite problem: the traffic is real, hungry diners who genuinely want to order. You don't break this loop, because there is no loop; you absorb and degrade. Put a queue in front to smooth the spike, from Backpressure; rate-limit per user at the door so no one starves everyone else, from Rate Limiting: Token Bucket, Leaky Bucket, Windows; shed the non-critical so the order and payment path always wins over recommendations and analytics; and serve menus from the edge cache so reads never touch your core, from CDNs: Caching at the Edge. Keep the money path alive and let the frills fall away.

Same symptom, three different animals, three opposite cures. That is the entire judgment this part of the course was built to give you.

Put every one of those moves onto one diagram and you have the marketplace that survives. Diners come in through an edge cache and a rate limiter at the front door — your admission control, so a flood is throttled before it ever reaches your core. A geographic load balancer routes each diner to the nearest of two active regions, and each region is built as cells from Cell-Based Architecture & Deployment Stamps, so a bad deploy or a poison order takes down one slice, not the whole region. Inside, the order, payments, and dispatch services call each other through circuit breakers and retry budgets, with a queue absorbing bursts. The two regions replicate asynchronously, and a health check drives automatic, statically-stable failover between them.

Underneath the happy path sit the things you only notice when they're missing. Backups you have actually restored, from Data Integrity: Backups That Restore, because replication faithfully copies a corruption and only a clean backup undoes a bad write. An out-of-band, break-glass path to operate the system that shares nothing with the system itself — the lesson written in blood by Facebook, 2021: The BGP Self-Lockout — so you're never locked out of your own recovery. And you keep an eye on the quiet, super-linear costs that crossed a hidden ceiling in AWS Kinesis, 2020: The Thread-Limit Cascade.

And wrapped around all of it is the reliability discipline from earlier in this part of the course: an SLO with an error budget that turns is it reliable into arithmetic; alerts that wake a human only for a symptom that's burning the budget; and an incident playbook that says mitigate first, coordinate under one commander, and write a blameless postmortem after, from Incident Response & Postmortems. You even rehearse the whole thing on purpose, with the game days of Chaos Engineering, so the vulnerable state is one you find before it finds you. This isn't a diagram to copy. It's a worked example to argue with while you design your own.

Now grade the design you'd actually build. Go down this list honestly — each line is a yes or no, and a no is a gift, because it's the exact gap to close before a real bad night finds it.

• Region loss: Does your failover run on a health check with no control-plane call in the hot path, and is each region sized to carry the full load alone? If your failover needs the failed region to work, it isn't a failover.
• Retry storm: Do your service-to-service calls have retry budgets, backoff with jitter, and circuit breakers? And can you tell a retry storm from a flash crowd, so you break the loop instead of scaling out when it's metastable?
• Flash crowd: Do you have admission control at the door — a rate limiter and a queue — and graceful degradation that protects the order and payment path while dropping the frills? Are your menu reads served from the edge?
• Data: Do you have backups you have actually restored and a defined recovery point, so a corruption or a bad write is survivable, not just a hardware loss?
• Blast radius: Is your system built as cells so no single failure can take down more than a slice?
• The discipline: Is there an SLO with an error budget, do your alerts fire on symptoms rather than causes, and do you have an incident playbook that mitigates before it diagnoses?
• No self-lockout: Does your recovery path — the way you get in to fix things — share nothing with the system it recovers?

A design that can answer yes to all seven is a marketplace that fails small, fails over, and stays honest about how well it's really doing. That's not a lucky system. That's an engineered one.

Step back and look at what your marketplace can now weather that the one from two passes ago could not. It can lose an entire region mid-rush and keep taking orders from the other. It can be hit by a self-feeding retry storm and starve the loop instead of feeding it. It can absorb a tenfold flash crowd by degrading gracefully instead of falling over. It can survive a bad write, contain a failure to a single cell, and recover without locking its own operators out. And it can tell you, in numbers, exactly how reliable it really is.

That is the whole of this part of the course made concrete: the distributed-systems theory of the early sections, the reliability discipline of the middle ones, and the hard-won lessons of five real outages, all standing up under fire on one system you designed. You didn't just read about staying up. You built something that does.

But a system that survives failures still has to be run — deployed, observed, secured, and paid for — and that's a different kind of hard. The next pass takes this same marketplace and puts it into production: split into services, shipped through a pipeline, watched through real observability, and held to a cost budget. Your resilient design meets the messy realities of operating it, in Capstone Pass 4: The Marketplace Goes to Production.