Capstone Pass 2: The Marketplace Data Layer

You built this system once already — in Pass 1, as a single-region MVP: one load balancer, a few stateless app servers, a cache where it paid, one honest database, and a queue fanning out order events. It was correct, and it was just enough. Then it grew.

"Plates" — the food-delivery marketplace — has gone national. Call it ~20M daily users, forty metros, the same two sharp meal spikes a day. And here's the thing the Pass 1 review warned you about: the cracks that growth calls in are not load problems. Pass 1's envelope already sized the fleet, the cache, and the queue for far more traffic than this. The problems now are about the data itself:

• The last pizza. Two customers order the final margherita in the same 50 ms. Today, both rows say confirmed. That's a correctness-under-concurrency bug, and it's shipping right now.
• "Best sushi near me under ₹500." The catalog's WHERE name ILIKE '%sushi%' was fine for 4,000 restaurants. It is a full-table-scan nightmare for a national marketplace — and it can't rank, or filter by distance, at all.
• Order history forever. Three years of orders is heading for ~0.9 billion rows — and analytics wants to query all of it fast, on the same table that's serving tonight's dinner rush.

Your deliverable is Pass 2 of the capstone: the data layer. Everything Container 2 taught — storage engines, replication and consistency, read scaling, transactions, partitioning, and the specialized indexes you just finished — aimed at one grown system.

The trailhead rule still stands: attempt before you peek. Grab paper, sketch your data layer and run the envelope before the guided build below. The build grades your instincts decision by decision; the reference afterward is for comparing, not copying.

Design reviews run in an order, and the order is the method: size the data, then let the shape of the load choose each store. Eight decisions. The first keeps the pass honest; the second is the envelope; the last six are one Container-2 section each — storage, transactions, replication, reads, partitioning, indexes.

Each choice is graded — the reference move, a defensible-but, or the trap it came from — and every wrong option is a trap this container named out loud. Watch the architecture draw itself as you decide.

Compare, don't copy. Here's the reference reasoning for the numbers — the part that decides everything after it.

The envelope. 20M DAU × ~40 requests = 8×10⁸ requests/day. A day is ~10⁵ seconds, so that's ~8,000 QPS average; food earns the top of the peak rule (×5), so ~40,000 read QPS at the meal rush. Orders are ~4% of users — ~800K/day, about 40 order-writes/sec at peak, roughly 1,000× fewer than the reads. And three years of orders at ~800K/day is ~0.9 billion rows.

Now read the asymmetry, because it is the design. Those three numbers don't just size hardware — they point in three different directions:

• Reads are enormous and repetitive (everyone browses the same popular menus) → they want a cache and read-replicas, and they can tolerate being a little stale.
• Writes are tiny but sacred (40/sec, but each is money and inventory) → they want a transaction, and they cannot be stale to the person who made them.
• History is huge and mostly cold (0.9 B rows, but tonight's rush only touches today's) → it wants partitioning and hot/cold tiering.

⭐ The load isn't one shape, so the data layer isn't one store. Every decision below is really just: which of these three pressures am I answering right now?

Here's the reference architecture — the six choices, assembled. Yours doesn't have to match it; it has to be defensible with numbers, the way this one is.

Storage engine. The append-only order/event firehose goes on an LSM-tree (sequential writes, no in-place update, no write amplification); the read-mostly catalog stays on a B-tree (fast point and range reads). Match the engine to the access pattern — one size fits neither.

The last pizza. The inventory decrement is a read-modify-write, so it's wrapped in a transaction that locks the item row (SELECT … FOR UPDATE, or serializable isolation) — the second order sees the decremented count and is refused. Crucially, that cost is scoped to the money + inventory path only; browsing never touches it.

Replication & consistency. A single leader takes writes. The money path is synchronous / read-your-writes (you always see the order you just placed); browse is served from asynchronous read-replicas where a few seconds of staleness is invisible. Consistency is paid in latency — so it's spent only where a human would notice.

Reads. 40,000 QPS never touches the primary head-on: a catalog cache (short TTL) absorbs the repeated menu reads, replicas carry the browse tail, and only writes plus must-be-fresh reads hit the leader.

Partitioning. The write firehose and 0.9 B rows are sharded by hash(user_id) so load spreads evenly (never by city — the biggest metro would melt one shard), and history is partitioned by time for hot/cold tiering.

Access patterns. The queries the B-tree can't answer each get their own shape: an inverted index + BM25 for search, a geohash / S2 index for “near me”, and a sketch for “trending”. §8, applied.

Every box earns its place from a number or a failure mode. That's the whole method.

Score your paper attempt — the sketch you made before the guided build — honestly. One point each:

The numbers (3):

1. Sized the envelope: ~40,000 read QPS at peak, ~40 order-writes/sec, ~0.9 B rows over three years.
2. Named the read/write/history asymmetry out loud — reads huge & repetitive, writes tiny & sacred, history huge & cold.
3. Let that asymmetry drive the stores, rather than reaching for one database to do everything.

The shape (6):
4. Matched the storage engine to the workload (LSM for the write log, B-tree for the catalog) — not one engine for both.
5. Stopped the last-pizza race with a row-locked / serializable transaction, and scoped it to the money path only.
6. Put read-your-writes on the money path and eventual consistency on browse — not everything-strong or everything-eventual.
7. Scaled reads with cache + replicas, left order status uncached, and pinned fresh reads to the leader.
8. Sharded by a high-cardinality key (hash of user/order), not by city or status — and could say why city runs hot.
9. Gave search, “near me”, and trending their own index shapes instead of more B-tree indexes or a LIKE.

The judgment (3):
10. Nothing in the design exists without a number or a failure mode justifying it.
11. Named the next ceiling — multi-region and real-time — without pre-building it.
12. Could walk another engineer through all of it in ~10 minutes.

10–12: staff-shaped. 7–9: solid pass — reread the reference tier that dropped points. Below 7: genuinely fine — reread The Reference against your sketch, mark the three biggest gaps, and redo it on a fresh sheet. A draft you correct teaches more than an answer you read.

This data layer is correct — and, like Pass 1's, it's carrying debts that the next stage will call in. The tell: every problem left is now about distribution, not data.

• One region is one point of failure. Everything here lives in a single datacentre. One regional outage takes Plates fully down — and the moment you copy it to a second region, the CAP tax you priced in the consistency lessons becomes a geography bill you pay on every write.
• The live map you deferred. Courier GPS every few seconds is a real-time fan-out — persistent connections, pub/sub — the one thing a request/response data layer can't serve.
• Coordinating across shards and regions. A cross-shard transaction, a global leader election, exactly-once on the event stream — these need consensus, not just a lock on one row.

Pass 2's problems were data — correctness, access patterns, growth. Pass 3's are distribution — geography, real time, coordination. That's the next container.

Look back at what your eight decisions quietly used — the whole of Container 2: Data Systems.

• Storage engines — LSM vs B-tree, and matching the engine to an append firehose or a read-mostly catalog.
• Transactions & concurrency — isolation levels, row locks, and the check-then-write race that oversells the last unit.
• Replication & consistency — single-leader topology, sync vs async, and read-your-writes; consistency spent only where it's seen.
• Scaling reads — caches, read-replicas, and the replication lag that makes “where's my order?” read stale.
• Scaling writes & partitioning — sharding by a high-cardinality key, and the hot-shard / celebrity problem that a city key walks straight into.
• Specialized indexes & sketches — §8's toolbox: an inverted index and BM25 for text, a spatial index for proximity, a sketch for frequency — each access pattern its own shape.

You started Container 1 with one honest box. You leave Container 2 able to take that box apart into the right stores — one for each pressure the data puts on it — and defend every one with a number. Next, in Container 3, that data layer has to survive being spread across the planet.