CDNs: Caching at the Edge

Your stack is, by now, genuinely fast. The database hides behind a cache with a 95% hit rate. Reads come out of RAM in a millisecond. The stampedes are locked down, the cache fleet scales without reshuffling the world. A user in Virginia — where your servers live — gets a page in 40 milliseconds and thinks your app is instant.

A user in Sydney thinks your app is broken.

Every request she makes crosses fifteen and a half thousand kilometres of Pacific fibre, waits for your (excellent, RAM-speed) answer, and crosses right back. Her requests spend almost nothing in your stack — and an eternity getting to it. All that work you did in this chapter — the ladder, the hit rates, the eviction, the stampedes — optimized everything except one thing: the map.

And the map doesn't care about your code. The trip itself is the cost, and no amount of server tuning shortens an ocean. There's only one move left, and it's the same move this chapter has made from the start — move a copy of the answer closer to the asker — except this time "closer" isn't a faster tier of memory. It's another continent.

That's a CDN — a content delivery network: the same caching ideas you now own, stretched over the planet, into what is genuinely the world's largest cache. Well over half of all internet traffic is served from one.

Scope: what the edge is, how content gets there (and the one big design fork: pull vs push), and how the stampede lessons come roaring back at planet scale. How you un-cache something from four hundred cities at once — purging the planet — is the next lesson.

Light in fibre travels at about two-thirds of its vacuum speed. That works out to a brutally simple rule of thumb: ~5 milliseconds per 1,000 kilometres, one way. Not "on a slow day." Not "until you optimize." It's physics — the hardest of hard floors.

Do the Sydney math. Sydney to Virginia is roughly 15,500 km, so one direction costs ~78 ms and a round trip ~155 ms — if the fibre ran perfectly straight and nothing else added a nanosecond. Now remember what the networking chapter taught about new connections: a fresh HTTPS visit isn't one round trip, it's several — TCP's handshake, then TLS's — before the first byte of your actual page moves. Three round trips at 155 ms is nearly half a second of pure geography, spent before your beautifully-cached backend even hears the question. (On the human scale from the latency-ladder lesson: a cross-continent packet was the "five years" rung. Sydney pays it twice per round trip.)

Sit with the shape of the problem for a second, because it's unusual: every other latency in this course had an engineering fix — a better algorithm, a warmer cache, a bigger box. This one has exactly one fix, and it's geometric: make the trip shorter. If the answer lives 50 km from Sydney instead of 15,500, the physics bill drops from 155 ms to under a millisecond of propagation — a 100× improvement your code could never buy.

So that's the whole idea, stated once and cleanly: put copies of the answer near every cluster of askers on Earth. Everything else in this lesson is the machinery for doing that well.

A CDN is a company (Cloudflare, Akamai, CloudFront, Fastly…) that has already done the impossibly expensive part: built points of presence — PoPs — in hundreds of cities, each one racks of edge servers sitting a few milliseconds from the people who live there. The scale is worth saying out loud: Cloudflare operates in 330+ cities and claims ~95% of the internet-connected world lives within ~50 ms of a PoP; Akamai runs thousands of PoPs across ~700 cities. You rent all of it by pointing your DNS at them.

The mental model is exactly the proxy cache you already know, multiplied by four hundred. An edge server generates nothing — it remembers. Your own server acquires a new name and a new job description: the origin — the single source of truth the edges fall back to.

The request flow reuses two ideas you've already learned, so this is quick:

• Sydney's request lands on the nearest PoP — that's the anycast and GeoDNS routing from Anycast: One IP, Many Places and Global Load Balancing, doing exactly what it was built for.
• The edge checks its cache. Hit → served locally in ~10–30 ms, and the request never leaves the city. Miss → the edge fetches from the origin (the full ocean crossing, once), caches the answer, and serves it — so the first user in a region pays ~100–300 ms, and everyone after pays ~20.

You already know how to judge such a system: hit rate is the whole game (that was L60, and it's still true at the edge — well-run static content hits 95%+). Operators track its sibling, origin offload — the share of bytes served without touching your origin. Above 90%, your origin spends its days seeing a trickle: the misses, and nothing else. The map problem — solved by the same arithmetic that solved the database problem.

Physics is more convincing when you watch it. Below is a world map with users in Sydney, Tokyo, London, and São Paulo, all hitting your app; the origin lives in Virginia. Requests fire continuously — and you can click any city to fire more.

Do this in order:

1. Start on NO CDN. Every request arcs across the planet to Virginia. Watch the per-city latency readouts — Sydney lives near 160 ms — and the origin gauge carrying every request on Earth.
2. Switch to PULL. Each city's first request still makes the long trip — watch for the first-miss badge — then caches at the local PoP, and every request after it bounces back at ~20 ms. Lazy, self-managing, and the origin quiets to a trickle. (You've seen this pattern before: it's cache-aside, wearing a passport.)
3. Switch to PUSH. Before traffic even starts, watch the pre-fill seed every PoP from the origin. Now nobody pays a first-miss — at the price that the origin paid to ship everything everywhere, asked-for or not.
4. Finally, press FLUSH ALL PoPS (every edge forgets at once — think of it as tomorrow's lesson knocking early). Without the origin shield, every PoP races back to Virginia simultaneously — a thundering herd made of caches. Toggle the shield on and flush again: the misses collapse into a single origin fetch. That's last lesson's single-flight idea, promoted to run a planet.

Keep an eye on the two numbers that tell the story: Sydney's latency (geography, deleted) and the origin's load (the offload, earned).

One design fork decides most of your CDN setup: how does content get to the edge? You've already met both answers in miniature.

Pull — the lazy default. The edge fetches from your origin the first time someone asks, caches it, and serves everyone after. This is cache-aside at planet scale, and its virtues are the same ones: nearly zero operations (point DNS at the CDN and you're live), only content people actually request occupies cache space, and the whole thing self-manages as your site changes. The costs are familiar too: the first user in each region eats the full origin round trip — once per PoP, so a 300-PoP network pays that miss up to 300 separate times — and your origin must stay healthy enough to answer the misses. For almost every website — HTML, CSS, images, APIs — pull is the right answer, which is why it's the default of every major CDN.

Push — pre-position before anyone asks. You upload content to the CDN ahead of demand, placing it in PoPs before the first request exists. Nobody ever pays a first-miss. In exchange, you own the hard questions: what to place where, when to update it, and paying to store it in hundreds of cities — including ones where nobody ever asks. Push earns its keep when content is large, changes rarely, and demand is predictable: video libraries, game patches, OS updates, launch-day downloads.

And push has one perfect story — Netflix Open Connect. Netflix built its own CDN out of cache appliances physically installed inside ISP networks — closer than close. Every night, during off-peak fill windows, predictive algorithms pre-load each appliance with what that region is likely to watch tomorrow. The result: about 95% of Netflix's traffic never crosses the internet backbone at all — it's served from a box a few kilometres from your TV that was filled while you slept. That's push, perfected: they know tonight what you'll press play on tomorrow.

Real setups are hybrids — pull for the living, changing site; push for the heavy, predictable library. (And for content that can't be cached at all — personalized pages, authenticated APIs — CDNs offer dynamic acceleration: your traffic rides their private backbone on kept-warm connections, cutting 30–70% of latency with no caching. Knowing it exists is enough for now; using the edge cleverly is next lesson's business.)

Here's the moment §8 comes full circle. A popular object expires — or you deploy a new version — and it vanishes from the edge. What happens next?

With four hundred PoPs, you get four hundred simultaneous misses converging on one origin. Look at the shape: it's the thundering herd from the stampede lesson, except the herd isn't made of user requests — it's made of caches. Your CDN, the thing shielding your origin, becomes the thing that tramples it. And one PoP's own users pile up too: ten thousand Londoners asking one edge for the same dead object is just the hot-key dogpile, relocated.

The cures are — satisfyingly — the ones you already own, wearing CDN uniforms:

• Request collapsing (the CDN name for single-flight): each PoP notices a hundred concurrent requests for the same uncached object, sends one fetch upstream, and fans the answer out to everyone waiting. Every serious CDN does this inside every PoP, by default.
• Tiered caching with an origin shield: instead of every edge missing straight to your origin, misses climb a hierarchy — edge → a bigger regional cache → one designated shield PoP parked in front of your origin. Each layer absorbs and collapses the misses of the layer below, so four hundred edge misses arrive at the shield… and leave as one origin fetch. (CloudFront calls it Origin Shield; Fastly, "shielding"; Cloudflare, tiered cache. Same idea, one checkbox.)

Notice what just happened conceptually: the CDN is running a cache in front of a cache in front of a cache — the latency ladder from L60, rebuilt out of geography — and defending itself with L64's coalescing at every rung. Nothing in this section was new. That's the point: you already knew how a CDN works; you just hadn't stretched the picture over a map yet.

• "CDNs are for big companies." Backwards, if anything — a pull CDN is one DNS change and often free. Most of the web, including tiny blogs, sits behind one. The expensive thing (400 PoPs) is exactly what you rent instead of build.
• "A CDN makes my server faster." It makes your server quieter and your users closer. Origin latency is unchanged — every miss still pays it, which is why hit rate and offload remain the numbers that matter.
• "Push beats pull — no first-miss!" Push trades that miss for owning placement, updates, and storage everywhere. For content that churns, that trade is awful. Pull's laziness is a feature — the same reason cache-aside won the last chapter.
• "The CDN just caches everything." It caches what's cacheable. Personalized pages and authenticated APIs need dynamic acceleration or careful rules — and deciding what's cacheable, for how long, and how to take it back is literally the next lesson.
• "A CDN is one big cache." It's hundreds of independent ones. Each PoP has its own copy, its own hit rate, its own first-miss. That independence is why the edge is fast — and why making four hundred cities forget something at once is a genuinely hard problem…
• "Latency is an engineering problem." Below everything else it's a physics problem — ~5 ms per 1,000 km, non-negotiable. The only winning move is the CDN's move: shorten the trip.

• Geography is the last latency, and code can't fix it — light in fibre costs ~5 ms per 1,000 km one way, so a Sydney↔Virginia round trip burns ~155 ms before your stack does anything. The only fix is geometric: move copies of the answer near the askers.
• A CDN is rented geography: PoPs in hundreds of cities, edge servers that remember rather than generate, your server demoted-and-promoted to origin. Requests find the nearest PoP (anycast + GeoDNS — you knew that); hits stay in the city; only misses cross the ocean. Hit rate is still the whole game — tracked at the edge as origin offload.
• Pull vs push is the fork: pull = cache-aside at planet scale (lazy, zero-ops, first user per region pays; the right default), push = pre-position (no first-miss; you own placement and updates — for big, stable, predictable content; Netflix Open Connect pre-fills appliances inside ISPs overnight and serves ~95% of its traffic without touching the backbone).
• The stampede rules run the planet: expiry with 400 PoPs = a herd made of caches; the cures are your own — request collapsing (single-flight in every PoP) and tiered caching with an origin shield that collapses the world's misses into one origin fetch.

Next — CDN Invalidation & Edge Strategies. The sneaker price changed again — but this time the stale copy isn't in one Redis box you can delete from; it's sitting in four hundred cities, each with its own independent cache. How do you take a lie back from the entire planet — fast, without stampeding yourself? That's purging the planet, and it's where this chapter's two hardest ideas — invalidation and stampedes — meet at global scale.