Shadow Traffic & Dark Launches

Data Integrity: Backups That Restore ended on an uncomfortable observation: almost every disaster in this whole section traces back to a single act: a change someone shipped. A bad migration, a buggy deploy, a new version that falls over under load. You've spent six lessons learning to survive those changes after the fact. This last lesson is about catching them before they can hurt anyone, and the cheapest backup of all is the restore you never had to do.

The obstacle is a comfortable lie almost every team tells itself: it passed in staging, so it's fine. It is not fine, because staging lies. No test environment on earth has production's true scale, its real data with all its weird historical shapes, its traffic patterns full of edge cases nobody thought to write a test for, its 3 a.m. spikes and malformed requests and that one customer with two million line items. You can pass every test you have and still meet a request in the wild that your new code has never seen and cannot handle. As the engineers who learned this the hard way put it: there is no environment like production except production.

So the honest way to test a change is to test it against production itself, on its real traffic, its real load, its real behavior, while making sure that, until you're certain, not one user depends on the result. This lesson is the two techniques that pull off that trick. Shadow traffic feeds the new code real requests while throwing its answers away, and dark launch ships the new code live but hidden, revealing it only when you choose. Both let you find out the truth in production, safely.

The first technique is traffic mirroring, also called shadowing, and the idea is almost too simple to trust. Your load balancer, the traffic director from Load Balancers: The Traffic Directors, is already sitting in front of everything, seeing every request. So you teach it to do one extra thing: for each live request, it still sends the request to production and returns production's answer to the user exactly as before, but it also sends a copy of that request to your new candidate version running quietly alongside. The candidate does all its work, produces its response, and that response is thrown away. The user never sees it. It's fire-and-forget.

Sit with what that buys you. Your new code is now being hit by real production traffic — the exact requests, in their true variety and volume, including all the ugly edge cases you would never have dreamed up in a test, and you can watch how it behaves: does it crash, does it throw errors on inputs the old code handled, is its p99 latency quietly twice as slow. You learn all of that under genuine production conditions, and the user impact of every bug you find is exactly zero, because no user ever received a shadow response. This is the closest thing to a real launch that isn't one, and it's cheap: it lives right in the proxy layer, where tools like Envoy, NGINX, and service meshes offer request mirroring as a built-in feature. You flip it on, and your candidate meets the real world without the real world ever meeting it. The console below lets you run exactly this, and then break the candidate to watch the shadow catch it while every user stays happy.

There is one way shadow traffic goes from brilliant to catastrophic, and it catches people who were sure they'd been careful. The safety of the whole technique rests on one word: the response is discarded. But discarding the response does not undo what the request did on the way to producing it.

Follow a mirrored payment. A user clicks the Pay button once. Production charges their card, correctly, and returns the receipt. And the mirrored copy, running through your candidate, also reaches out to the very same payment service and charges the very same card a second time. You threw away the candidate's response, so the user never saw the second receipt — but the money is gone, twice, from a real person. The same trap springs for any action that touches the outside world: a mirrored request that sends an email double-sends it, one that writes a row inserts it twice, one that decrements inventory double-decrements it. The discarded response hid the symptom and left the damage.

This is why the mirror is only safe when the candidate cannot touch the real world. In practice that means one of three things: keep shadow traffic to read-only requests, which is why teams start by mirroring reads; point the candidate's writes at a separate shadow datastore and a stubbed payment sandbox so its side effects land somewhere harmless; or make every side effect idempotent in the deep sense of Idempotency: Safe to Retry — keyed so that the mirror's duplicate is recognized as the same action and collapses into one. The mirror is an extra invocation of your system, and every non-idempotent side effect it triggers happens for real. Get this wrong and your safe test becomes a production incident with your name on it.

Mirroring tells you the new code didn't crash and isn't too slow. It does not tell you the new code gave the right answer — a candidate can respond quickly, with a 200, and be quietly, confidently wrong. The technique that closes that gap is shadow diffing: run the old path and the new path on the same real input, compare their outputs, and log every place they disagree.

The cleanest version of this runs right inside the code, and GitHub built a small, famous library for it called Scientist. You wrap the risky refactor in an experiment with two branches: the control is the trusted old code, and the candidate is the new one. When the experiment runs, both branches execute on the live input, but the user is always returned the control's result — the old, trusted answer — so a bug in the new code can never reach them. Behind the scenes, Scientist compares the two results and records any mismatch, along with how long each took. Now you let real production traffic pour through it and you watch the mismatches accumulate. Most of the time a mismatch means the new code has a bug or is missing some behavior the old code quietly had — the exact regressions your unit tests never imagined, because they were triggered by real, strange inputs. And every so often you get the gift of the whole exercise: a mismatch where the new code is right and the old code has been wrong for years, and nobody knew. Measure twice, cut once — and only replace the old path when the disagreements have dropped to zero.

The second technique flips the arrangement. Where shadow traffic runs the new code on real requests but hides its output, a dark launch puts the new code fully live in production but hides its existence. The feature is deployed, running, and completely invisible — wired up behind a feature flag that's turned off, or built with no entry point in the interface — so that not one user can reach it even though it's really there.

The reason this is powerful is that it breaks apart two things teams almost always fuse into one scary event. Deploying code means the code is running in production. Releasing a feature means users can actually use it. Bundle them and every release is a high-wire act. Separate them — deploy dark, release later — and the deploy becomes boring: the code goes out quietly, and you can exercise it in production with no one watching. It can warm its caches, compile its hot paths, fill its connection pools, and even generate real load, all before its first real user arrives. The classic story is Facebook's chat launch: for a while before anyone could send a message, their pages silently opened connections to the brand-new chat servers and simulated message traffic — with no chat box drawn anywhere on the screen — purely to prove the backend could carry hundreds of millions of users before the feature existed for a single one of them. The flag that keeps it dark is the same feature flag machinery from Feature Flags; here you're using it not to configure a feature but to keep a finished one holstered until you're ready.

A dark launch is only half the move; the other half is how you bring it into the light. Because the feature is gated behind a flag, you don't have to reveal it to everyone at once in a heart-stopping big-bang release. You ramp it: turn it on for 1% of users, watch your error rates and latency and business metrics, and if all is well, take it to 5%, then 25%, then everyone — progressive exposure, one careful step at a time, with a tiny blast radius at every stage.

And here is the property that makes engineers sleep at night: the very same flag that reveals the feature is an instant kill switch. The moment something looks wrong — errors climbing, latency spiking, the support queue lighting up, you flip the flag back off, and the feature vanishes for everyone in seconds. No rebuild, no redeploy, no waiting twenty minutes for a pipeline while the incident burns; just a configuration value changing from on to off. Rollback stops being an emergency deployment and becomes a toggle.

It's worth being precise about how this differs from its cousins, because the mechanics look similar. A canary from Rolling, Blue-Green, Canary shifts a slice of real traffic onto a whole new version of the service to check its technical health; a dark-launch ramp reveals a hidden feature inside the version that's already deployed. And an experiment from A/B Testing Infrastructure also shows a feature to a fraction of users, but for a different question entirely: A/B asks which version do users prefer — a business question about behavior — while a ramp asks is this safe to turn on — an engineering question about risk. Same dial, different reason for turning it.

• Staging lies; production is the only real test. No environment matches production's scale, data, and traffic, so test the change against production itself — while no user depends on the result yet.
• Shadow traffic mirrors real requests to the new code and discards its answers. The candidate meets real production traffic; the user meets only the old code; every bug you find costs zero user impact.
• The side-effect trap is the one deadly catch. Discarding the response doesn't undo a write — a mirrored payment double-charges. Keep shadows read-only, sandbox their writes, or make side effects idempotent.
• Shadow diffing proves the answer is right, not just alive. Run old and new on the same input, return the old, log every mismatch — and sometimes discover the old code was wrong all along.
• Dark launch decouples deploy from release. Ship the code hidden behind a flag; it can warm and take load before anyone sees it. Then ramp exposure 1% at a time, and the same flag is an instant kill switch — rollback without redeploy.

And with that, the reliability discipline is complete. You can define what reliable means and measure it, page the right human when it breaks, respond to and learn from the incident, recover from a lost region or a corrupted database, contain a failure to a single cell, restore data you can actually trust, and now ship changes without betting the whole system on them. Seven lessons, one through-line: reliability is not the absence of failure; it's the discipline of staying up anyway. The next stop gathers all of it into a single test — Checkpoint: The Reliability Discipline.