Schema Evolution & API Versioning

Last lesson ended with you decoding a protobuf message using the wrong schema — and watching it misread the bytes without a single error. That little experiment is about to become the whole lesson, because here's the uncomfortable truth about schemas: they're load-bearing, they're shared between writers and readers… and they will not stay still. Products grow. Fields get added, renamed, retired. The User message you froze in a .proto file last quarter is already wrong for what the product does today.

The naive fix sounds reasonable: when the schema changes, just update everything at the same time. And here is the load-bearing fact of this entire lesson — you can't. Not "it's hard"; it's structurally impossible, three different ways:

• Rolling deploys replace your servers a few at a time — for minutes (or hours), old code and new code run side by side, both reading, both writing. That's not an accident; it's how zero-downtime deployment works — you built it in §5.
• Clients you don't control — a mobile app ships its update through an app store and users install it whenever they feel like it. Some fraction of your traffic will speak last year's schema for… years.
• Data at rest — every message sitting in a Kafka topic, every file in a data lake, every cached blob was written by code that may no longer exist. Data outlives the code that created it.

So at any given moment, old readers meet new data and new readers meet old data — both directions, simultaneously. Every schema change you make either survives that crossfire or breaks something for someone, silently or loudly. The discipline of making changes that survive is called schema evolution, and it comes down to one matrix, two carefully-named words, and a shortlist of rules you can learn in an afternoon and use for a career.

The vocabulary trips up almost everyone, so let's nail it down with the one trick that makes it stick: name the reader.

• Backward compatibility: the NEW reader can read OLD data. Your freshly-deployed code looks backward in time at everything already written — the Kafka topic, the database rows, the cached responses — and copes. New code, old data.
• Forward compatibility: the OLD reader can read NEW data. Code you haven't updated yet (the not-yet-deployed servers, the two-year-old mobile app) receives data from the future — written under a schema it has never seen — and copes. Old code, new data.

Draw it as a 2×2 — readers down the side, data across the top — and the shape of the problem appears. The diagonal is free: old-reads-old and new-reads-new work trivially, by definition. The entire discipline lives in the two off-diagonal cells. And the killer fact from the intro: during any rollout, both off-diagonal cells are live at the same time. The new server (deployed first) is reading data the old servers are still writing — backward. The old servers are reading data the new one just started writing — forward. There is no deploy order that avoids the crossfire; there's only surviving it.

One practical corollary worth knowing (it's how Kafka's Schema Registry frames its compatibility modes): if a change is only backward compatible, you must upgrade consumers first; only forward compatible, producers first; and when you can't control the order at all — hello, mobile apps — you need full compatibility, both directions, every change.

So which changes survive both directions? The honest answer fits on an index card, and it's the most valuable index card in API design.

The workhorse safe change: add an optional field with a default.

• Forward: the old reader meets the new field and — following the rule every good format bakes in — ignores what it doesn't recognize. No crash, no complaint; it just doesn't use the new field.
• Backward: the new reader meets old data where the field is missing and fills in the default. Complete record, no error.

Both lanes green. This is why disciplined APIs evolve almost exclusively by addition — and why removing an optional-with-default field is equally safe (run the same logic in mirror image).

The kill list:

• Adding a required field breaks backward: old data doesn't have it, and the new reader — told the field must exist — rejects every record ever written. (This is why protobuf v3 removed required entirely: the language stopped trusting you with the foot-gun.)
• Renaming a field is a delete plus an add wearing a trench coat. The old reader doesn't know the new name; the new reader doesn't know the old one. Both directions lose the value — and in JSON they lose it silently: no error, the field just quietly reads as absent. Grep your incident history for "the field was there, but null."
• Changing a field's type — readers decode garbage or crash, depending on the format's mood.
• And the champion of catastrophes, protobuf-specific: reusing a tag number. Remember from last lesson — the wire carries only numbers. If tag 3 used to be email and you reuse 3 for phone, then every old byte-stream with tag 3 decodes, cleanly and without any error, into the wrong field. Not a crash — silent data corruption, the failure mode you find weeks later in a support ticket. The rule is absolute: field numbers are forever. Never renumber. Never reuse. When you retire a field, tombstone its number: reserved 3; — protobuf will then refuse to compile anyone's attempt to reuse it. That's what the tombstone is for.

Avro plays the same game with different levers: add/remove fields with defaults and you're fully compatible; the idiom for optional is a union with null listed first — ["null", "string"], "default": null. And because Avro readers resolve the writer's schema through the registry, the registry can enforce a compatibility mode (BACKWARD, FORWARD, FULL) at publish time — schema changes get CI, and incompatible ones bounce before they ever reach production. JSON, of course, enforces nothing: the same rules apply, but the only enforcement is your code review — which is why the tolerant reader pattern (ignore unknown fields, default missing ones, never drop fields you don't understand when round-tripping) is a written rule at serious JSON shops rather than a compiler feature.

The two directions stop being confusable the moment you watch them decode — so here's a v1 schema, a stack of proposed changes, and both mixed pairings running live: the forward lane (old reader × new data — the rolling-deploy direction) and the backward lane (new reader × old data — the migration direction).

Start gentle: add age as optional with a default. Watch both lanes stay green — the old reader shrugs past the field it doesn't know; the new reader fills age: 0 into old records. That's the workhorse. Now add a required field and watch precisely one lane die — backward, because history can't be retrofitted. Rename email to contact and watch both lanes lose the value — the trench coat in action.

Then run the one this interactive exists for: reuse tag 3 for phone. The forward lane doesn't go red — it goes amber. Look closely at what the old reader decoded: a phone number, sitting in the email field, with no error anywhere. That's silent corruption — the failure mode that doesn't crash, doesn't log, and doesn't get caught until a customer asks why their invoice went to +1-555-0147@. Then click reserved 3; and watch the compiler-level tombstone refuse the reuse outright.

Every change you try fills in its row of the compatibility matrix at the bottom — by the time you've tried all six, you've built the lesson's core table yourself, and the two words will never swap on you again.

Sometimes the change you need genuinely can't be made additive — the resource model was wrong, the field semantics changed, the response shape has to break. Now you need API versioning, and the first thing to know is the strategies' actual shapes:

• URI versioning — /v1/orders, /v2/orders. Visible in every URL, trivially cacheable and routable, obvious in logs; the default for public APIs. The cost arrives later: every live version multiplies your routes, docs, tests, and support surface — and old versions linger, because clients never migrate on your schedule.
• Header versioning — Api-Version: 2024-06-01 on a stable URL. Cleaner URLs, REST-purists approve; the cost is discoverability — the version is invisible in a browser bar, a copy-pasted curl, or a junior engineer's first read of the logs.
• Media-type versioning — Accept: application/vnd.example.v2+json. Maximum purity, minimum ergonomics; awkward with CDNs and most tooling. Rare in the wild for good reason.
• Stripe's date pins — the strategy worth studying even if you never copy it: every account is pinned to the API version (a date) it first integrated against, forever, until it explicitly opts into a newer one. Internally there's one modern codebase plus transform layers that reshape responses per pinned version. Breaking changes ship continuously; no customer ever gets broken. It's the gold standard for consumer safety — and a reminder that versioning is a compatibility delivery mechanism, not a URL aesthetic.

But the real lesson about versioning is the one hiding above the strategy menu: versioning is the most expensive tool in the box, so the best teams almost never reach for it. GitHub and Stripe have held a single major version for over a decade — not because they never changed anything, but because they changed things additively: new optional fields, new endpoints, never a rename, never a removal. Bumping /v2 for a change that could have been additive doesn't just cost you a second version to run — it trains every client to fear upgrades. Version when the break is real and unavoidable; evolve additively the other 95% of the time. And whichever strategy you pick, plan deprecation from day one — sunset timelines, migration guides, usage metrics on old versions — because the strategy's syntax matters far less than whether retiring a version was designed into the contract or bolted on during a crisis.

Schema evolution is a lesson in paying small costs early to avoid enormous ones later — worth naming the ledger:

• Additive evolution costs you schema bloat: retired-but-present optional fields, reserved tombstones, a User message with three generations of geological strata. The alternative costs you broken clients. The bloat is annoying; the breakage is a pager. (Cleanups happen — batched, rarely, as their own carefully-planned migration, not as a casual Tuesday rename.)
• Full compatibility (both directions, every change) is the strictest discipline and the only safe one when you don't control deploy order — mobile, public APIs, long-lived data. Internal service pairs with controlled deploys can sometimes get away with one-directional compat plus deploy ordering (consumers-first for backward-only) — a legitimate optimization if the ordering is enforced by machinery, not by a wiki page.
• Registry-enforced compat (Avro/Kafka world) turns these rules from tribal knowledge into CI: incompatible schemas are rejected at publish. The cost is running the registry and arguing with it occasionally; the benefit is that the intern literally cannot ship the tag-reuse bug. JSON shops pay the mirror price: total flexibility, zero enforcement, and the tolerant-reader rules live or die in code review.
• Versioning buys you the right to break — at the price of running, documenting, testing, and eventually sunsetting N parallel realities. Every version you add is a promise measured in years.

The meta-lesson echoes L1: contracts calcify. The moment a schema has two independent parties on either side of it, changing it stops being an edit and becomes a negotiation with the past. Design fields as optional-with-defaults from day one, number them like they're forever (they are), and the negotiation stays polite.

• "Backward and forward compatibility are roughly the same idea." They're opposite arrows, and mixing them up inverts your deploy order. Fix it forever: name the reader. NEW reader looking backward at old data; OLD reader looking forward at data from the future. During a rollout, both arrows fire at once.
• "We'll just coordinate the deploy." There is no simultaneous deploy. Rolling updates overlap by design; mobile clients update on the user's whim; Kafka topics and data lakes hold bytes from code that's already deleted. 'Coordinate everything' is not a strategy — it's the absence of one.
• "Renaming a field is cosmetic." A rename is a delete plus an add in a trench coat, and it breaks both directions — in JSON, silently. The value doesn't error; it just stops arriving. Add the new name, keep the old, migrate, retire — or don't rename.
• "Protobuf/Avro handle evolution automatically." They handle it mechanically, given discipline. Protobuf's tags are forever — reuse one and old bytes decode into the wrong field with no error at all (corruption, not crash; reserved exists precisely for this). Avro is only automatic when every field carries a sensible default.
• "Something changed? Bump to /v2." Versioning is the most expensive tool you own — N versions of everything, forever, plus clients trained to dread upgrades. GitHub and Stripe rode one major version for a decade on additive discipline. Version for real breaks; evolve for everything else.

Twenty minutes with protoc (you installed the muscle memory last lesson) and the rules become scars — the good kind:

1. Write v1 and freeze some history. A user.proto with int64 id = 1; string name = 2; string email = 3;. Encode a few records with protoc --encode and save the binary files — these are your Kafka topic, your data lake: bytes from the past.
2. Make the safe change. Add int32 age = 4; to the schema. Decode yesterday's files with the new proto — everything works, age just reads as 0 (backward ✓). Encode a new record with age and decode it using the old proto — works, the unknown field is skipped (forward ✓). You've just verified full compatibility with four commands.
3. Now commit the crime. Change field 3 from string email to string phone — wait, that's just a rename on the wire (tags match, types match: old data decodes fine, which is its own lesson about what protobuf does and doesn't know). The real crime: delete email, then add string phone = 3; with a different meaning, and decode an old file. Watch the old email address arrive, silently, in phone. No error. Wrong data. Sit with that for a full minute.
4. Install the guardrail. Change the schema to reserved 3; and try to declare string phone = 3; — protoc now refuses to compile. That refusal is the entire institutional memory of this failure mode, encoded in a keyword.
5. Bonus, JSON edition: write a "tolerant reader" in ten lines (ignore unknown keys, default missing ones) and a strict one (KeyError on anything unexpected). Feed both a record with an extra field and a record with a missing field. The tolerant one shrugs twice; the strict one dies twice. That's the entire JSON evolution story — the format doesn't help you, so the reader must.

Schemas change; the world doesn't pause for them. Everything in this lesson hangs off that one fact:

• You can never update both ends at once — rolling deploys, uncontrolled clients, and data-at-rest guarantee that old and new readers meet old and new data simultaneously.
• Two words, two arrows: backward = NEW reader, old data; forward = OLD reader, new data. The 2×2 matrix's diagonal is free; the off-diagonal is the discipline — and both cells are live during every rollout.
• The index card: add optional-with-default ✓✓ (the workhorse) · add required ✗ backward · rename = delete+add in a trench coat ✗✗ · change type ✗✗ · reuse a protobuf tag = silent corruption — tags are forever, reserved is the tombstone. Avro: defaults make you compatible, the registry makes it CI. JSON: same rules, zero enforcement — be a tolerant reader.
• Version as a last resort: additive evolution held GitHub and Stripe on one major version for a decade. When you must break: URI versions for visibility, headers for clean URLs, Stripe's date-pins for consumer safety — and plan the sunset from day one, because the strategy is cheaper than the discipline.

The section's arc so far: contracts (L1), honest walking (L2), safe repetition (L3), the bytes themselves (L4), and now contracts that can change without breaking. Next, a challenger appears: what if the client could just ask for exactly the shape of data it wants — no fixed response shapes, no over-fetching, one endpoint? That's GraphQL — its genuine superpowers, and the costs its fans mention more quietly.