Data Formats & Serialization: JSON, Protobuf, Avro

Three lessons of this section have been about the envelope — the verbs, the paths, the walk, the retry. Time to open it. When your server "sends JSON," what actually crosses the wire? Bytes. And somebody — you, whether you noticed or not — decided which bytes: how a number becomes byte patterns, whether field names travel with their values, where the quotes and braces go. That decision is serialization: turning the structures in your program's memory into a stream of bytes, and back.

It feels like a non-decision because every web tutorial made it for you: JSON, obviously. And for plenty of systems, JSON is the right answer — we'll defend it properly in a minute. But the decision has a price tag most teams never read: if your services exchange a million messages an hour, every byte of overhead is a million bytes an hour of network, parse time, and cloud bill, forever. At that scale, "user_id": — ten characters of pure bookkeeping repeated in every message — stops being invisible.

Here's the frame that turns three competing formats into one clean idea. Every message needs two things: the values and the schema — the knowledge of what those values mean (which field is which, what type it is). The formats differ on exactly one axis: where the schema lives. JSON ships it inside every message (that's what the field names are). Protobuf moves the names out into a shared file, leaving only tiny numeric tags on the wire. Avro moves the schema out entirely — nothing but raw values travel. Less on the wire means more agreement needed beforehand. You're not choosing an encoding; you're choosing where the agreement lives. Everything else — the sizes, the speeds, the tooling, the failure modes — falls out of that.

Look at a JSON message the way the network sees it:

{"user_id": 1234567, "premium": true, "name": "Ada"}

Every character is a byte. "user_id": — ten bytes that say nothing about this user; they're the schema, riding along. The number 1234567 travels as seven digit-characters, not as a number. The braces, quotes, and commas are structure-as-text. This one record: ~47 bytes, of which the actual information — one number, one boolean, three letters — is maybe a quarter.

Before the bytes make JSON look foolish, say clearly why it conquered the world, because these strengths are real engineering properties, not conveniences:

• Zero coordination. Any client that can parse text can consume your API — no shared files, no build step, no version handshake. For a public API, where you'll never meet your callers (lesson one!), self-description isn't waste; it's the whole contract traveling with every message.
• Human-readable. curl it, read it, spot the bug. At 3am, during an incident, "I can see the payload" is worth real money.
• Universal. Every language, browser, log pipeline, and debugging proxy speaks it natively.

The costs are the mirror image: the per-message name overhead forever; text-parsing on every read (scan characters, match quotes, convert digit-strings to numbers — optimized to death in modern runtimes, but physics is physics); and a type system with real holes — no distinction between int and float, no safe 64-bit integers (JavaScript rounds beyond 2⁵³ — real bug, real production incidents), no binary data without base64's +33% tax. JSON is a fine default. It's just not free, and above a certain message volume the bill becomes visible.

Protocol Buffers' move is surgical: keep the values on the wire, move the names out. Both sides agree — ahead of time, in a shared file — on a schema:

message User {
  int64  user_id = 1;
  bool   premium = 2;
  string name    = 3;
}

The = 1 is not an assignment — it's the field tag, and it's the entire trick. On the wire, each field is: one tag byte (the field number plus a wire-type hint), then the value. "user_id": 1234567 — thirteen-plus bytes of JSON — becomes tag 0x08 plus a varint: ~4 bytes total. The whole record: ~11 bytes (seven value bytes + three tags + one length byte) against JSON's 47.

Two encoding ideas do the compression-without-compressing:

• Varints — integers take only as many bytes as they need. Each byte carries 7 bits of number plus a "more coming" bit: 0–127 fits in one byte, 300 in two, and your typical enum/count/flag — small numbers dominate real data — almost never pays for the 8 bytes a fixed int64 would take. This is why numeric-heavy payloads collapse under protobuf.
• Strings stay strings — tag, length, then the raw UTF-8 bytes, same as JSON minus the name and quotes. Binary encoding does not shrink your strings. Hold that; it becomes the punchline of the benchmark section.

The schema file also buys the speed: protoc generates typed classes for your language, so parsing is "read tag, read that many bytes into a struct" — no text scanning, no reflection, no guessing. Benchmarks put protobuf at 3–6× faster to serialize and 5–10× faster to parse than JSON. The price: the wire is no longer human-readable (you need the .proto and a tool to even look at a message), there's a build step, and both ends must hold the schema — the agreement moved from the wire into your repo. (Where this shines brightest — service-to-service RPC — has a name and its own lesson: gRPC, two stops ahead.)

Protobuf still spends one byte per field on the wire — the tag — so a reader can locate fields without help. Avro asks the radical follow-up: what if we remove even that?

An Avro message is just the values, back to back, in schema order. Our record — varint(1234567), byte(1), len+"Ada" — is ~8 bytes (Protobuf minus the three tags), with nothing identifying any field. Smallest possible encoding; nothing on the wire but the data itself.

Which raises the obvious question: how does a reader make sense of naked bytes? It can't — not without the writer's schema. That's the design: Avro is schema-on-read. The agreement doesn't live in the message (JSON) or in per-field tags (protobuf); it lives outside the pipe entirely, and the reader must fetch it. In practice — and this is why you meet Avro everywhere in the Kafka world — that means a Schema Registry: each message carries a tiny 5-byte header with a schema ID; the reader fetches that schema from the registry once, caches it, and decodes everything after at full speed. The agreement became a service.

Three more traits complete the picture: Avro schemas are defined in JSON and don't require code generation (friendly to dynamic languages and data pipelines); it's row-oriented with a splittable container-file format (why big-data tooling adopted it); and its size/speed land just ahead of protobuf on the wire (no tag bytes) and just behind it on raw parse speed. If protobuf's habitat is service-to-service RPC, Avro's is the event stream and the data lake — places where the registry infrastructure already exists and billions of small messages make every byte count.

Everything above is claims about bytes — so here are the bytes. Below is a record you can edit, wired to three live encoders: every keystroke re-encodes into JSON, Protobuf, and Avro, byte by byte, drawn and colored.

Watch where the bytes come from. In the JSON strip, the amber cells are the field names — schema, riding inside your message; type a longer value and only the sky cells grow. In the protobuf strip, each amber name has collapsed into one violet tag byte; crank user_id from 42 to a million and watch the varint grow one byte at a time instead of one digit at a time. In the Avro strip there are no tags at all — just values — and its schema sits pointedly off to the side, in the registry, where its agreement lives.

Then run the experiment that most comparisons bury: flip between the numeric-heavy preset (IDs, timestamps, counts — binary wins ~4–5×) and the string-heavy preset (bios, URLs — the gap collapses to a few percent, because strings are strings in every format). Same three encoders, wildly different verdicts — the data's shape decides. Finish with the at-scale slider: bytes-per-message × messages/day → gigabytes and dollars per month. That number is how this lesson wins arguments in real meetings.

Benchmarks on serialization are a swamp of unlabeled axes, so here are the honest headline numbers with their conditions attached:

• Size: on numeric-heavy payloads (IDs, timestamps, metrics, enums), protobuf runs 4–5× smaller than JSON — one published benchmark: 8.7 KB vs 39.2 KB for the same data. On string-heavy payloads the advantage nearly vanishes — ~4% — because strings are stored as-is everywhere; the savings were always the names and the numbers. Typical real-world APIs land between: 50–85% smaller. Avro shaves a further tag-byte per field below protobuf.
• Speed: protobuf serializes ~3–6× faster and parses ~5–10× faster than JSON — no text scanning, no reflection, straight into generated structs. Avro sits close behind.
• The confound everyone forgets: compression is orthogonal. gzip on JSON squeezes out much of the redundancy (those repeated field names compress beautifully), narrowing the size gap on big payloads — at the cost of CPU on both ends. Binary formats + compression stack. Any benchmark that doesn't say whether compression was on is telling you very little.

And translate to what actually matters: at 10 requests/second, none of this is worth a meeting — JSON's overhead is rounding error. At 50,000 messages/second between internal services, 60% smaller × parse-speed × egress pricing is latency, capacity, and a line item on the cloud bill. The formats don't have a winner; the workload picks one.

Strip the syntax and you're placing one thing — the schema — in one of three homes, each with rent:

The honest decision procedure:

• Strangers call you? (public API, unknown clients, humans debugging) → JSON. Self-description is the feature; universality beats bytes. Add gzip before you add a binary format.
• You control both ends and volume is real? (internal service-to-service) → Protobuf. The build step is cheap when it's your own CI, and 50–85% × millions of messages is real capacity.
• Messages outlive their producers? (event streams, Kafka topics, files in a lake read years later) → Avro. The registry holds the agreement independent of any running service — which is exactly what long-lived data needs.

Notice what all three homes have in common: the agreement exists either way. JSON doesn't remove the schema — it just hides it in your validation code and your docs, unenforced. The binary formats make it explicit and machine-checked. Which raises the question that decides careers: what happens when the agreement has to change — new fields, renamed fields, old readers meeting new data? That's the next lesson, and it's the real reason schemas matter.

• "JSON is bloated legacy; binary is what professionals use." JSON's self-description is a capability — zero coordination, universal tooling, readable payloads — and for public APIs it's usually the right call. Professionals match the format to the workload, not to the fashion.
• "Protobuf compresses my data." It doesn't compress anything — it encodes less: names become tags, numbers become varints. Your strings travel at full size, which is why string-heavy payloads save only ~4%. Compression (gzip/zstd) is a separate decision that stacks on top of any format.
• "Binary means undebuggable." It means the readability moved — from the wire into schema-plus-tooling (protoc --decode, registry UIs). With the schema, a binary payload is perfectly inspectable; without it, admittedly, it's noise. That's the trade you signed.
• "Avro is protobuf with extra steps." Architecturally opposite: protobuf keeps per-field tags on the wire so any reader with the .proto can decode a message in isolation; Avro strips even the tags, so the reader is helpless without the writer's schema — in exchange for the smallest possible bytes. Tags-on-wire versus schema-registry is a real infrastructure decision, not a syntax preference.
• "We'll just switch formats later if we scale." The bytes are easy to switch; the agreement isn't — schemas calcify exactly like contracts do (lesson one). Starting with JSON is fine; starting without thinking about where your schema lives is how "later" becomes a migration project with its own name.

Thirty minutes, and the bytes stop being abstract:

1. Weigh a real payload. Take an actual JSON response from your own API (or any public one). wc -c it, then: gzip -c payload.json | wc -c. Notice how much of JSON's "bloat" the compressor already recovers — that's the repeated field names being squeezed. This number is your honest baseline before any binary migration.
2. Encode the same data in protobuf. Write a five-line .proto for it, then — no code needed — use protoc --encode=YourMessage your.proto < payload.txt | wc -c (protobuf's text format in, binary out). Compare all three numbers: raw JSON, gzipped JSON, protobuf. Then gzip the protobuf too and see the stack.
3. Read the wire. Pipe your binary through protoc --decode_raw — you'll see field numbers and values, no names. That's what the wire knows: this is both why it's small and why you need the .proto. Now you've personally verified where the schema went.
4. Break the agreement on purpose. Decode the message with a wrong .proto — swap two field numbers, or change an int64 to a string. Watch it misread silently or explode. Sit with how much trust the wire is placing in the shared schema… and how nothing on the wire itself can check it.
5. Bonus, varint by hand: encode 300 — 100101100 → two groups of 7 bits, little-endian, MSB continuation: 0xAC 0x02. Verify against --decode_raw. Ten minutes, and varints are yours forever.

Step 4 is the cliffhanger made physical: the schema is load-bearing, both sides must agree, and changing it without breaking anyone is a discipline of its own — next lesson.

Serialization is where your API's abstractions become bytes, and the three big formats are one decision wearing three costumes — where does the schema live?

• JSON ships the agreement in every message: field names on the wire, readable by anything, zero coordination — and you pay the name-tax per message, parse text forever, and live with a loose type system. The right default for public APIs and humans.
• Protobuf moves names into a shared .proto, leaving 1-byte tags and varints: 50–85% smaller (4–5× on numeric data, ~4% on strings — shape decides), 5–10× faster parsing via generated code. The internal-RPC workhorse, and the fuel for gRPC.
• Avro removes even the tags — values back-to-back, schema in a registry: smallest bytes, and the reader must hold the writer's schema. The event-stream and data-lake native.
• Compression is orthogonal and stacks; gzip your JSON before you migrate formats. And the workload picks the winner — at low volume this is a non-decision; at millions of messages it's latency and money.

The thread through the section so far: L1 wrote the contract, L2 made walking it honest, L3 made repeating it safe — and today you learned the contract's physical form is a schema both sides must share. Which sets up the scariest question in API design: schemas don't stay still. Fields get added, renamed, retired; old readers meet new data and new readers meet old data — often mid-deploy, in both directions at once. Changing the agreement without breaking anyone — backward and forward compatibility, and the rules that make evolution safe — is the next lesson.