The Fallacies of Distributed Computing

Last lesson ended with an accusation: the network lies. We proved it with numbers — links and devices failing every day inside a fleet its own engineers call "highly reliable," a research network that logged 508 isolating partitions in five years, two "independent" datacenters that turned out to share one router. Welcome to Hard Mode left you with the general truth. This lesson makes it specific.

Because the network doesn't lie in some vague way. It lies in eight particular ways — eight assumptions so natural, so obviously true, that you will make every one of them without noticing, the first time you split a program across two machines. They have a name. In the 1990s at Sun Microsystems, engineers watched newcomers to distributed systems make the same wrong bets over and over, and they wrote them down. Peter Deutsch and James Gosling are the names most attached to the list — Gosling, Java's creator, added the eighth in 1997 — and it has been quietly correct for thirty years:

Here is the thing to understand before we take them one at a time, because it's the whole reason the list exists. Every one of these eight statements is true on a single machine. On one box, the "network" between your function and the data it calls is a few nanometers of silicon: it never drops a call, latency is genuinely near zero, bandwidth is the memory bus, nobody is sniffing your function arguments, the "topology" is fixed, there's one operating system in charge, a call costs nothing, and everything speaks the same language. You spent years writing code where all eight were safe to assume — so of course you assume them. Then you move one function call onto a wire, and all eight flip from true to false at once, silently, and your code doesn't get a warning.

In this lesson: the eight, in four groups that make them stick — the mother lie, the physics lies, the adversary lie, and the not-yours-to-command lies — what each one costs you when you believe it, and, in the interactive, the skill that actually matters: spotting which one a design is unconsciously betting on.

Scope: this is the catalogue of the lies, not the toolbox of cures. Each fallacy names its fix and points to the lesson that teaches it — retries and circuit breakers, service discovery, encryption, efficient serialization — but here we're learning to see the assumption, which has to come first.

We start where the list starts, and we can move fast, because you already watched this one die.

The network is reliable. Switches fail — even a basic one has a mean-time-between-failure measured in tens of thousands of hours, which sounds like forever until you have a thousand of them and someone trips over a cable. Power blips. A wireless client wanders behind a wall. A deploy misconfigures a route. A partner's credit-card service — the half of the connection you don't control — goes down. This is fallacy number one, and it's the mother of the list because every other fallacy is a special case of the world not behaving as reliably as your single-machine instincts expect.

You met the evidence in Welcome to Hard Mode, so we won't re-litigate it. What matters here is the shape of the mistake in code: a program that assumes the network is reliable is a program with no retry, no timeout, no circuit breaker — one that makes a call over the wire exactly the way it makes a local call, and simply crashes when a switch reboots mid-request instead of trying again. The cure is a whole discipline of its own — retries with backoff, timeouts, circuit breakers, idempotency so a retry doesn't double-charge anyone — and it has its own home later in this course, in Resilience & Overload Control. For now, just carry the naming: the network is not reliable, and pretending otherwise is the first bet every distributed system loses.

The next three fallacies are a matched set, and grouping them is the trick to remembering them: they're all the same wish — that the wire is free — told three different ways. It isn't free in time, it isn't free in capacity, and it isn't free in money. Physics and accounting both send a bill.

Latency is zero. This is the one that fools senior engineers, so slow down. Latency is the time for a message to travel; on one machine it's the memory bus, effectively instant, so a program that makes a thousand tiny calls to nearby objects runs fine. Move those objects onto another continent and the same thousand tiny calls become a thousand round trips across an ocean, and your snappy app grinds to a halt. And here's the part that makes latency worse than any other fallacy: you cannot buy your way out of it. Latency has a floor set by the speed of light, and no purchase order moves it. New York and London sit about 5,600 kilometres apart — and the cables run longer; light in glass crawls at roughly two-thirds of its vacuum speed, so the round trip has a hard floor near 56 milliseconds. The fastest transatlantic cable ever laid — built specifically so traders could shave microseconds — gets a round trip down to about 59 ms, and your ordinary production traffic sees 70 to 90. You can buy a fatter pipe by tomorrow afternoon; you cannot buy a shorter path for light. The fix isn't a faster network; it's fewer, fatter calls — batch, cache, and move work next to the data — and the numbers behind it live in Numbers to Know over in Fundamentals of System Design.

Bandwidth is infinite. The weakest of the eight, because bandwidth genuinely does keep getting cheaper — and still a fallacy, for two reasons. First, our appetite grows to fill any pipe (video, telemetry, chatty JSON, one real project firehosing more than three terabytes of new data a day). Second, and less obvious: on a real wide-area link, packet loss caps your throughput far below the link's rated speed. The classic calculation makes it vivid — push data New York to Los Angeles over a link with a 40-millisecond round trip and a mere 0.1% packet loss, and classic TCP's throughput tops out around 6.5 Mbps. Not because the link is slow, but because latency and loss together govern how fast TCP is willing to send, no matter how fat the pipe. Modern congestion control (the BBR family) has raised that ceiling a long way — but the tax never went to zero: loss and distance still set what you actually get, and it is still routinely a fraction of the number on the invoice. Believing bandwidth is infinite is how you ship an endpoint that over-fetches a giant payload nobody paginated — fine on your laptop's loopback, a bottleneck the day real traffic hits. The cures — compression, efficient formats, pagination — you already met in Data Formats & Serialization: JSON, Protobuf, Avro and the pagination lesson, back in Fundamentals of System Design.

Transport cost is zero. The wire bills you twice. In CPU: every call has to serialize your objects into bytes and deserialize them on the far side — marshaling that costs real cycles and adds real latency, on top of the trip itself. And in cash: cloud providers charge by the gigabyte to move data between regions and out to the internet, a number that rounds to zero on the pricing page and becomes a five-figure surprise when a backup job quietly replicates terabytes across regions every night. The pricing page hides it well: the first hundred gigabytes a month are free, then it's about nine cents a gigabyte to send data out to the internet. Nine cents sounds like nothing — until you notice that a terabyte a day out the door is roughly $2,700 a month, and every region you replicate across adds its own per-gigabyte toll on top. "Someone, somewhere will have to pick the tab and pay these costs," as the original essay put it. The money side gets its own lesson, Cloud Cost Engineering, in the next course.

Fallacy number four stands alone because it's the only one where the network isn't merely indifferent to you — it's contested. Someone else is on the wire, and some of them are hostile.

The network is secure. On one machine, your function arguments are private by construction — nobody is between the caller and the callee. Put that call on a network and the bytes cross cables, switches, and routers you don't own, any of which can be tapped. Send a password, a token, or a customer's data in the clear and you should assume someone read it. The classic failure is the "moat and castle": teams that harden the perimeter and then trust everything inside it, so the first attacker who gets past the wall walks the whole datacenter freely. And this is not a rare tail risk you're insuring against — IBM's long-running study of real breaches prices the **average incident at about $4.4 million** (2025; it peaked near$4.9M the year before). The security researcher Gene Spafford's old line still sets the honest baseline — "the only truly secure system is one that is powered off, cast in a block of concrete and sealed in a lead-lined room with armed guards — and even then I have my doubts" — which is another way of saying security is never done, it's a budget you keep spending. In practice that means encrypting everything in transit and at rest, and trusting the network segment no more than the open internet — the machinery (TLS, mutual TLS, zero-trust, secrets management) is taught in Security & Cost, in the next course, and builds on the identity basics from Fundamentals of System Design. The naming to carry: the wire is hostile by default; privacy is something you add, never something you inherit.

The last three fallacies share a single delusion — that the network is a thing you control, like the box on your desk. It isn't. It reshapes itself under you, other people run pieces of it, and it's a patchwork you didn't design.

Topology doesn't change. In the test lab it doesn't. In production it never stops: operations add and retire servers, a box fails over to a standby, and in the cloud autoscaling and rolling deploys rearrange the map by the minute. The mistake this produces is beautifully concrete — a hardcoded IP address, or a specific hostname, baked into a config. It works perfectly until the morning that machine moves — and then it fails, and nothing in the code explains why. The oldest fix in the book is one you use every day without thinking: a DNS name instead of an IP, so the address can change underneath you and callers still find the service — and its grown-up form, service discovery, is taught with the Microservices Patterns in the next course.

There is one administrator. On your laptop, you're root. Across a real system, no single person controls the whole path. Internally, different admins own the databases, the network, the Windows boxes, the Linux boxes. Externally, your packets cross internet providers and other companies' networks that answer to nobody you can email, and if your app talks to a partner, they run their half with their own rules, quotas, and firewalls. The cost of forgetting this shows up at 3 a.m. as the worst question in operations — "is it their problem or ours?" — and the defenses are structural: decouple systems so one team's change can't silently break another's, and instrument everything so the answer to that question takes minutes, not hours. That instrumentation is the whole subject of Observability, in the next course.

The network is homogeneous. Even a home network is a zoo — a Linux media box, a couple of Windows laptops, a NAS, a phone, all on the same Wi-Fi. A real system is worse: fiber and cellular and Wi-Fi in the path, different hardware, different operating systems, and — the part that actually bites — different protocols and data formats at the application level. Assume everything speaks your dialect and you'll ship a proprietary format that can't talk to the partner you integrate with next quarter, or an app that's crisp on fiber and unusable on a train's cell signal. The defense is boring and correct: prefer open standards and common formats — HTTP, JSON, Protobuf, gRPC — so the heterogeneity you can't prevent is at least something everything already knows how to speak.

Naming the eight is the easy part. The skill that pays your salary is spotting them — catching the quiet assumption in a design before it ships, when it's still a one-line fix instead of a 3 a.m. page. That's a muscle, and you build it by reps.

Try this first: take the first architecture, look for the assumption it's leaning on — a hardcoded address? a chatty loop? a token in the clear? — and commit to which of the eight it is before you reveal it. You'll be wrong on a couple, because the subtle ones are subtle on purpose. Watch what the wrong bet actually does when the widget runs the failure, then move to the next. By the eighth you'll be reading designs the way a reviewer does.

You should be pushing back by now, because a lot of this reads like it's from 2006 — and it is. The fallacies were written when you racked your own servers. Today a managed load balancer retries for you, a service mesh encrypts every hop for you, autoscaling reshapes the topology automatically, and the cloud provider runs the network so you never see a switch. So the fair question is: didn't the cloud just solve these?

It didn't. It moved them — and, worse, it hid them behind a green dashboard, which means they bite harder the day they finally surface. Every fallacy is still there; it just changed owners:

• The network is reliable — the managed load balancer does retry for you, but someone set that retry budget, and a retry storm that takes down your backend is still entirely yours to cause.
• Transport cost is zero — this is the one people call obsolete, and they're half right: the CPU cost of serializing got cheap. The cash cost went the other way. Egress is now a line item that can rival your compute bill, and you can't see it until it arrives.
• Topology doesn't change — autoscaling doesn't stop topology from changing, it changes it more, by the second. The mesh re-discovers services for you, right up until a connection pool holds a stale handle to a node that scaled away, and that's a real outage.
• The network is secure — mutual TLS everywhere is one mesh setting away. It is also one misconfiguration away from off, and nobody gets paged for encryption that silently isn't happening.
• There is one administrator — you have fewer admins to hire and one enormous one you can't email. When their control plane in one region has a bad morning, it is your outage, and you'll read about the anatomy of exactly that kind of day near the end of this course.

The deeper reason the fallacies didn't age out is simple: we build far more distributed systems than we did in 1997 — microservices, serverless, edge, multi-region, model inference at scale — so the eight assumptions get violated more often, more dramatically, by more engineers who never had to learn them because the abstraction was so smooth. Until the morning it wasn't. The classic essay that taught a generation these fallacies put it right in its subtitle: the more things change, the more they stay the same.

• Eight assumptions, all true on one machine, all false across a network — and you'll make every one by default, because you learned to program where they were safe.
• The mother lie: the network is reliable. It isn't; design for lost calls (retries, timeouts, circuit breakers).
• The physics lies — the wire is free: latency is zero (no — and you can't buy your way past the speed of light; make fewer, fatter calls), bandwidth is infinite (no — loss caps real throughput; compress and paginate), transport cost is zero (no — it costs CPU to serialize and cash to move; watch egress).
• The adversary lie: the network is secure. The wire is hostile; encrypt everything, trust no segment.
• The not-yours-to-command lies: topology doesn't change (use names/discovery, not hardcoded IPs), there is one administrator (decouple + instrument; expect "their problem or ours?"), the network is homogeneous (use open standards).
• The real skill is spotting them in a design before it ships — that's the rep you just did.

The fallacies tell you what you wrongly assumed. They don't tell you how a system actually breaks when the assumption fails — whether a node stops cleanly, lies, or just goes slow. Naming those failure shapes is next, in Failure Models: Crash, Omission, Timing.