Single Point of Failure — and Redundancy, the Antidote

The last lesson handed you a whole toolkit for failing without falling — timeouts, breakers, bulkheads, graceful degradation. This lesson zooms all the way in on the simplest and most dangerous fault of them all, and its bluntest, most powerful antidote.

The fault is the single point of failure — one component, sitting quietly on your critical path, with no backup, whose death is the death of your entire system. Not a slow degrade, not a partial outage: everything, dark, because of one box. Every experienced engineer has a war story about the server nobody remembered, the single database, the one load balancer — the thing that took the whole product down at the worst possible moment.

The antidote is almost insultingly simple to say and surprisingly deep to do: don't have just one of anything that matters. Duplicate it. That's redundancy, and it's the foundation under high availability, replication, failover, and half of what makes large systems survivable.

By the end of this lesson you'll be able to do the single most useful reliability exercise there is: look at any architecture and hunt its single points of failure — find every box whose death is fatal — and know how to neutralize each one. You'll also learn why you can never quite kill the last SPOF, why a backup you've never tested might as well not exist, and why duplicating a database is a thousand times harder than duplicating a web server. Let's start hunting.

A single point of failure is any component that satisfies two conditions: it sits on the critical path (a request needs it to succeed), and there is only one of it. Kill it, and the whole system goes down with it. It's the weakest link in a chain — and a chain is only ever as strong as that one link, no matter how strong the others are.

Finding SPOFs isn't mystical. You trace the path a request takes through your system, and at every single stop you ask one question:

"If this one component dies right now, does the whole system stop?"

If the answer is yes and there's only one of that component, you've found a single point of failure. Walk a typical request — client → DNS → load balancer → app server → database — and ask it at each hop. In a naive system, most of those hops are SPOFs. The usual suspects:

• The load balancer. Everything routes through it; if the one LB dies, nobody reaches the servers behind it — even though every server is perfectly healthy.
• The database. One database, no replicas: it goes down and there's no data to serve, so the whole service halts.
• DNS. Rely on a single DNS provider and its outage is your outage — in 2016 an attack on the DNS provider Dyn took down Twitter, Reddit, Spotify, and half the internet with it.
• The one everyone forgets. A shared config service, a single NAT gateway, an auth service, that one VM someone spun up in 2019 that three critical things quietly depend on.

That last category is the dangerous one. The SPOF that takes you down at 3 a.m. is almost never the one you knew about — it's the one you didn't. Which is exactly why you hunt them on purpose, before they hunt you.

Every single point of failure has the same cure: stop having exactly one of it. Run more than one, so that no single death is fatal. That's redundancy, and it comes in two classic shapes you should be able to name on sight.

Active-active. All the copies are running and serving traffic at once, with a load balancer spreading requests across them. Lose one and you lose a slice of capacity — not the service. The survivors just pick up the load (you met this exact idea in the horizontal-scaling lesson: a stateless fleet where any box can serve any request). It uses all your hardware and there's no "switch-over" moment, but it only works when the work is spreadable — which is why statelessness matters so much.

Active-passive. One copy does the work while a standby waits in the wings, idle, as a backup. When the active one dies, the system fails over — it promotes the standby to active. It's simpler and sidesteps some nasty problems (more on those soon), but the standby's capacity sits wasted until it's needed, and there's a brief lag while the switch happens.

The sizing rule of thumb is N+1: run at least one more instance than you actually need, so you can lose one and still have enough. Need three app servers to handle your load? Run four. Want to survive two simultaneous failures? That's N+2. And you apply this everywhere it matters — a redundant load balancer, redundant app servers, database replicas, multiple DNS providers — because a system is only as available as its least-redundant critical component. The best part: you already know how to make the app tier redundant, because you made it stateless. The database is a different, much harder story — we'll get there.

Redundancy is only useful if the system actually uses the backup when the primary dies — and does it fast, without a human being paged at 3 a.m. That switch-over is called failover, and a good one has three distinct jobs:

1. Detect. Notice the failure, automatically and quickly. This is what health checks and heartbeats are for — the load balancer pings each server, the standby watches the primary's pulse. The faster you detect, the shorter the outage (remember MTTR from the availability lesson — detection time is a big chunk of it).
2. Switch. Route around the dead component: stop sending traffic to the failed server, or promote the standby to primary. In active-active this is nearly instant (the LB just stops picking the dead one); in active-passive there's a promotion step.
3. Recover redundancy. This one is easy to forget and dangerous to skip. After failover you're running on your last copy — you've spent your spare. A good system immediately rebuilds the redundancy (spins up a replacement, re-syncs a new standby) so you can survive the next failure, not just this one. A redundant pair that becomes a lone survivor and stays that way is a SPOF again, wearing yesterday's success as a disguise.

The dream is that failover is so fast and automatic that users never notice — the request that would have failed simply lands on a healthy box instead. But "automatic and fast" hides a trap that has burned every serious operations team, and it's coming up. First, the fun part: go find the SPOFs yourself.

Reading about SPOFs teaches you the words; killing components yourself teaches you the instinct. Below is a live system — Client → DNS → Load Balancer → App → Database — with requests flowing through it. Your job is to find every single point of failure and harden them out.

• Click a component to 💥 kill it. If the system was relying on that one box, the flow stops cold and the verdict flips to SYSTEM DOWN — you just found a SPOF.
• Click ＋ add redundancy on a component to duplicate it. Now kill one of the pair and watch the traffic fail over to its twin while the system stays UP — that death went from fatal to survivable.
• Keep going until the verdict reads "survives any single failure."

Watch for the twist: the moment you duplicate the load balancer, notice what's now the single thing everything points at. Removing one SPOF has a way of promoting the next one — that's not a bug in the sim, it's the deepest truth in this whole lesson, and it's what we'll talk about next.

Here's the truth the hunt reveals that catches everyone off guard: redundancy doesn't delete a single point of failure. It moves it.

Watch the SPOF migrate up the stack. You have one app server — that's a SPOF, so you run three behind a load balancer. Great — except now the load balancer is the single thing everything routes through, so it is the SPOF. Fine: you run two load balancers. But now, what points traffic at the two load balancers? Something has to — a DNS record, a floating virtual IP, an anycast route — and that is now your single point of failure. Duplicate that, and you'll find the next one behind it.

It's turtles all the way down. Each time you kill a SPOF, you expose the one it was hiding. And you can't chase it forever, because eventually you reach a shared-fate floor you can't cheaply climb past: everything is in one availability zone (so the zone is the SPOF), or one region, or one cloud provider, or ultimately depends on DNS and the public internet themselves.

So the goal was never "eliminate all single points of failure" — that's impossible, and chasing it bankrupts you. The real, achievable, professional goal is this:

Survive any single failure. No one component whose death takes everything down — and the residual shared-fate risk pushed down to something rare enough (a whole region, a whole provider) that you accept it, or pay a lot more to span it.

"Any single failure" is a bar you can actually hit and verify. "No failures ever" is a fantasy you go broke chasing.

Two copies on the diagram is not the same as two copies in reality. Redundancy fails, silently and completely, in two ways that a system's own dashboard will happily hide from you.

It isn't redundant if the copies share a fate. Two "redundant" servers in the same rack die together when that rack loses power. Two instances in the same availability zone die together when that AZ floods. Two copies deployed from the same bad release crash together the instant that bug runs. This is the correlated-failure lesson coming back to collect: the parallel-availability math only works when failures are independent, so real redundancy lives in different failure domains — different racks, zones, regions, even providers. A two-engine plane is beautifully redundant right up until you learn both engines share one fuel line. Count independent copies, not copies.

It isn't redundant if you've never tested the failover. A standby that has never actually taken over is Schrödinger's backup — simultaneously working and broken until the moment you observe it, which is the middle of a real outage. It might be misconfigured, running stale data, missing a firewall rule, or itself dead. The industry's blunt proverb: an untested backup is functionally the same as no backup at all. Worse, teams that do practice failover almost never practice failback — running on the secondary long enough to switch cleanly back — so that path gets executed cold, under maximum pressure. The only fix is the habit from the last lesson: drill it on purpose. Deliberately kill the primary on a calm Tuesday and watch the standby take over — because the alternative is discovering it doesn't during the storm.

There's a reason we keep saying the app tier is easy to make redundant and the database is hard. Duplicating a stateless server is trivial: the copies are interchangeable, any one can serve any request, and if one dies the others already know everything they need (because they know nothing — the state lives elsewhere). That's the whole payoff of the statelessness lesson.

But the moment a component holds state — a database, above all — redundancy gets genuinely dangerous, because now the copies have to agree on what's true. Run two databases that can both accept writes, and picture a network hiccup that cuts them off from each other. Neither can tell whether the other is dead or just unreachable — so each one, reasonably, assumes it's now in charge and keeps accepting writes. Now you have two "primaries" independently editing the same data, and when the network heals you have two divergent, conflicting versions of the truth and no clean way to merge them. This is split-brain, and it can be worse than an outage: an outage loses you time, split-brain loses you data and trust.

The fixes are the beginning of a much bigger story. You can require a quorum — a node may only become primary if it can reach a majority of the cluster, so a minority partition can't crown itself. Or you can fence the old primary — forcibly power it off or cut its network before promoting the standby (the wonderfully named STONITH: "Shoot The Other Node In The Head"). Making redundant, stateful systems agree on a single version of the truth — safely, through failures and partitions — is one of the deepest problems in all of computing. It's the entire reason the data-systems course exists, and we'll spend real time there. For now, just carry the instinct: duplicating stateless things is easy; duplicating state is a whole discipline.

Redundancy is not free, and the reflex to duplicate everything is as wrong as the mistake of duplicating nothing. Every copy you add costs money and complexity, and — as with the whole last lesson — the machinery you add to survive failures is itself something that can fail.

The choice	You gain	You pay
Active-active	all hardware used, instant failover	needs statelessness; risks split-brain
Active-passive	simpler, split-brain-free	the standby sits idle/wasted; failover lag
More redundancy	survive more simultaneous failures	~N× cost + failover code that can break

The active-active vs active-passive call is a real trade, not a ranking: active-active wrings value out of every machine and fails over instantly, but demands stateless design and opens the door to split-brain; active-passive is simpler and safer but pays for a standby that does nothing until disaster strikes. And piling on more redundancy has diminishing returns — past a point, the extra failover logic and coordination cause more outages than they prevent, and you still can't beat the shared-fate floor.

So redundancy is triage, priced with the economics from the availability lesson: spend it on the critical path — the components whose death is genuinely fatal and genuinely likely — and let the truly disposable stay single. The question is never "can we make this redundant?" It's "what does this component's death actually cost, and is a second copy cheaper than the outage?" Duplicate the load-bearing walls. You don't need a backup for the welcome mat.

"Two copies means we're safe." Only if they're independent (different racks/zones/regions) and the failover has actually been tested. Two copies in one rack are one SPOF you paid for twice.

"We have a standby, so we're covered." An untested standby is Schrödinger's backup — you find out it's broken during the outage. An untested backup ≈ no backup.

"Active-active is just better than active-passive." It's a trade-off: active-active uses all your capacity and fails over instantly but risks split-brain; active-passive is simpler and safer but wastes the standby. Pick per situation.

"Add redundancy and the SPOF is gone." You moved it — whatever now routes to or coordinates the redundant pair (the LB, DNS, the failover controller) is the new SPOF. Turtles all the way down.

"The database is just another server to duplicate." It's stateful — duplicating it needs replication plus a single writer or consensus, or you get split-brain. A whole discipline, not a checkbox.

"Our system has no single points of failure." Almost every system does — the forgotten shared service, the one NAT gateway, the single DNS provider. The deadly SPOF is the one you didn't know you had; hunt them explicitly.

"More redundancy is always more available." Past a point the failover complexity and split-brain risk cause more outages than they prevent — and you still can't beat the shared-fate floor. Redundancy is triage, not a maximum.

Fifteen minutes turns this into a reflex you'll use in every design review for the rest of your career. Predict first: think of an app or service you've worked on. What's the one component whose death would take the whole thing down?

1. Draw the path and interrogate it. Sketch a request from the user to the data and back — client, DNS, load balancer, servers, database, any third-party API. At each box ask the one question: "if this dies, does everything stop?" Circle every yes-with-only-one-of-them. That circle is your SPOF list, and it's almost always longer than you expected.
2. Follow the turtle. Take your scariest SPOF and mentally make it redundant. Now ask: what's the new single thing that points at or coordinates the pair? Keep going until you hit the shared-fate floor (an AZ, a region, a provider). Notice where you'd actually choose to stop — that's a real engineering decision, priced against cost.
3. Test one backup for real. Find something in your world that has a "backup" or "standby" — a replica, a failover, a restore-from-backup procedure. Ask the uncomfortable question: when was it last actually tested? If the honest answer is "never," you've found a Schrödinger's backup, and you now know it might as well not exist.

Do this once on a system you care about and you'll never look at an architecture diagram the same way — you'll see the skulls hovering over every box that only appears once.

The simplest fault and its bluntest antidote, in seven lines:

• A single point of failure is a component on the critical path with only one of it — its death is the whole system's death. Hunt them with one question: "if this dies, does everything stop?"
• The antidote is redundancy — don't have just one of anything that matters — in two flavors: active-active (all serve; lose one, lose capacity not service) and active-passive (a standby, promoted by failover).
• Failover = detect → switch → recover the spare (or your survivor becomes a SPOF again).
• Turtles all the way down: redundancy moves the SPOF up the stack, so the real goal is survive any single failure, not eliminate all failure.
• Redundancy is only real if copies are independent (different failure domains) and the failover is tested — an untested backup is no backup.
• Duplicating stateless things is easy; duplicating state invites split-brain and needs quorum/consensus — a whole discipline (the data course).
• Redundancy is triage — spend it on the load-bearing walls, priced against the cost of the outage.

That's the whole arc of failure: what availability is (the nines), how to fail gracefully (the toolkit), and how to survive the death of any single thing (redundancy). Your system is now available and resilient. But a system that's up isn't necessarily good — the next questions are about speed and volume. How fast does a single request come back, and how many can you handle at once? Those turn out to be two very different things that people constantly confuse. Next: Latency vs Throughput vs Bandwidth.