Facebook, 2021: The BGP Self-Lockout

On October 4, 2021, at 15:39 UTC, Facebook disappeared. Not slowed down, not degraded — gone, along with Instagram, WhatsApp, Messenger, and Oculus, for roughly 3.5 billion people at once. And it stayed gone for about six hours, which is an eternity at that scale. Here's the part that should stop you: Facebook employs some of the most capable infrastructure engineers on the planet, they knew within minutes what had broken, and for those six hours they still could not fix it. Some of them reportedly couldn't even get into the buildings.

That paradox is the whole reason this lesson exists. A system doesn't stay down for six hours because nobody knows what's wrong. It stays down because the failure has quietly severed the very paths you'd use to repair it. This outage is the cleanest example in modern computing of a system that locked itself out — one where a single, ordinary command didn't just break the service, it deleted the company from the internet and then hid the keys.

You've spent this whole part of the course building the tools that failure demands: partition tolerance, health checks, retry budgets, blast-radius containment, disaster recovery, incident response. Now we watch those exact ideas play out in one real event, at the worst possible scale, with billions of dollars and billions of people on the line. Read it as a detective story, because that's what a good postmortem is.

In this lesson: how one command took down a global backbone; how withdrawing a handful of routes erased Facebook from the internet; the circular dependency that made recovery so slow; why bringing it back was as dangerous as the outage itself; and the single design question this leaves on your desk.

Scope: the general shape shared by all such failures gets its own treatment in Anatomy of a Cascading Failure; here we stay with this one event and what it teaches.

It started with routine maintenance. An engineer ran a command whose only job was to check how much spare capacity the global backbone had available, a read-only-sounding question you might ask a hundred times a year. The backbone, in Facebook's own words, is the network that ties everything together: tens of thousands of miles of fiber linking every data center to every other. It is the thing all the data centers stand on.

The command did not measure the backbone. It took down every connection on it. In an instant, the data centers could no longer talk to each other or to the smaller edge sites that face the public internet. The backbone didn't degrade; it went to zero.

Now, Facebook is not naive about this. They run an audit tool whose entire purpose is to catch exactly this class of mistake — a command that would do more damage than the operator intended is supposed to be stopped before it executes. But the audit tool had a bug, and the bug meant it failed to stop the command. So this is already two failures stacked, not one: a command that did the wrong thing, and the guard rail that was built to catch it sitting silently broken. Hold onto that pattern. Almost every giant outage is not a single failure but a safety mechanism that didn't fire sitting behind the triggering mistake. The backbone cut was, in the vocabulary of Network Partitions & Split Brain, the mother of all partitions: not one link lost, but the entire spine severed at once.

Losing the backbone is a bad internal day. It should not, by itself, make facebook.com vanish from your phone — your phone doesn't touch Facebook's backbone. To understand why the whole world lost Facebook, you have to follow one more step, and it's the step that gives this lesson its name.

The internet finds Facebook using two systems working together. BGP, the Border Gateway Protocol, is how networks announce I am here, route traffic for these addresses to me — Facebook's network, known as AS32934, constantly advertises the blocks of IP addresses it owns so the rest of the internet knows how to reach it. DNS is the phone book that turns facebook.com into one of those IP addresses. And this detail matters: Facebook ran its own DNS servers, and those servers sat at the edge sites, announcing their own addresses over BGP so resolvers everywhere could reach them.

Here's the mechanism. Facebook's DNS servers follow a safety rule: if I can't reach the data centers behind me, then I'm not healthy, and I should stop advertising myself so traffic routes somewhere better. It's a sensible rule: a health check, exactly the kind you met in Heartbeats & Failure Detection. But when the backbone vanished, every DNS server at every edge site simultaneously discovered it couldn't reach the data centers, decided it was unhealthy, and withdrew its BGP routes — including the routes to the DNS prefixes 185.89.218.0/23 and 129.134.30.0/23. Within minutes, Facebook's name servers had erased their own paths from the global routing table.

The DNS servers were fine. They were powered on, running, ready to answer. But there was no longer any route on earth that led to them. So when your phone asked where is facebook.com?, the resolvers of the world — Cloudflare's 1.1.1.1, Google's 8.8.8.8, your ISP's — looked for Facebook's name servers, found no route, and returned SERVFAIL. Facebook hadn't been hacked and its servers hadn't crashed. It had, very precisely, told the internet to forget where it lived.

Sit with the cruelty of this for a second, because it's the real lesson, not the trivia. The thing that turned a serious-but-internal backbone failure into Facebook no longer exists on the internet was a safety feature working exactly as designed.

Withdrawing your routes when you're unhealthy is good behavior in isolation. If one edge site loses its link to the data centers, you absolutely want it to stop advertising itself — otherwise the internet keeps sending users to a dead end. That rule protects users every normal day. The trouble is that it was written to answer a local question — am I, this one site, healthy? — and then it was asked by every site at the same moment, about a global failure. Each site answered correctly and honestly: no, I can't reach the data centers, I should withdraw. And the sum of all those correct local answers was a global act of self-erasure that no single site intended and no human approved.

This is a pattern worth carrying out of this course: a guardrail tuned for a small, independent failure can amplify a large, correlated one. The health check assumed that if it was cut off, the world was still fine and traffic should flow elsewhere. It had no concept of everything is cut off at once, and in that case the right move was the opposite of what it did. The mechanism didn't malfunction. It was obedient at the worst possible scale — which is a far more dangerous thing than a bug, because it passes every test you'd think to write.

So the diagnosis was quick. Engineers knew, early, that the backbone was down and the routes were withdrawn. The fix, in principle, is almost embarrassing: restore the connections, let the DNS servers see the data centers again, let them re-advertise their routes. Minutes of work — if you can reach the equipment.

They couldn't. And this is the heart of it, the thing that makes this outage a masterclass instead of a footnote. To fix the network, you need to reach the network's control plane, and the tools you'd reach it with were all running on the network that was down. Remote access to the routers went over that network — gone. The internal dashboards engineers use to see and change the system needed DNS to resolve their own internal hostnames — and DNS was exactly what had vanished — gone. The paging and communication tools, the runbooks, the chat where an incident is coordinated: many of them leaned on the same infrastructure, so they were degraded or gone entirely. Even the badge readers on the data-center doors are reported to have failed, because they, too, phoned home over the corporate network. The people rushing to fix the outage couldn't get into the rooms holding the machines.

This shape has a name, and once you see it you'll see it everywhere: a circular dependency. Your recovery path depends on the very thing you're trying to recover. When the system is healthy the loop is invisible, because everything works, so nothing reveals that your fix-it tools quietly ride on the thing they're meant to fix. The failure is what exposes it — and by then it's too late to redesign. The single most useful question this entire lesson hands you is this: does any part of my recovery path depend on the thing it recovers? The console below drops you into that exact moment. You're on call, the site is dark, and every obvious lever is a trap. Find the one that isn't.

The way out of a circular dependency is a path that shares nothing with the thing that's broken. Engineers call it out-of-band access: a way to reach the equipment that doesn't ride the production network, ideally with its own separate connectivity, its own power, and its own way of proving who you are. In the worst case it comes down to the oldest trick there is — a person standing physically in front of the machine, plugging a laptop straight into the router's console port, bypassing the network completely. That's the break glass in case of emergency path, and on that day it was the only one that worked.

Even that was slow, for a reason worth respecting: data centers are built to be hard to get into. High physical security, hardened rooms, routers deliberately difficult to modify even with hands on them — all excellent protection against attackers, and all friction against your own engineers on the worst day of the year. Facebook's out-of-band access reportedly leaned on services that themselves depended on the down network, so the supposedly independent path wasn't independent enough. The clock ran while people drove to buildings and worked through the security designed to keep everyone out.

And then came a second trap most people never think about: turning it back on is its own outage risk. With the whole service dark, power draw in individual data centers had dropped by tens of megawatts. Slam all of it back at once and you get a surge that can trip electrical systems, and a thundering herd of billions of clients hitting stone-cold caches and empty connection pools — enough to knock the service straight back down. So recovery had to be staged: bring capacity back region by region, let caches warm and load settle, then bring back more. Facebook credited its regular storm drills — deliberately taking regions offline to practice, the discipline you met in Chaos Engineering — for giving them the muscle memory to do this without a second collapse. Recovery isn't a light switch; it's a controlled descent, and Disaster Recovery: RTO & RPO is where you learned to budget for exactly this.

One more piece, because it breaks a comforting assumption: your outage is not contained to you. When Facebook's name servers went silent, the world's DNS resolvers kept getting asked where is facebook.com? — by billions of phones and apps and browsers that, as you learned in Retries, Budgets & Retry Storms, will not take an error for an answer. Every failed lookup was retried, aggressively, by software and by frustrated humans mashing refresh. Public resolvers like Cloudflare's 1.1.1.1 saw their Facebook-related query load spike to roughly 30 times normal — a retry storm aimed at systems that had done nothing wrong and couldn't answer anyway.

The spillover went further. Users who couldn't reach one app went looking for others, so Twitter, Signal, Telegram and the rest saw sudden traffic surges of their own. And thousands of unrelated websites discovered a dependency they'd forgotten they had: every Log in with Facebook button stopped working, because it quietly called back to the servers that were now unreachable. A small e-commerce shop with no connection to social media suddenly had users who couldn't sign in — collateral damage from someone else's backbone command.

This is the reach of a truly large failure: it doesn't respect your org chart or your perimeter. Which is the whole argument, made in blood, for containing blast radius by design — the cells and regions of Cell-Based Architecture & Deployment Stamps and Multi-Region Patterns. A failure that can take down everything, will.

You're not running a global backbone, so it's tempting to file this under interesting, not mine. Don't. The exact same shape hides in systems a thousand times smaller, and the questions it raises are ones you can act on this week.

Is your recovery path independent of the thing it recovers? Write down, honestly, how you'd fix your service if it were fully down. Now check each step: does it need the service? Does your deploy pipeline need the app it deploys? Does your monitoring die with the thing it monitors? Does logging in to your cloud console depend on a service hosted in that cloud? If the runbook's step one is open the dashboard and the dashboard is down, you have Facebook's problem in miniature.

Do you actually have an out-of-band way in? A break-glass path — a separate network, separate credentials that don't depend on the main identity system, a documented physical or console fallback, and more importantly, have you used it recently? An out-of-band path you've never tested is a story you tell yourself, not a capability you have. Facebook's wasn't independent enough, and they only found out during the fire.

Would a health check betray you at scale? Any rule that removes capacity when it looks unhealthy — withdraw the route, kill the pod, fail the node out — is safe when a small fraction trips it and dangerous when all of it trips at once. Ask what your automation does when everything looks unhealthy simultaneously, and whether removing everything is really the behavior you want in that case.

The honest caveat: you can't eliminate every circular dependency. Real systems are full of them, and chasing all to zero is neither possible nor worth it. The job is to find the load-bearing ones — the paths your recovery truly depends on — and make those independent, then test them by triggering real failures on purpose before a real one finds you first.

Here's a scenario to reason through before you read the answer. A mid-size company runs its whole stack in one cloud region. Their infrastructure-as-code, their deploy pipeline, their monitoring dashboards, their alerting, and their engineers' SSO login all run inside that same region. One night, a bad network config rolls out region-wide and everything goes dark. Where is the circular dependency, and what's the one change that would hurt the most on the night — and help the most?

Work it out, then check yourself:

• The trap: to push the fix, engineers need the deploy pipeline; the pipeline runs in the dead region. To even see what's wrong, they need the dashboards; also in the dead region. To log in and do any of it, they need SSO; same region. Every recovery lever depends on the thing that's down — the Facebook shape, exactly.
• What hurts the most on the night: the SSO dependency. If engineers can't authenticate, they can't touch anything, so the whole recovery is blocked at step zero, the equivalent of Facebook's badge readers failing.
• The change that helps the most: a genuinely independent break-glass path — a separate, minimal control plane (different region or provider) with its own credentials that don't depend on the main SSO, able to reach and redeploy the network config. Bonus: a monitoring signal that lives outside the region, so you can still see. The test that proves it: take the primary region offline on purpose, during business hours, and recover using only the break-glass path. If you've never done that drill, assume it doesn't work.

The story got compressed and mangled as it spread. The mangled versions are where the wrong lessons hide.

• Myth: it was a hack, or a BGP hijack by attackers. No. It was entirely self-inflicted: a routine maintenance command plus an audit tool too buggy to stop it. The scariest outages are usually own-goals, not attacks.
• Myth: DNS was the root cause. DNS going dark was the most visible symptom, not the cause. The root cause was the backbone being withdrawn; the DNS servers withdrawing their routes was a safety mechanism amplifying that root cause into total global invisibility. Blaming DNS is like blaming the smoke.
• Myth: they should've just logged in and reverted the command. That's the entire point of the lesson — the login path, the dashboards, and the remote access all rode the dead network. The circular dependency is why a two-minute fix took six hours.
• Myth: the real drama was the angle grinder and the locked doors. The badge-reader failure is a true and vivid illustration of shared fate reaching into the physical world, and it's worth remembering for that. But the widely repeated they cut into a server cage with an angle grinder detail is the least-verified part of the whole story, and Facebook denied it. The real story is the dependency, not the power tool.
• Myth: once BGP was restored, they could flip everything back on. No — a dark service means cold caches and dropped power draw, so slamming it all on at once risks a power surge and a thundering herd that causes a second outage. Recovery had to be staged and careful, which is part of why it wasn't instant even after the fix was known.

• A single command plus a broken guardrail took down a global backbone. Big outages are rarely one failure; they're a triggering mistake sitting behind a safety mechanism that didn't fire.
• Facebook deleted itself from the internet via BGP and DNS. When the backbone vanished, the edge DNS servers followed a health-check rule and withdrew their own routes, so resolvers worldwide could no longer find Facebook's name servers and returned SERVFAIL. The servers were fine; the way to reach them was gone.
• A local safety rule amplified a global failure. Withdraw your routes when unhealthy protects users every normal day; asked by every site at once about one shared failure, it became self-erasure. Guardrails tuned for small independent failures can amplify large correlated ones.
• The circular dependency is why recovery was slow. The tools to fix the network ran on the network — remote access, dashboards, even the door badges. The way out was an out-of-band, break-glass path that shared nothing with the failure, and even that was slow.
• Turning it back on was its own hazard. Cold caches, dropped power draw, and a thundering herd forced a staged recovery — practiced in advance with real failure drills.
• The one question to carry: does any part of my recovery path depend on the thing it recovers? Find the load-bearing loops, make those independent, and test them with real failures before a real one tests you.

One command, six hours, 3.5 billion people. Next we watch a very different failure with the same DNA — a hidden capacity ceiling nobody knew existed, tripping a slow-motion collapse — in AWS Kinesis, 2020: The Thread-Limit Cascade.