How to Read a Postmortem

Here's a trade almost nobody takes, and it's the best one in this whole field. You can spend twenty years on call, absorbing outages the hard way, one 3 a.m. page at a time, until you've seen enough disasters that new ones start to feel familiar. Or you can read other people's postmortems for an afternoon and get a surprising fraction of that same hard-won intuition, without any of the pages. Every serious public incident report is a company handing you their worst day, fully documented, for free.

You've just done this with three of them, closely: a company that erased itself from the internet, a fleet that melted from a routine change, a region the whole internet quietly leans on. That was reading with training wheels, me pointing at the important parts. This lesson takes the wheels off. It's about the skill of reading any postmortem on the internet and pulling the real lessons out of it, which is not the same as passively believing whatever the document tells you.

Because here's the thing most people get wrong: a postmortem is not a verdict to absorb. It's evidence to interrogate. The stated root cause at the top is where your reading starts, not where it ends. A good report rewards a skeptical, active reader with patterns you'll see again in your own systems; a bad report, read carelessly, will actively teach you the wrong lesson.

In this lesson: why other people's disasters are the cheapest education in the field; the shape a good report takes; how to read it without fooling yourself; the red flags that mark a bad one; and the single question that turns everything you read into something useful.

Scope: how to run an incident and write a good postmortem of your own is Incident Response & Postmortems. This lesson is the other side of that skill, reading everyone else's.

There's a line from Richard Cook's famous little essay on how complex systems fail that belongs over every engineer's desk: failure-free operation requires experience with failure. You cannot build a reliable system if you've never seen an unreliable one break. The catch is obvious, breaking your own systems to gain that experience is expensive and career-limiting. So you borrow. Every published postmortem is someone else's expensive experience, offered up at no cost to you.

And it compounds, because outages are not infinitely varied. Read a hundred of them and you stop seeing a hundred unique disasters and start seeing the same dozen shapes wearing different clothes, which is exactly the anatomy from Anatomy of a Cascading Failure. A trigger, an amplifier, a sustaining loop, a slow recovery. A hidden dependency nobody mapped. A safety mechanism that fired at the worst possible moment. Once those shapes live in your head, you start spotting them in your own architecture during design reviews, months before they'd ever have a chance to page you.

That's the real return on reading postmortems. It isn't trivia about other companies' bad days. It's a growing library of failure patterns that you get to install in your judgment cheaply, so that when you're staring at your own system, some quiet voice says I've seen this shape before, and I know how it ends.

Before you can read one well, you need to know what a well-built one contains, so you can notice what's missing. Good reports vary, but nearly all of them have the same five bones, and each one is there to answer a specific question.

The impact tells you how bad was it, who was affected, for how long, and how much it cost or hurt. Read this first, because it calibrates everything after: you judge decisions very differently for a five-minute blip than for a six-hour global outage. The timeline tells you what happened, and in what order, ideally minute by minute, from before anything broke through to full recovery. The contributing factors, note the plural, tell you what combined to cause it, the several things that each had to be true. What went well tells you what saved you, the alert that fired, the mitigation that worked, the guardrail that held, and it matters more than people think. And the action items tell you what changes so it can't recur, ideally each one specific, owned by a name, and dated.

You'll notice this maps almost exactly onto how you'd write your own, from Incident Response & Postmortems. That's the point: knowing the ideal shape is what lets you read a real one critically. When a report has no timeline, or lumps everything under one root cause, or ends with action items like be more careful, you'll feel the gap immediately, because you know what should have been in it.

Now the single most important habit in reading a postmortem, and the one almost everyone gets wrong on their first hundred: do not judge the people in the story with information they didn't have.

When you read a postmortem, you already know the ending. You know that command took down the backbone, that this deploy was the bad one, that the alert everyone ignored was the real signal. And that knowledge quietly poisons your reading, because, as Cook puts it, knowledge of the outcome makes it seem that events leading to the outcome should have appeared more salient to practitioners at the time than was actually the case. You start thinking how did they not see it? This is hindsight bias, and it's the primary reason people take the wrong lesson from an outage. It makes every disaster look obvious and every engineer look careless, which is almost never true.

The fix is a deliberate mental move: read the timeline in foresight. Start before anything went wrong and walk forward, and at each step, cover up the ending and ask, what did this person actually know at this exact moment, and what would I have done with only that? The alert they ignored was one of forty that fire every day. The command they ran had worked a thousand times. Seen forward, with only the information they had, their decisions usually look reasonable, and that's the whole point, because a reasonable decision that still led to disaster tells you something is wrong with the system, not the person. A careless-idiot story teaches you nothing. A reasonable-person-still-failed story teaches you everything.

Most postmortems have a line near the top that says Root Cause: followed by a single thing. Treat that line with polite suspicion, because in a complex system, it is almost always incomplete, and often actively misleading.

The reason is a property of complex systems that Cook states flatly: catastrophe requires multiple failures; single-point failures are not enough. Big outages are not caused by one thing going wrong. They're caused by several things going wrong at once, each of which was individually survivable, and only their combination was fatal. The exact phrase for it is that each factor is necessary but only jointly sufficient. Look back at Facebook: it wasn't just the command, it was the command and a broken audit tool and a health check that withdrew every route and recovery tools that ran on the dead network. Remove any one of those and there's no six-hour outage. Which was the root cause? The question barely makes sense.

So when you read Root Cause: an engineer ran the wrong command, your job as a reader is to silently rewrite it as a question: what else had to be true? Hunt for the contributing factors the single-cause framing hides, the missing guardrail, the untested config path, the system already running at the edge, the slow error handler. A trick from John Allspaw makes this automatic: instead of asking why it happened, which pulls you toward one tidy causal chain, ask how it happened, which forces out the messy list of conditions that actually combined. The single root cause is a story the human brain likes. The contributing factors are the truth, and they're where the real lessons hide.

A postmortem tells you two stories at once: the story of the outage, and, in how it's written, the story of the culture that produced it. A careful reader learns to spot the second one, because a report full of red flags will teach you the wrong lessons no matter how interesting the outage was. Here are the ones to watch for.

Blame wearing a lab coat. Root cause: operator error. An engineer failed to follow the runbook. Any report whose deepest explanation is a person is waving the biggest red flag there is. Sidney Dekker's line is the antidote: underneath every simple, obvious story about human error, there is a deeper, more complex story about the organization. Naming a human isn't a root cause, it's where a lazy investigation stopped.

The single root cause, which you now know is almost always a fiction. Hindsight dressed as insight, any sentence like they should have noticed or if only they had checked, which is the bias from the last section printed right on the page. Vague, ownerless action items, we will be more careful, we will add more monitoring, with no specific change, no name, and no date, which means nothing will actually happen. And the dismissive shrug, an isolated incident, this will not happen again, or quietly blaming a vendor, which tells you the document is a piece of public relations, not a piece of learning.

None of these mean the outage wasn't real or interesting. They mean you should read that report defensively, extracting the facts of what broke while ignoring the company's self-serving interpretation of it. The console below is a short postmortem with some of these traps buried in it. Read it like a professional, flag the rot, and see how good your eye already is.

Read enough postmortems and a strange thing happens: you start being able to guess the ending. A handful of shapes cause a wildly disproportionate share of real outages, and once you know them, you see them coming. The engineer Dan Luu has read more public postmortems than almost anyone, and a few patterns dominate his notes.

Configuration changes are the quiet killer. Searching for the worst global outages, roughly half trace back not to a code bug but to a config change, a flag, a limit, a routing rule, pushed out without the testing and staging that code gets. You met it in every case study this section. Error-handling code is the other giant: one study found that 92% of catastrophic failures came from incorrectly handled errors, the code paths that only run when something's already wrong and therefore almost never get exercised. It's the same truth Cascading Failures stated, the code path you never use is the one that doesn't work. Then there's the be-careful anti-pattern: a report where the safety measure for a risky operation was a human being very careful, which held right up until the human, inevitably, was not. And monitoring blind spots, never the sole cause of anything, but a faithful supporting actor in a huge fraction of outages, the reason a small problem had time to grow into a large one.

Keep a running tally of these as you read, because the payoff is direct. The next time you're about to push a config change by hand, or you notice an error path in your code that has no test, or someone proposes we'll just be careful as a safety plan, a postmortem you read six months ago will tap you on the shoulder.

Everything so far builds to one move, and if you only take one habit from this lesson, take this one. A postmortem you read and merely found interesting was entertainment. A postmortem you read and acted on was education. The difference is a single question you ask at the end of every one:

Does my system have this same shape?

That's the transfer. When you finish reading about Facebook's self-lockout, you don't file it under interesting thing that happened to Facebook. You turn to your own system and ask: is any part of my recovery path wired through the thing it's meant to recover? When you finish the Kinesis thread-limit cascade, you ask: where in my system does a cost grow faster than the thing I'm scaling? When you read about a config change that wasn't staged, you ask: do we stage config changes like code? Each of those questions, honestly answered, becomes a small list of things to go check on Monday, and a few of them become real work that quietly prevents an outage nobody will ever know you avoided.

This is the whole reason the skill is worth building. Reading postmortems isn't about collecting war stories to tell at lunch. It's a pipeline: someone else pays the full price of an outage, writes it down, and you convert their pain into a checklist item in your own backlog. Do this consistently and you accumulate the scar tissue of an engineer far more experienced than your years, having borrowed almost all of it. There is no cheaper way to get good at this, and it starts every time with those five words: does my system have this?

Here's a postmortem excerpt to practice on before you read the analysis. A payments company publishes this:

Root cause: A backend engineer deployed a change during the freeze window that they should have known was risky. Timeline: 14:02 errors began; 14:40 engineers identified the bad deploy; 15:30 rolled back. Action items: (1) Remind all engineers about the deploy freeze policy. (2) The engineer involved will receive additional training. This was an isolated incident and we are confident it will not recur.

Read it like a professional. What are the red flags, and what's the real story the report is hiding? Work it out, then check yourself:

• Red flag, blame: root cause is a person who should have known. That's the Bad Apple Theory, and it's where the investigation stopped instead of started.

• Red flag, hindsight: should have known it was risky judges the engineer with the ending in hand. What did they know at 14:00? Probably that the deploy tool let them ship, which is the actual problem.

• Red flag, vague and blaming action items: remind everyone and train the engineer change nothing about the system. Not owned, not dated, not systemic.

• Red flag, dismissive: isolated incident, will not recur is public relations, not analysis.

• The real story it hides: how did a risky change ship during a freeze at all? The contributing factors the report buries: the deploy system didn't enforce the freeze, there was no automated check on the change, and rollback took nearly an hour. The fix isn't a reminder, it's make the system enforce what the policy only requests. That's what a good action item would say, with an owner and a date.

• The tell to remember: when a postmortem's deepest cause is a person and its fixes are reminders, you're reading blame, not learning. The real lessons are always in the how.

Reading postmortems well means unlearning a few instincts that feel like common sense.

• Myth: the stated root cause is the cause. Treat that line as where your reading starts. Real failures have multiple contributing factors, each necessary but only jointly sufficient, and the single-cause framing hides most of the lesson.
• Myth: if they'd just been more careful, it wouldn't have happened. That's hindsight bias plus the Bad Apple Theory. Read the timeline in foresight, with only the information they had, and a reasonable-person-still-failed story points you at a broken system, which is the useful conclusion.
• Myth: reading postmortems is war stories and entertainment. It's the cheapest operational experience you can buy. The value only lands when you transfer it: does my system have this shape?
• Myth: a postmortem belongs to the company that had the outage. A public one is a gift to everyone. The entire skill in this lesson is reading other people's, and converting their expensive day into your cheap prevention.
• Myth: a good postmortem finds who's accountable. A good one is blameless and finds the systemic factors and the guardrails. The moment a report starts assigning blame, the honesty that made it worth reading evaporates, and so does the lesson.

• A postmortem is evidence to interrogate, not a verdict to absorb. The stated root cause is where your reading starts. Read actively, skeptically, and in foresight.
• Read the timeline in foresight, not hindsight. Judge each decision with only the information the person had at that moment. A reasonable decision that still failed indicts the system, not the human, and that's the lesson worth having.
• There is no single root cause. Catastrophe takes multiple contributing factors, each necessary but only jointly sufficient. Ask how, not why, and hunt the factors the single-cause line hides.
• Learn the red flags. Blame as a root cause, a single cause, hindsight printed as insight, vague ownerless action items, and the dismissive isolated incident all mark a report to read defensively, for its facts, not its conclusions.
• Expect the repeating patterns. Config changes, mishandled error paths, human caution used as a safety control, and monitoring blind spots cause a wild share of real outages, and once you know them, you see them coming in your own system.
• Always end with the transfer question: does my system have this same shape? That single question is what turns someone else's expensive outage into your cheap prevention.

And that closes the section. You've watched three real outages up close, pulled the shared anatomy out of them, and learned to read any postmortem on the internet for its real lessons. The last stop gathers all of it into a single test of judgment, Checkpoint: Learning from Real Outages.