Alerting Philosophy

SLIs, SLOs & Error Budgets ended on a loaded sentence: a budget burning at fourteen times its allowed rate at three in the morning is only useful if it wakes the right person — and a budget that pages for every one-percent blip will get silenced within a week, and then it wakes no one at all. Alerting is the bridge between a number on a dashboard and a human being's phone, and it is the most quietly destructive system you will ever tune, because it fails in two opposite directions and both are fatal.

Alert too much, and you get the pager that cried wolf. The on-call engineer is woken four times a night for things that didn't matter, learns that the pager is usually noise, and starts acknowledging alerts half-asleep without reading them. Then the one real page — the outage that's actually costing money — arrives at 3 a.m. and gets swiped away with all the others. Alert too little, or on the wrong things, and real damage runs for an hour before anyone notices. The entire craft of alerting is finding the narrow path between those two cliffs: every page should be urgent, actionable, and real — a human needs to do something about it, right now, and it is genuinely happening. Everything else belongs somewhere quieter. This lesson is how you decide what those somewheres are.

The first and most important rule of alerting is a rule about what you watch, and almost everyone gets it backwards at first. You alert on symptoms — the things your users actually experience — not on causes — the internal conditions that might produce them.

A symptom is: requests are failing, or they're too slow. There is essentially one of these per user journey, it maps directly to your SLO, and it is exactly what a user would file a complaint about. A cause is: a disk is 90% full, CPU is pegged, a pod restarted, a cache went cold, a dependency got slow, a queue is backing up. There are a hundred of these, they change every week as the system evolves, and here is the thing that makes cause-based alerting a trap: most of the time, a cause firing doesn't hurt anyone. A disk at 90% is fine if it never reaches 100%. A restarted pod is invisible if the others absorbed the traffic. Page on the cause and you page for non-events; and worse, you'll still miss the outage caused by something not on your list of a hundred, because there's always a hundred-and-first.

So point the pager at the symptom — the one signal that means a user is in pain, usually a failing SLO or a blown latency target from Tail Latency: Living at p99. The hundred causes don't disappear; they move to a dashboard, where they belong, because when the symptom does page you, the causes are exactly what you'll scroll through to find out why. Alert on what the user feels; debug with what you can see. The pager asks is something broken; the dashboard answers why.

Not everything that's wrong deserves a human at 3 a.m., and the second discipline of alerting is refusing to pretend otherwise. Every signal you monitor has exactly three honest destinations, and choosing the right one is most of the job:

• Page — wake a human now. Reserved for a symptom that is actively hurting users, or imminently will, and that needs a person to act immediately. This is the loudest tool you own, and it should be the rarest. If a human doesn't need to do something in the next few minutes, it is not a page.
• Ticket — a human looks within a day. For things that are genuinely wrong but can wait for business hours: a slow burn that will blow the budget next week, a degraded replica, a certificate expiring in ten days. Real work, no adrenaline.
• Dashboard — no notification at all, just recorded. The vast majority of your signals live here: the hundred causes, the trends, the raw material for diagnosis and for the postmortem. Nobody is interrupted; the data is there when someone goes looking.

The single most common alerting mistake is a routing mistake: paging for something that should have been a ticket, or ticketing something that should have just been a chart. Each misroute upward is a small withdrawal from the on-call engineer's trust in the pager, and that trust is the only thing that makes the next page work. Guard the page lane like the scarce resource it is.

Once you've decided to page on a symptom, there's still the question of when — at what threshold does "some errors" become "wake someone"? The naive answer is a fixed line: page when the error rate crosses 5%. It seems reasonable and it is quietly broken, because a raw threshold has no idea what it's costing you. Five percent errors for ten seconds is nothing; five percent for ten hours blows your entire quarter. The same number means wildly different things depending on how long it lasts, and a fixed threshold can't tell the difference — so it either pages on the ten-second blip (noise) or waits out the ten-hour bleed (too late).

The fix is to alert on the error budget burn rate from SLIs, SLOs & Error Budgets: not "how many errors right now" but "how fast are we spending the budget we're allowed to spend." A burn rate of 1× is sustainable — you'll spend exactly your budget by the end of the window. A burn rate of 14× means a month's budget will be gone in two days. Burn rate ties the alert directly to the only thing that matters, the SLO, and it gives you a natural severity dial: a fast burn is a page, a slow burn is a ticket, and a burn under 1× is a dashboard. The question stops being "is this number high" and becomes "is this going to hurt, and how soon."

Burn rate alone still leaves one hard problem, and solving it is the centerpiece of modern alerting. You measure a burn rate over some window of time — but which window? Make it short, say five minutes, and you detect a real outage almost instantly, but a harmless ten-second blip also spikes it and fires a false page. Make it long, say six hours, and the blip is averaged away and it's beautifully stable, but now you're slow to notice a real outage, and — the subtle killer — the alert keeps firing for hours after you've fixed the problem, because the long average is still full of the bad period. Short windows flap; long windows are sluggish and slow to clear. Neither is good enough alone.

The trick, and it is genuinely clever, is to require both a long and a short window to agree before firing: burn over the long window is high AND burn over the short window is high. The long window supplies the significance — this burn has lasted long enough to actually threaten the budget, so it isn't a blip. The short window supplies the currency — it's still happening right now, so the moment you fix it the short window drops and the alert clears fast. Together they fire quickly on a real outage, ignore the blip, and reset cleanly.

Then you stack a few of these at different speeds, because one pair can't catch every kind of trouble. A fast, high pair (roughly 14× burn over an hour, confirmed over five minutes) catches a hard outage and pages. A slow, low pair (roughly 3× over a day, confirmed over a couple of hours) catches the quiet leak that never trips the fast one but will still blow the budget, and files a ticket. A hard outage trips the page; a slow bleed trips the ticket; a blip trips nothing. Drive all three through the console below and watch which alert fires, how long it takes, and how the long window vetoes the blip that a naive alert would have paged you for.

Before any alert earns the right to ring a phone, hold it against four questions. Miss any one and it is not a page.

• Is it real? An actual, ongoing or imminent problem for users — a symptom, not a guess about a cause. False pages are the fastest way to destroy a pager.
• Is it urgent? Does it need action in minutes, not hours? If it can wait until morning, it's a ticket. The whole point of waking someone is that waiting makes it worse.
• Is it actionable? Can the person you woke actually do something about it right now? Paging someone about a problem they can only watch is cruelty with extra steps — that's a dashboard.
• Is it novel? Is this a new condition, not the same flapping alert that fires every night and gets swiped away? A page that fires predictably and harmlessly has already become noise.

And every page needs one more thing waiting at the other end: a runbook, the short document that tells a half-asleep responder what this alert means and what to check first, which is a craft of its own in Alerts & Runbooks in Practice. Treat your pager as a system you maintain, not weather you endure. Track how many times it fires per shift; if on-call is being woken more than a couple of times a night, that is a bug in your alerting, not a fact of life — noisy alerts are outages of their own, quietly burning down the one resource you can't provision more of, which is a rested human who still trusts the pager.

• Alerting fails in two directions, and both are fatal. Too many alerts and the pager cries wolf until the real page is ignored; too few and damage runs unseen. Aim for pages that are urgent, actionable, and real.
• Alert on symptoms, debug with causes. There's one symptom the user feels and a hundred ever-changing causes; the pager watches the symptom, and the causes live on a dashboard.
• Every signal is a page, a ticket, or a dashboard — and the page lane is the scarcest of the three. Misrouting upward is how you spend the on-call engineer's trust.
• Alert on the burn rate, not a fixed threshold, so severity tracks the SLO: fast burn pages, slow burn tickets, sub-1× is a chart.
• Use two windows, at several speeds. The long window gives an alert meaning, the short window gives it speed and a clean reset; a fast-high pair pages for outages, a slow-low pair tickets the leaks, and a blip trips neither.

You now have a pager that rings rarely, means it when it does, and hands the responder a runbook instead of a mystery. Which raises the question this whole lesson has been walking toward: the page fired, it was real, and someone is now awake and staring at it. What do they do? Who else gets pulled in, who talks to customers, who decides it's over — and afterward, how do you make sure this exact page never has to fire again? That is the discipline of Incident Response & Postmortems.