SLIs, SLOs & Error Budgets

Checkpoint: Resilience & Overload Control left you with a question every lesson in that section quietly assumed you could answer: is the service actually working? You reached for the obvious gauge — is it up? — and that gauge is broken. A modern service is never simply up or down. It is serving 99.97% of requests while 0.03% get a 500; it is fast for most users and slow for the unlucky tail; it is perfectly healthy in one region and on fire in another. "Up" is a light switch bolted onto something that behaves like a dimmer, and it will happily glow green while a slice of your users are having the worst day of their week.

So reliability engineering starts by throwing that switch away and replacing it with three things: a number that measures how well the service is actually doing (the SLI), a target for that number that you promise yourself (the SLO), and a budget for how much you're allowed to miss it (the error budget). Together they do something quietly radical: they turn reliability from an argument — the eternal standoff between the team that wants to ship and the team that wants to keep it up — into arithmetic. This lesson builds that stack from the bottom, and it ends with you spending a budget you can watch drain in real time.

An SLI, a service level indicator, is a carefully chosen number that tracks one aspect of how well your service is serving its users. The word that matters is chosen: you have a thousand metrics, and almost all of them are noise. CPU is at 80% — is that good or bad? You cannot say, because CPU is not something a user feels. An SLI is the small set of numbers a user would complain about if they went the wrong way.

The best SLIs almost all take the same shape, and it's worth burning into memory:

SLI = (good events / valid events) × 100%

The proportion of things that went well out of all the things that counted. The proportion of requests that returned without an error. The proportion of requests that came back in under 300 milliseconds. The proportion of queue items processed within a minute. Framing an SLI as a ratio of good to valid is what makes it a percentage you can set a target on, and it forces you to define, precisely, two things: what counts as an event (a valid request — not health checks, not your own load tests) and what counts as good (the threshold that separates a happy user from an unhappy one).

Two rules turn a mediocre SLI into a good one. First, measure where the user feels it. A request that never reached your server because the load balancer was out of connections is, to the user, a failure — but your server-side metrics never saw it and will report a cheerful 100%. Measure as close to the user as you can: at the load balancer, at the edge, or best of all from the client. Second, prefer the signals that track pain. Google's four golden signals — latency, traffic, errors, and saturation — are the usual raw material, but for user-facing SLIs two of them do most of the work: errors (did the request succeed?) and latency (was it fast enough?). Traffic and saturation tell you about load and headroom; errors and latency tell you whether users are suffering. A good SLI moves when your users are unhappy, and stays flat when they're fine.

An SLO, a service level objective, is a target for an SLI over a window of time. "99.9% of valid requests succeed, measured over a rolling 28 days." "95% of requests complete in under 300 milliseconds." The SLI is the measurement; the SLO is the line you draw on it and promise not to cross. It's the number that, when you meet it, means the service is reliable enough, and when you miss it, means you have work to do.

Which raises the question everyone gets wrong the first time: why not aim for 100%? It sounds like the only honorable target. It is, in fact, the wrong target for almost everything, for three compounding reasons. It's impossible — every dependency, network, and machine you sit on has its own failures, and the moment you promise 100% you have promised something no system on earth delivers. It's imperceptible — your user reaches you through a phone radio, home wifi, and an ISP that are, together, far less reliable than 99.99%, so the difference between your 99.9% and a hypothetical 100% vanishes into noise they were already living with. And it's ruinously expensive — each additional nine costs roughly ten times the engineering effort of the last, and every hour you pour into a nine nobody can feel is an hour you didn't spend shipping the thing they actually wanted.

So the real skill is picking the right target: the last nine your users can actually notice, and not one more. Set it too low and you ship a service people don't trust; set it too high and you've quietly decided that reliability nobody perceives is worth more than every feature you didn't build. The nines you met in Availability & the Nines are the menu; the SLO is you ordering deliberately from it. And when the SLI is a latency percentile, this is where the tail you studied in Tail Latency: Living at p99 becomes a line in a contract — you don't set an SLO on the average, you set it on the p95 or p99, because the average is the number that lies.

One more term, because people mix it up with the SLO constantly, and the difference is the difference between engineering and money. An SLA, a service level agreement, is an SLO written into a contract with your customers, with consequences attached: miss it, and you owe refunds, service credits, or an awkward call with your biggest account. The SLO is a promise you make to yourself; the SLA is a promise you make to someone who can invoice you for breaking it.

The practical rule that follows is one of the most useful in this whole section: your internal SLO must be stricter than your external SLA. If you promise customers 99.9% in the contract, you hold yourself to 99.95% internally, so that when you breach the SLO your own alarms are already screaming and your team is already fixing it before you breach the SLA and start writing checks. The SLO is the smoke detector; the SLA is the fire department's bill. You want to be running for the extinguisher long before the second one arrives.

Not every service needs an SLA — most internal services never sign a contract with anyone. But every service that matters needs an SLO, because the SLO is the engineering tool and the SLA is just the SLO after the lawyers have been in the room.

Here is the idea that makes the whole discipline worth the trouble, and it comes from a single line of arithmetic:

error budget = 100% − SLO

If your SLO is 99.9%, then 0.1% of requests are allowed to fail. Over a 30-day window that is about 43 minutes of being completely down, or its equivalent spread across millions of slightly-failed requests. That 0.1% is not a failure you scramble to prevent. It is a budget you are handed at the start of every window, and the entire reframe of modern reliability lives in that one word. You are not trying to never fail. You are trying to not overspend.

This flips the oldest fight in the building. The product team wants to ship fast, which means taking risks — new code, migrations, experiments — every one of which might burn reliability. The reliability team wants to keep it up, which means saying no. With an error budget, nobody has to win that argument, because it's now a number both sides can read. Budget in the bank? Ship. Launch the risky feature, run the migration, push on Friday if you like — you have reliability to spend, so spend it on velocity. Budget spent? Freeze. New features stop, and the whole team turns to reliability work until the budget refills. This agreement, written down in advance, is called the error budget policy, and its magic is that it converts a political standoff into a thermostat: the system regulates itself, and no one has to be the person who says no.

There's a subtlety here that trips people up, and it's the most important one. There is no reward for beating your SLO. A quarter that ends at 99.99% against a 99.9% target isn't a triumph — it means you had budget you never spent, which means you were too cautious, shipped too slowly, and left velocity on the table. Unspent error budget is wasted opportunity. The goal is not to maximize reliability; it is to hit the SLO — from either side. Now stop reading and go spend one. Set a target, pick a window, and drive the service through a good day, a bad day, and an outright incident. Watch the budget drain, watch the verdict flip from ship to freeze, and try to launch when the tank is empty.

A budget you can only check at the end of the month is useless while the month is happening, so the number you actually live by is the burn rate: how fast you're spending the budget, relative to spending it evenly across the whole window. A burn rate of 1× means you're on pace to spend exactly your budget by the window's end — steady, sustainable, fine. A burn rate of 10× means you're spending ten times too fast, and a budget meant to last a month will be gone in three days. Burn rate is the difference between "we had a rough afternoon" and "we will breach the SLO by Thursday," and you can only see it if you're watching the rate, not the total.

This is also the seam where measurement becomes action. A slow burn — a little elevated, 1× to 2× — is a ticket to look at tomorrow. A violent burn — 14× and climbing, the kind that torches a month of budget in a day — is a human being's phone ringing at 3 a.m. Turning burn rates into the right pages, at the right speed, without crying wolf, is a craft of its own, and it's the whole subject of the next lesson. For now, hold the intuition the simulator gave you: the total tells you whether you're in trouble; the burn rate tells you how fast trouble is coming.

The theory is clean; the practice has a few sharp edges worth naming, because a badly-set SLO is worse than none — it either cries wolf until everyone ignores it, or sits so slack it never fires while users suffer.

• Start from the user's journey, not the service's internals. Ask what a user is trying to do — load the feed, complete the checkout, get search results — and set SLIs on those outcomes. One to three SLIs per critical journey is plenty; a dashboard of forty SLOs is a dashboard nobody reads.
• Measure first, then target. Look at how the service actually performs today before you promise anything. An SLO you're already missing is a source of alert fatigue on day one; an SLO you beat in your sleep protects no one. Aim for a target that's achievable but honest — usually a little better than today, tightened over time.
• Write the error budget policy before you need it. Decide, while everyone is calm, what happens when the budget runs out: what freezes, who can grant an exception, when it lifts. A policy written mid-incident is a policy written by whoever is angriest.
• Review on a cadence. User expectations move, the product changes, and last year's target may be this year's embarrassment in either direction. Revisit SLOs quarterly, and treat a chronically over- or under-spent budget as a signal the target itself is wrong.

Do this and you've built something most engineering organizations never quite manage: a shared, quantitative definition of "good enough" that a product manager, an engineer, and an on-call responder all read the same way.

• The SLI is the number, measured where the user feels it. Good events over valid events, times 100 — usually built on the errors and latency golden signals, measured at the load balancer or the client, not deep in a server that never saw the failure.
• The SLO is the target you promise yourself, and 100% is the wrong one. Aim for the last nine your users can actually notice; more than that is impossible, imperceptible, and ruinously expensive. Set it on the percentile, never the average.
• The SLA is the SLO with a lawyer, and your internal SLO is stricter. The smoke detector fires before the fire department bills you.
• The error budget is 100% minus the SLO — permission to ship. Budget in the bank, ship and take risks; budget spent, freeze and fix. There's no prize for beating the SLO, because unspent budget is velocity you left on the table.
• Burn rate is how fast you're spending it, and it's the signal you act on in real time — a slow burn is a ticket, a violent burn is a page.

You now have a language for how reliable a service should be and a budget that tells you when to slow down. But a budget burning at 14× at 3 a.m. is only useful if it wakes the right person — and a budget that pages you for every one-percent blip will get silenced within a week, and then it wakes no one at all. Turning these numbers into alarms that are urgent, actionable, and rare enough to trust is the hardest, most human part of the whole discipline. That is Alerting Philosophy.