Disaster Recovery: RTO & RPO

Everything you built in Resilience & Overload Control — the timeouts, the breakers, the retries, the load shedding — quietly assumes one thing: that the system is still there to recover. You roll back because the old version still exists. You fail over because a healthy replica is still running. You retry because the request can still succeed. Those are the tools for a system that is struggling.

This lesson is about the day that assumption breaks. A region goes underwater in a flood, or loses power for eight hours. An engineer runs a migration that corrupts the orders table, and it replicates to every copy before anyone notices. A bad deploy deletes data that no rollback can bring back. There is nothing to retry, nothing to fail over to, no healthy replica — because the damage already reached all of them. Now the only move left is to recover: rebuild the service and restore the data from something you saved earlier. And here is the uncomfortable truth about recovery: it is not something you figure out during the disaster. It is a plan you had to build, pay for, and rehearse before the disaster, and its whole design is governed by two numbers that decide, in advance, how bad your worst day is allowed to be.

Every recovery plan is judged by two objectives, and the entire discipline is learning to tell them apart, because they measure two completely different kinds of pain.

RPO, the Recovery Point Objective, is how much data you can afford to lose, measured in time, looking backward from the disaster. It is the distance from the moment of failure back to your last good, usable copy of the data. If your only backup runs at 2 a.m. and the database dies at 7 p.m., you have lost seventeen hours of orders, comments, and sign-ups — your RPO for that day was seventeen hours. RPO answers: when we come back, how far in the past do we come back from? An RPO of zero means you lose nothing; an RPO of a day means yesterday's work is gone.

RTO, the Recovery Time Objective, is how much downtime you can afford, measured in time, looking forward from the disaster. It is the distance from the moment of failure forward to the moment service is usable again. Rebuilding servers, restoring the data, re-pointing traffic — all of it counts. RTO answers: how long are we dark? An RTO of a minute means users barely notice; an RTO of a day means you are closed for a day.

The trap is treating them as one number. They are not even correlated. You can come back fast but stale — flip to a standby in thirty seconds that is missing the last hour of writes (tiny RTO, large RPO). Or come back slow but complete — spend six hours rebuilding, but lose not a single row (large RTO, tiny RPO). One word says it best: RPO is your tolerance for lost data, RTO is your tolerance for lost time. And critically, both are objectives: targets the business sets, not achievements engineering brags about. The number on the plan is a promise. Whether you actually hit it is a different question, and one most teams never test.

Because RTO and RPO measure different pain, you buy them down with different money, using two independent knobs. This is the insight that makes the whole subject click: you are not buying one thing called "disaster recovery," you are turning two separate dials, and each has its own price.

RPO is a data-freshness knob. How much data you lose is set entirely by how recently you copied it somewhere safe. A nightly backup means your worst-case RPO is almost a full day. Snapshot every hour and it drops to an hour. Stream every change continuously to another region — asynchronous replication, from Sync vs Async Replication & Failover — and your RPO shrinks to the replication lag, usually seconds; the lag you studied in Read Replicas & Replication Lag is quite literally your RPO for a region failure. Turn the knob all the way and use synchronous replication, where a write isn't acknowledged until a second copy has it, and your RPO reaches zero — at the cost of adding that second copy's round trip to every single write, which is why almost nobody runs synchronous replication across distant regions.

RTO is a standby-capacity knob. How fast you come back is set by how much of your system is already built and running somewhere else when disaster strikes. If nothing is running, you must provision servers, deploy code, and restore data before you can serve a single user — hours. If a scaled-down copy is already running and just needs to grow — minutes. If a full second site is already serving traffic — you barely fail over at all. Warm hardware costs money whether or not disaster ever comes, which is exactly why a lower RTO costs more.

Two knobs, two bills. You can spend heavily on the RPO knob and lightly on the RTO knob, or the reverse, and matching each knob to what the business actually fears is the real skill. The next diagram is the menu those two knobs add up to.

Turn both knobs together and the industry has settled on four recognizable rungs, each a package of RTO, RPO, and monthly cost. Climb the ladder and recovery gets faster and fresher; so does the bill. They are worth knowing by name, because "which rung are we on for this service?" is one of the most useful questions in a design review.

• Backup & Restore is the ground floor. Nothing of yours runs in the recovery region at all — only your saved backups sit there cheaply. When disaster strikes, you rebuild the whole environment from infrastructure-as-code and restore the data from the last backup. It is the cheapest option by far and the slowest to recover: RTO and RPO both measured in hours. Right for anything where a day in the past and a few hours dark is survivable.
• Pilot Light keeps the heart beating. Your data stores replicate continuously and stay switched on in the second region, while the application servers sit deployed but switched off, costing almost nothing. On disaster you switch them on and scale up around the already-warm data. RTO in tens of minutes, RPO in minutes. The data is current; only the compute needs waking.
• Warm Standby keeps a smaller version of the whole thing actually running. A scaled-down but complete copy of your stack is live in the second region and could take real traffic right now, just not at full volume. On disaster you scale it up to full size and cut traffic over. RTO in minutes, RPO in seconds. You pay to run a fraction of production around the clock.
• Active/Active removes the idea of failover entirely. A full copy of your system runs in two or more regions at the same time, all of them serving live users. When one region dies, there is nothing to "start" — you just stop sending traffic there, and the others were already handling their share. RTO and RPO both near zero. It is the most expensive and most complex option, because now you are running everything at least twice and must keep the copies consistent — which is a whole topic of its own, coming next in Multi-Region Patterns.

Notice the shape: each rung buys down RTO and RPO by keeping more of your system warm and your data fresher, and each rung costs more to do it. There is no free lunch and no single "right" answer, only a price you're willing to pay for a promise you need to keep. The console below lets you buy a rung and then find out, under fire, whether it keeps its promise.

Now the subtlety that catches even strong teams, and the reason the console has two disaster buttons instead of one. Not all disasters are the same shape, and the fanciest recovery tier defends beautifully against one of them and not at all against the other.

Losing the place is the disaster everyone pictures: a region goes dark, a data center floods, an availability zone loses power. The system is fine; it just isn't reachable. Here replication is your hero. A copy of your data has been streaming to another region this whole time, so you fail over to it and carry on. Every rung above the ground floor is built for exactly this, and a real region can and does vanish — that's the story waiting in us-east-1: The Region the Internet Leans On.

Losing the data is the disaster that turns your defenses against you. Someone drops a table. A bad migration rewrites every price to zero. Ransomware encrypts the volume. The system is perfectly reachable — it is faithfully serving wrong data. And now watch what your beautiful continuous replication does: it copies the corruption to every replica, in every region, in seconds, because replication cannot tell a legitimate write from a catastrophic one. Failover is useless — every copy is poisoned. Your real-time RPO of zero just became worthless, because the freshest copy is the most corrupt. The only way out is to travel backward in time to a point-in-time backup taken before the damage, and accept losing everything since. This is why a team running flawless active/active can still lose a day to a dropped table, and why "we replicate to three regions" is not the same sentence as "we can recover our data."

So a real plan needs both stories: replication for losing the place, and independent, versioned, point-in-time backups for losing the data, ideally backups that a compromised account can't also delete. Making those backups actually trustworthy, byte for byte, is a big enough problem to get its own lesson: Data Integrity: Backups That Restore.

On the last day of January 2017, an engineer at GitLab, cleaning up after a spam attack late at night, ran a delete against what turned out to be the primary database instead of the broken replica. Three hundred gigabytes of live data vanished in seconds. No problem — that's what backups are for. Except the nightly database dump had been silently failing for days because of a version mismatch, and produced empty files. The replica had just been wiped. The disk snapshots weren't enabled for that server. The cloud bucket backups had never been configured. The one usable copy was a manual snapshot a lucky engineer had taken by hand six hours earlier. Five separate backup mechanisms, and, in their own words, none of them were working reliably or set up in the first place. GitLab lost six hours of everyone's work — their real RPO that day, discovered the hard way, was six hours, not the near-zero their plan implied.

The lesson is not "GitLab was careless." The lesson is that a backup you have never restored is not a backup — it is a hope. RTO and RPO written in a plan are guesses until you have actually run the recovery and timed it. The gap between the number you promised and the number you'd really hit is invisible right up until the disaster, and then it is the only thing that matters. So you close that gap on purpose, in advance:

• Rehearse the restore. On a schedule, restore your backups into a fresh environment and confirm the data is actually there and usable. Restoring is often a slow, fiddly control-plane operation — the worst possible moment to discover that is at 3 a.m. with the site down.
• Rehearse the failover. Actually cut traffic to the standby, in production, on a normal Tuesday. This is Chaos Engineering pointed straight at your recovery plan — and it's how Netflix can evacuate an entire region in about seven minutes, because they practice it constantly rather than hoping it works.
• Measure, don't assume. Every drill gives you a real RTO and RPO. If the measured numbers miss your targets, you either buy a higher rung or lower the promise. A target you have never hit in a drill is marketing, not engineering.

One question is still open: where do the target numbers even come from? Not from engineering taste, and not from "lower is always better" — because lower always costs more, and the money is real. The targets come from the business, by asking a blunt question about each service: what does it actually cost us when this is down or this data is lost?

That is a business-impact question, not a technical one. An hour of downtime on a payments system is measured in direct lost revenue plus churned customers plus, sometimes, regulatory penalty — it can run into millions, so it earns tight targets and an expensive tier to hit them. An hour of downtime on the internal analytics dashboard costs a few annoyed analysts, so it earns a cheap tier and generous targets. The move is to tier your services: mission-critical paths get minutes of RTO and seconds of RPO on a warm or active setup; important-but-survivable services get tens of minutes on pilot light; background work gets hours on plain backup & restore. You do not buy a second live data center for the reports. You buy it for the checkout.

This connects straight back to SLIs, SLOs & Error Budgets: RTO and RPO are just more reliability targets, chosen deliberately and matched to cost, and the money you spend buying them down should be no larger than the loss they prevent. Spend where an outage hurts; save where it doesn't. That single sentence — spend on recovery in proportion to the cost of not recovering — is the whole discipline in one line.

• Recovery begins where resilience ends. Rollback, retry, and failover assume the system and its data still exist. When they don't — region gone, data corrupted — you recover from something you saved earlier, on a plan built in advance.
• Two numbers, two kinds of pain. RPO is how much data you can lose (measured backward to your last good copy); RTO is how much time you can lose (measured forward to service restored). They are independent — fast-but-stale and slow-but-complete are both real outcomes.
• Buy them with different knobs. RPO is set by data freshness (backup cadence and replication lag); RTO is set by standby capacity (how much is already warm). Each smaller number costs more.
• The ladder is a price list. Backup & Restore (hours, cheap) → Pilot Light (minutes) → Warm Standby (seconds of data) → Active/Active (near zero, costly). Pick the rung whose price matches the promise you need.
• Replication is not a backup. It defends against losing the place, but it copies corruption to every replica — so losing the data is only survived by an independent point-in-time backup you can roll back to.
• A plan you haven't drilled is fiction. Rehearse the restore and the failover, and measure the RTO and RPO you truly hit. Set the targets from the cost of downtime, service by service.

The strongest rung on the ladder — a full system already running in more than one region — stopped being "recovery" somewhere along the way and became an architecture. There was no failover to perform because the second region was never a spare; it was always live. How to build that, how active-passive and active-active actually route traffic and keep their data in step, and the fresh problems that appear when your system runs in two places at once, is Multi-Region Patterns.