Data Integrity: Backups That Restore

Introduction

Everything in The Reliability Discipline so far has been about keeping a system running. You measured it, you paged the right person, you failed over, you spread across regions, you sliced it into cells so no single failure could take down more than a slice. And every one of those tools shares a quiet blind spot: they all assume the data is correct. Failover promotes a replica that holds a faithful copy of whatever was written, right or wrong. Replication streams every write onward, right or wrong. A cell durably persists whatever its customers send it, right or wrong. If a bug writes garbage, all your reliability machinery does is preserve that garbage beautifully, in triplicate, across three continents.

This is the distinction that this lesson turns on, and it is worth saying slowly: durability is not integrity. Durability is the promise that the bytes you wrote will still be there tomorrow — that a disk failure, a power cut, or a lost data center won't lose them. The best storage on earth offers eleven nines of it, so reliable that you'd lose one file in ten thousand roughly once every ten million years. Integrity is a completely different promise: that the bytes are right. A system can give you perfect durability and perfect wrongness at the same time, storing your corruption with the same loving permanence it stores your best work. So this lesson is about the two things durability cannot give you: knowing your data is still correct, and getting the correct version back when it isn't.

Durability is not integrity, drawn as two safes. On the left, a heavy vault labeled DURABLE holds a document, ringed with copies and stamped eleven nines — the bytes will survive almost anything. But the document inside reads WRONG: its numbers are garbage. The vault kept it perfectly, and perfectly wrong. On the right, a second vault labeled CORRECT holds the same document from before the damage, its numbers intact, guarded by a checkmark. An arrow shows the only fix: reach past the durable-but-wrong copy to the correct one from before. Law band: durability means the bytes survive; integrity means the bytes are right — and they are not the same promise.

Two Ways Data Goes Bad

Data corruption comes in two completely different shapes, and confusing them is how teams build the wrong defense. The tell is simple: was the write itself legitimate?

Silent corruption is when no one wrote anything wrong, but the bytes rotted anyway. A cosmic ray flips a bit in memory, a magnetic domain decays on an aging disk, charge leaks out of a flash cell, a cable garbles a byte in flight. This is bit rot, and it is genuinely silent: nothing crashes, no error is logged, the file just quietly stops being what it was. The defense is a checksum, the fingerprint you built in Checksums: Trust but Verify — you record a small hash of each block when you write it, and if the block ever stops matching its fingerprint, you know it rotted. And because you kept a redundant copy, you can heal it: throw away the block that fails its checksum and rebuild it from one that passes. This is exactly why plain mirroring isn't enough on its own — if you keep two copies and they disagree, mirroring alone can't tell you which one is right; the checksum, computed independently, can.

Logical corruption is the opposite, and far more dangerous. Here the write was perfectly legitimate — a bad migration that sets every price to zero, an engineer who drops the wrong table, an application bug that mangles a field, ransomware that encrypts the lot. Every one of those is a valid write as far as the system can tell, so a fresh checksum matches the corruption perfectly, and replication, failover, and cells all copy it onward without a flicker of complaint. Checksums are completely blind to this, because the data isn't broken, it's wrong — and nothing computed from the current data can tell you it's wrong, because the mistake is baked in. The only thing that can save you is a copy of the data from before the bad write happened. That copy is a backup, and everything left in this lesson is about making that backup one you can actually trust.

The two ways data goes bad, side by side, because they need opposite cures. On the left, SILENT corruption: a stored block with a bit physically flipped by a cosmic ray or aging disk, no bug and no human involved. Its checksum — a fingerprint taken when it was written — no longer matches, so the system can detect it and heal the block from a good copy. Checksums catch this. On the right, LOGICAL corruption: a bad migration or a dropped table writes wrong-but-valid data, so a fresh checksum matches the corruption perfectly and every replica and cell faithfully copies it. Checksums are blind here; the only cure is a backup from before the damage. Law band: a checksum catches a flipped bit, but a valid wrong write it cannot see — that is what backups are for.

What Makes a Backup Real

A backup is easy to have and shockingly hard to rely on, and the difference is a small set of properties that the industry has compressed into a rule: 3-2-1, now often extended to 3-2-1-1-0. Read it as a checklist for a backup you can bet the company on:

3 copies of the data, so the loss of any one is a shrug, not an incident.
2 different kinds of media, so a failure mode that kills one kind — a bad drive batch, a filesystem bug — can't take both.
1 copy offsite, outside the blast radius of the original. A backup on the same disk, in the same account, or in the same region as the data it protects is not a backup; it's a second victim. Everything you learned about regions and cells says the copy has to live somewhere the original's disaster can't reach.
1 copy immutable, write-once and air-gapped or locked, so it can't be deleted or encrypted. This one is newer and it exists because of ransomware: attackers now hunt down and destroy the backups first, before they trigger the encryption, precisely to take away your escape. Object-lock and write-once storage, the kind you met in S3 & Object Storage, Inside-Out, make a copy that even a compromised admin account cannot touch.
0 errors on a restore you have actually run — which gets its own section, because it's the one everybody skips.

Two more choices decide what a backup can do for you. First, its shape: a full backup copies everything and restores in one step but is big and slow; an incremental copies only what changed since the last backup of any kind, which is tiny to write but chains every backup together, so one broken link in the chain breaks the whole restore; a differential copies everything since the last full, growing over time but restoring from just the last full plus the latest differential. Second, and most powerful against logical corruption: point-in-time recovery. If you continuously ship the transaction log — the write-ahead log from WAL & Durability: Why Committed Data Survives — you can replay it up to any moment you choose and stop the instant before the bad write landed, rewinding reality to the last good second rather than the last nightly snapshot.

What makes a backup real, drawn as the 3-2-1-1-0 rule as five bold cards. THREE copies of the data, so losing one is a shrug. TWO different kinds of media, so one failure mode can't kill both. ONE copy offsite, outside the blast radius of the original. ONE copy immutable and air-gapped, write-once, so ransomware cannot encrypt or delete it. And ZERO errors on a restore that has actually been run. Each card has an icon and a one-line reason. Law band: a backup in the same blast radius, that an attacker can delete, and that no one has ever restored, is not a backup.

The shapes of a backup, drawn as three restore strategies plus point-in-time recovery. FULL: one big block copied every time, simple to restore in a single step but heavy. INCREMENTAL: a chain of small pieces, each copying only what changed since the last one, cheap to write but the restore needs every link, so a single broken link in the middle breaks the whole chain. DIFFERENTIAL: growing pieces that each copy everything since the last full, restored from just the last full plus the latest one. Below, a transaction-log timeline runs left to right with a red mark where a bad write landed, and a point-in-time marker set one second before it, showing you can replay the log and stop just before the damage. Law band: incremental is cheap to write and fragile to restore; the log lets you rewind to the last good second.

Restore It, or It Isn't a Backup

There's a grim joke among people who run storage for a living: the state of any backup is unknown until a restore is attempted. Call it Schrödinger's backup. A backup job that reports success every night proves only that a job ran and wrote some bytes somewhere. It does not prove those bytes are complete, uncorrupted, restorable, or even the right data — and the moment you actually need them, at 3 a.m. with the site down, is the worst possible time to discover they were none of those things.

This is not hypothetical. Recall the story from Disaster Recovery: RTO & RPO: a company with five separate backup mechanisms lost hours of data because every single one was silently broken and no one had ever tried to restore from them. Five backups, zero restores, and therefore, it turned out, zero backups. The blunt truth is the last item in that rule, the 0: if you have never tested a full restore, you do not have a backup strategy, you have a hope strategy. So you make restoring routine. On a schedule, you restore your backups into a clean, isolated environment and confirm the data is really there — ideally comparing it byte-for-byte against a known-good fingerprint, not just eyeballing that the server came up. You time the restore, because that number is your real recovery-time objective, not the one on the plan. And you automate the whole drill, so that a backup which has quietly stopped restoring sets off an alarm on an ordinary Tuesday instead of during the disaster. A backup you have restored this month is an asset. A backup you have never restored is a guess with good intentions.

Proving Two Copies Agree: Merkle Trees

Restore-testing tells you the whole backup works. But there's a subtler, constant question underneath it: does a copy still match its source, exactly, right now? Has a replica silently drifted from the primary? Did one block in a billion rot last week? You cannot answer that by shipping the entire dataset across the network to compare it byte-for-byte every hour — for anything real, that's terabytes, and it would never finish. The tool that makes this cheap is the merkle tree from Skip Lists & Merkle Trees, and this is the problem it was born for.

Build it bottom-up. Take the fingerprint idea from before and hash every block of data into a leaf hash. Then hash each pair of leaf hashes together into a parent, and each pair of parents into a grandparent, and so on up until a single root hash sits at the top — a fingerprint not of one block, but of the entire dataset, because every byte fed into it. Now comparing two copies becomes almost free. You compare only their roots. If the two roots match, then every byte underneath matches, and you have just verified a billion rows with a single comparison. If the roots differ, you know something diverged, and here is the beautiful part: you descend into the two children, compare their hashes, and follow only the one that disagrees, ignoring the entire half of the tree that still matches. You repeat that all the way down, walking a single path from the root to the exact block that changed, in about log₂(n) hash comparisons instead of n. A tree over a million blocks pinpoints the one bad block in roughly twenty checks.

This one structure quietly runs an enormous amount of the digital world. It's how Cassandra, Dynamo, and Riak run anti-entropy — the background repair from Read Repair & Hinted Handoff that finds and fixes drifted replicas without shipping everything. It's how Git proves your repository's history is intact and how blockchains prove theirs. And it's how the ZFS filesystem builds integrity into its very bones: every pointer to a block also stores that block's checksum, so the whole filesystem is one giant merkle tree, and it can catch a single flipped bit anywhere on a disk. The console below lets you break a block and watch the tree find it.

Proving two copies agree with a merkle tree, drawn as two block rows under a tree of hashes. Each block is hashed into a leaf; each pair of leaf hashes is hashed into a parent; and so on up to a single ROOT hash that fingerprints the whole dataset. To compare a source and a backup you compare only their roots: if they match, every byte matches, proven in one check. Here they differ, shown in red, so the comparison walks down only the branch whose child hashes disagree — left is green and skipped, right is red and followed — again and again, a single red path from the root to the one corrupt block at the bottom. A side note reads O(log n) checks, not all n. Law band: don't compare a billion rows; compare one root hash, then follow the disagreement down.

Catch a corrupt block the way real systems do. Below sit a source and its backup, each a row of data blocks under a merkle tree — every block hashed into a leaf, every pair of hashes folded upward into a single root hash for the whole dataset. Click any block in the backup to corrupt it, and watch its new hash ripple all the way up, changing every hash on its path to the root. Now run the comparison: instead of checking all the blocks, it compares just the two roots, and when they disagree it follows only the branch that differs, down and down, until it lands on the exact block you broke. Corrupt more blocks, or grow the tree, and watch the number of checks stay tiny while the dataset gets huge — the whole reason a system can verify a billion rows against a backup in a heartbeat.

Keeping the Bytes Alive

One layer remains: how does the store that holds your data — and your backups — reach that eleven-nines durability in the first place, so the silent rot from earlier almost never wins? Two mechanisms, working together.

The first is redundancy done cleverly. The obvious way to survive a dead disk is replication: keep three full copies, and if one dies you still have two. It works, but it costs three times the storage for everything you own. The smarter way, from Erasure Coding vs Replication, is to split each object into, say, ten data shards, compute four extra parity shards from them with a little linear algebra, and scatter all fourteen across different disks and availability zones. Any ten of the fourteen are enough to rebuild the original perfectly, so you can lose any four shards and lose nothing — the same survival as replication, for about one-point-four times the storage instead of three. This is how object stores like the one in S3 & Object Storage, Inside-Out offer eleven nines without charging you triple.

The second is never trusting a byte you wrote once. Redundancy only helps if the redundant copies are still good when you finally need them, and bit rot is patient — it can quietly ruin a second copy while you weren't looking, and then a disk failure finds you with no intact copy left. So storage systems run a scrubber: a background process that endlessly walks every block, re-reads it, re-checks it against its checksum, and the instant it finds a block that has rotted, rebuilds it from the parity or the other copies before its neighbors can rot too. It is the digital equivalent of a museum conservator quietly checking every painting, forever. Durable storage plus a tireless scrubber is what keeps silent corruption from ever accumulating — and together with the versioned, restorable backups from the rest of this lesson, it is what lets you promise, and mean, that the data is both there and right.

Keeping the bytes alive, drawn as how a store reaches eleven nines. On the left, REPLICATION: three full copies of an object, simple but three times the storage. On the right, ERASURE CODING: the object split into ten data shards plus four parity shards spread across many disks, where any ten of the fourteen can rebuild the whole — the same survival for about one-point-four times the storage instead of three. Below both, a SCRUBBER robot walks the shelves of disks, re-reading and re-checking each block, and when it finds one rotting it rebuilds it from the others before the rot can spread. Law band: never trust a byte you wrote once — re-check it forever, and rebuild it from the redundancy before the second copy rots too.

Key Takeaways

Durability is not integrity. The bytes surviving is a different promise from the bytes being right; the most replicated system on earth will preserve your corruption perfectly.
Two corruptions, two cures. Silent bit rot is caught by checksums and healed from redundancy; logical corruption — a valid but wrong write — is invisible to checksums and can only be undone from a backup taken before the damage.
A real backup is 3-2-1-1-0. Three copies, two media, one offsite, one immutable against ransomware, and zero errors on a restore you've actually run. Point-in-time recovery from the log lets you rewind to the second before the bad write.
A backup you haven't restored isn't a backup. Its state is unknown until a restore is attempted, so make restoring a routine, automated, timed drill.
Compare copies with a merkle tree. Hash blocks into leaves and up to one root; match the roots to verify everything at once, or follow the disagreement down to the exact bad block in log n checks.
Keep the bytes alive with erasure coding and a scrubber. Parity shards give replication's safety at a fraction of the cost, and a background scrubber re-checks every block forever so rot never accumulates.

That completes the reliability discipline for a system that's already running: you can measure it, defend it, recover it, isolate it, and now trust and restore its data. But notice where nearly all of this danger actually came from — a bad write, and the most common source of a bad write is a change you shipped, a new version with a bug that corrupts data or falls over under real load. The cheapest backup is the one you never have to restore, because you caught the bad change before it ever touched production data. The last move in this section is to test a change against production's own real traffic and behavior before it can hurt anyone — mirroring live requests at it, launching it dark behind a flag. That is Shadow Traffic & Dark Launches.