Media Chunking

You're uploading a 4 GB drone video from a hotel on conference Wi-Fi. At 97% the connection blips — the elevator, the microwave, someone streaming in the next room — and the upload dies. With a single PUT request, you don't resume at 97%. You start over. From zero. Four gigabytes again, on the same Wi-Fi that already failed you once.

That's the first lie of the single request, and there are three of them:

• No resume. A PUT is one HTTP request. If the socket dies, the whole thing is gone — the server has a half-written object it will throw away, and you begin again.
• No bounded memory. To sign and stream one 4 GB request cleanly, the naive client wants the whole file staged — 4 GB of RAM (or a temp file) just to send it. Do that for a hundred concurrent uploads and your uploader falls over.
• A hard ceiling. There's a maximum size for a single request. On Amazon S3 a single PUT tops out at 5 GiB — one byte more and the API simply refuses. AWS suggests switching approaches once an object passes ~100 MB, long before the ceiling.

The fix is the same move you've seen all through this section — in Distributed File Systems a file became 64 MB chunks across machines; in Erasure Coding vs Replication an object became data and parity shards. Here we split for a different reason: to move it. Cut the object into parts, send each part as its own small request, and suddenly the upload is resumable, parallel, memory-bounded, and unbounded in size. Here's the whole flow on one page; the rest of the lesson earns it.

The protocol S3 (and every S3-compatible store) uses for this is multipart upload, and it's three acts.

Act one — announce. You call CreateMultipartUpload for the key. The server hands back an uploadId: a ticket that groups all the parts you're about to send. Nothing is stored yet; you've just opened a folder.

Act two — send the parts. You cut the object into parts and call UploadPart for each one, tagged with the uploadId and a part number (1, 2, 3, …). Each part is an independent request — send them in parallel, retry any one on its own. The server stores each part and returns an ETag, a content fingerprint you hold onto. The rules are worth knowing: a part is 5 MiB to 5 GiB (only the last part may be smaller), you get up to 10,000 parts, and 10,000 × 5 GiB is why the maximum object size is 48.8 TiB — five orders of magnitude past the single-PUT ceiling.

Act three — assemble. You call CompleteMultipartUpload with the ordered list of {partNumber, ETag}. Only now does the store concatenate the parts, in the order you specify (not the order they arrived), into one object. (If you give up instead, AbortMultipartUpload throws the parts away so you're not billed for orphaned storage.) Here it is, for real, against a live S3 store:

$ CreateMultipartUpload  hotel-wifi/4k-drone.mp4   →  uploadId: NDEyZjhh…
$ UploadPart #1  ETag="208dc7df…"
$ UploadPart #2  ETag="d139e572…"
$ UploadPart #4  ETag="f5143d95…"

One detail that trips people up: the finished object's ETag is not the MD5 of the file. For a multipart object it's md5( the concatenated MD5s of every part ) + "-N", where N is the part count — our 5-part object completed with ETag = 682c013a…-5. That trailing -5 is the giveaway that an object was uploaded in parts. (We just uploaded 1, 2, and 4 — then the Wi-Fi drops. That's the next section.)

Here's the payoff the single PUT could never give you. The connection died after parts 1, 2, and 4 landed; parts 3 and 5 were in flight and lost. You don't start over. You ask the store what it already has — ListParts on the uploadId — and it tells you:

$ ListParts  →  landed: [1, 2, 4]   (18 MiB of 30 MiB)
  missing = {1..5} − {1,2,4} = [3, 5]      ← re-send ONLY these
$ UploadPart #3  ✓
$ UploadPart #5  ✓
$ CompleteMultipartUpload  →  30 MiB object, ETag 682c013a…-5
  downloaded MD5 == original MD5  (bit-for-bit)  →  True

Resume is a set difference. The parts you meant to send, minus the parts the server confirms it has, equals the parts to retry. We moved 12 MiB, not 30 — and after a real 4 GB drone video drops at 97%, you move the last 3%, not the whole thing again. This works because UploadPart is idempotent: re-sending part 3 (maybe the first attempt did land and only the ack was lost) just overwrites it with identical bytes. No duplicates, no corruption — exactly the discipline from Idempotency as a Discipline.

S3 resumes by part number. There's a second style worth knowing, because you'll meet it in browser and video uploaders: the tus resumable-upload protocol resumes by byte offset. The client sends a HEAD; the server answers Upload-Offset: 5242880 ("I have the first 5 MiB"); the client sends a PATCH with the remaining bytes starting at that offset, and repeats HEAD → PATCH until the offset equals the file length. Same idea — never re-send what landed — expressed as one growing stream instead of a set of numbered parts. Vimeo and Cloudflare Stream upload this way.

Chunking isn't only an upload trick — the download side has its own, and it's the reason you can drag the scrubber to the middle of a two-hour movie and have it play in a second. The mechanism is the HTTP range request (RFC 7233).

A server that supports it advertises Accept-Ranges: bytes. Instead of GET-ting the whole object, the client asks for a slice — Range: bytes=<start>-<end> — and the server replies 206 Partial Content with a Content-Range header naming exactly which bytes it sent, streaming only that slice. Real, against the same object:

$ GET  Range: bytes=0-99          →  206 | Content-Range: bytes 0-99/31457280            | 100 bytes
$ GET  Range: bytes=31457270-     →  206 | Content-Range: bytes 31457270-31457279/…      |  10 bytes

That one header buys three things. Seeking: jump to the 10-minute mark of a video by fetching only the bytes around that timestamp — the player never downloads 0:00–9:59. Resumable downloads: a transfer that died at byte 900,000,000 resumes with Range: bytes=900000000-, guarded by If-Range: <ETag> so that if the file changed underneath you, the server sends the whole thing fresh instead of stitching old and new bytes into garbage. Parallel download: open several range requests at once and reassemble — the same speed-up as parallel part uploads, in reverse. Upload chunking and download chunking are the same idea pointed in opposite directions.

So far, where we cut the object hasn't mattered — parts are parts, and the store glues them back in order. The moment you want deduplication — storing identical content once, re-uploading only what actually changed — the cut positions become the whole game. This is the part most explanations skip, and it's the one worth slowing down for.

Start with the obvious approach: fixed-size chunking. Slice every 8 KiB. Hash each chunk. To sync an edited file, upload only the chunks whose hash changed. Simple — and it works, right up until someone inserts a byte near the front. Watch what that does, measured on a real 1 MiB file:

insert 8 bytes at the FRONT, then re-chunk:
  FIXED-SIZE (8 KiB)      128 → 129 chunks | changed 129/129 | re-upload 1024/1024 KiB | dedup  0.0%
  CONTENT-DEFINED (~8 KiB) 96 →  96 chunks | changed   1/96 | re-upload   21/1024 KiB | dedup 97.9%

Fixed-size chunking re-uploaded everything. Eight bytes at the front shoved every subsequent boundary over by eight — chunk 2 now holds what used to straddle chunks 1 and 2, chunk 3 holds what was in 2 and 3, and so on down the file. Every chunk's content changed, so every hash changed, so dedup found nothing in common between the old file and the new one. This is the boundary-shift problem, and it's why naive chunking silently wastes bandwidth.

Content-defined chunking (CDC) fixes it by cutting at boundaries the content chooses, not the ruler. Slide a small window over the bytes and keep a cheap rolling hash — the gear hash is just fp = (fp << 1) + GearTable[byte], one shift and one add per byte. Declare a chunk boundary wherever the low bits happen to be zero (fp & mask == 0); with a 13-bit mask that fires about once every 8 KiB, so chunks average ~8 KiB but land on content, not on offsets. Now insert those same 8 bytes: the boundaries near the edit wobble, but a few chunks later the window has moved past the insertion and the hash re-synchronizes to the exact same byte patterns — so the same cut points come back. Only one chunk changed. 97.9% dedup instead of zero. (Insert in the middle and it's even starker: fixed re-uploads half the file, 65 of 129 chunks; CDC still re-uploads one.)

This is the concept the interactive is built for. Type a stream, flip between fixed and content-defined, then insert a byte at the front and watch the difference — one turns the whole stream red, the other turns one chunk red. Real rolling hash, real chunk hashes, live.

If small chunks dedup better — and they do, because a change dirties a smaller unit — why not chunk into 512-byte pieces and dedup everything? Because chunk size is a U-shaped trade, and both ends cost you.

Small chunks: finer dedup, finer resume granularity, better edit-locality — but every chunk needs a fingerprint in an index, so a terabyte cut into 4 KiB chunks is 256 million fingerprints to store, look up, and keep in memory. More chunks also means more requests, more per-chunk overhead, more CPU spent hashing. Large chunks: cheap metadata, few requests, fast — but coarse dedup (change one byte, re-upload the whole big chunk) and coarse resume. Push either way far enough and total storage rises: tiny chunks drown in metadata, huge chunks store redundant data. The sweet spot sits in the middle and depends entirely on your data — measurements put source code around 2 KiB, VM images around 8 KiB, office documents around 4 KiB.

You can see the choice in two real systems that landed in different places. Dropbox uses 4 MB chunks: it's optimizing for cheap metadata and fast big-file transfer, and it accepts coarser dedup. restic, a backup tool, uses content-defined chunks averaging ~1 MB: it's optimizing for dedup across nightly snapshots, and it pays the extra metadata to get it. Neither is wrong — they weighed the same U-curve for different jobs.

The instinct behind all of it — name a chunk by the hash of its contents — is called content addressing, and once you see it you see it everywhere.

• Dropbox splits every file into 4 MB blocks, hashes each with SHA-256, and uses that hash as both the deduplication key and the storage address. A file is stored as an ordered list of block hashes; two users who upload the same movie share one copy of every block; editing a file re-uploads only the blocks that changed. Block-level dedup saves cloud providers 30–50% of raw storage.
• restic and Borg are backup tools built entirely on content-defined chunking (Borg uses a Buzhash rolling hash). A re-backup of a mostly-unchanged large dataset re-uploads almost nothing — 95–99% dedup — because only the touched chunks are new.
• git is content addressing too: every blob, tree, and commit is named by the hash of its contents, so identical files across your history are stored once.
• tus (byte-offset resumable uploads) runs Vimeo's and Cloudflare Stream's upload paths; FastCDC, the modern content-defined chunker, runs about 10× faster than the classic Rabin-fingerprint approach at nearly the same dedup ratio.

The through-line: a chunk's name is a hash of what's inside it, so identical content is identical everywhere, automatically. Dedup, resume, and integrity checking all fall out of that one idea.

• ⭐ Chunk big objects to move them politely. One PUT can't resume, can't stream in bounded memory, and hits a size ceiling (S3 single-PUT = 5 GiB). Multipart upload — Create → uploadId → UploadPart (each returns an ETag) → Complete — makes the transfer resumable, parallel, and unbounded (≤10,000 parts, up to 48.8 TiB).
• ⭐ Resume is a set difference. After a drop, ListParts tells you what landed; you re-send only the missing parts (S3, by part number) or resume from Upload-Offset (tus, by byte). Idempotent parts make retries safe.
• Range requests chunk the download. Range: bytes=… → 206 Partial Content gives you seeking, resumable downloads, and parallel fetch — upload chunking pointed the other way.
• ⭐⭐ How you cut decides whether dedup works. Fixed-size chunking dies on an inserted byte — the boundary shift changes every chunk (measured: dedup 0.0%). Content-defined chunking cuts on a rolling hash of the content, so edits stay local and dedup survives (measured: 97.9%). It's why Dropbox, restic, and git are all content-addressed.
• Chunk size is a U-curve. Small = better dedup, more metadata/CPU; large = cheap metadata, coarse dedup. Tune to your data — and don't chunk tiny objects at all; the overhead isn't worth it.

You now know how to get a 4 GB video in and out of storage without heartbreak. But a raw 4 GB upload isn't what a viewer streams — a phone on 4G needs a small version, a TV needs a big one, and the player needs to switch between them mid-play. Next, Video Transcoding Pipelines: taking that one uploaded object and re-encoding its chunks into a ladder of qualities for adaptive streaming.