Video Transcoding Pipelines

A creator uploads one file: a 4K master, tens of gigabytes, encoded for quality, not delivery. Then a million people press play — one on a phone crawling through a subway tunnel, one on a smart TV wired to gigabit fiber, one on a laptop sharing hotel Wi-Fi with three hundred strangers. You cannot send them all the same file. The phone would buffer forever; the TV would get a soft, over-compressed picture it has the bandwidth to beat.

So before anyone watches, the platform does something that feels wasteful and is actually the whole game: it takes that one master and produces many renditions of it — the same video at a range of resolutions and bitrates — chops each into small segments, and lets every viewer's player pick, second by second, the rendition that fits their screen and their network right now. That pick, made continuously and automatically, is adaptive bitrate streaming, and it's the reason video 'just plays' on a train.

This lesson is the machine behind that: how one upload becomes a ladder of renditions, how they're packaged so a player can switch between them mid-stream, and how the pipeline that builds them scales to the firehose of the world's uploads. Here's the whole pipeline on one page; the rest of the lesson walks it.

The first step is transcoding: re-encoding the master into a set of renditions, each a rung on a bitrate ladder. A rung is a (resolution, bitrate) pair — high rungs for big screens on fast networks, low rungs to keep a bad connection playing at all. This isn't resizing; it's a full re-encode at each target. Here's a real ladder, built with ffmpeg from one 12-second 1080p master:

master.mp4   11.5 MB   (1920x1080, 30fps, 12s)
  rung     resolution    bitrate     file
  1080p    1920x1080     ~4488 kbps  6.43 MB
  720p     1280x720      ~2799 kbps  4.01 MB
  480p      854x480      ~1402 kbps  2.01 MB
  360p      640x360        ~800 kbps 1.15 MB
  240p      426x240        ~401 kbps 0.58 MB

The same twelve seconds of video ranges from 6.43 MB at 1080p to 0.58 MB at 240p — about 11× smaller at the bottom. That bottom rung looks bad on a TV, but on a phone dropping to one bar it's the difference between watching and staring at a spinner. Every rung is a real, separate encode: notice the actual bitrates land almost exactly on the targets, because the encoder was told to cap them (viewers on a metered rung must not blow past it).

How many rungs, and where? That's a real design question we'll come back to — a fixed ladder is simple but wasteful, and the best platforms tune it per video. First, the ladder is useless unless a player can move between rungs while you watch. That takes segments.

A player can only switch quality if the video is delivered in small, independent pieces. So each rendition is chopped into segments — typically 2 to 6 seconds each — and a manifest lists them. This is the same chopping idea from Media Chunking, aimed at a different problem: there, chunks made an upload resumable; here, segments make a download switchable.

The dominant packaging is Apple's HLS. The player first fetches a multivariant playlist that advertises every rung, then a media playlist per rung that lists the actual segment files. Real, from the same master packaged into three rungs:

# master.m3u8 — the multivariant playlist (fetched FIRST)
#EXT-X-STREAM-INF:BANDWIDTH=4950000,RESOLUTION=1920x1080,CODECS="avc1.640028"
v0.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1540000,RESOLUTION=854x480,CODECS="avc1.64001f"
v1.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=440000,RESOLUTION=426x240,CODECS="avc1.640015"
v2.m3u8

# v0.m3u8 — the 1080p media playlist (the segment list)
#EXT-X-TARGETDURATION:4
#EXTINF:4.000000,
v0_seg0.ts
#EXTINF:4.000000,
v0_seg1.ts
#EXTINF:4.000000,
v0_seg2.ts

The player reads master.m3u8, sees each rung's BANDWIDTH and RESOLUTION, and starts pulling segments from whichever one fits. Twelve seconds at four-second segments is three segments per rung. (DASH is the other big packaging format — one .mpd XML manifest instead of .m3u8 playlists; and CMAF lets HLS and DASH share the same underlying fMP4 segment files, so you encode once and package for both instead of storing two copies.)

There's one non-negotiable rule that makes switching possible, and it's where a lot of pipelines quietly break: every segment must start with a keyframe, and the keyframes must line up across all rungs. A keyframe (an IDR frame) is a fully self-contained picture; the frames after it are just diffs against it. If a segment starts on a keyframe, it can be decoded on its own — so the player can grab segment 5 from the 480p rung right after segment 4 from the 1080p rung. Check it on our encode:

first 8 frame types:  I  P P P B P P P
keyframes: 6 of 361 frames  →  one every 60 frames (= 2s @30fps)

A keyframe exactly every two seconds, aligned to the segment grid — because we forced a closed GOP (keyint=60) and turned scene-cut detection off. Leave scene detection on and the encoder inserts extra keyframes at cuts that don't align across rungs, and the clean switch becomes a visible glitch. This is the picture worth holding in your head:

Now the hard part, and the reason all of this exists. The manifest offers rungs; the segments are switchable; but who decides which rung to play, and when? Not a server — the player, on its own, for every segment, using two signals: how fast segments are downloading (throughput) and how many seconds of video it has already buffered ahead of the playhead (buffer occupancy).

The logic is a balancing act. Pick a rung whose bitrate comfortably fits the measured throughput, but hedge with the buffer: if you've got twenty seconds banked, you can gamble on a higher rung and ride out a dip; if the buffer is draining toward empty, drop now, before it hits zero. Because if that buffer ever empties, playback rebuffers — the spinner — and rebuffering is, by a wide margin, the thing viewers hate most. The whole art is threading between two failures: too timid and someone watches 240p on a gigabit line; too greedy and the buffer starves.

There are two schools. Throughput-based ABR picks the highest rung whose bitrate is under your recent download speed times a safety margin — simple and responsive, but bandwidth estimates are noisy on mobile and it can panic. Buffer-based (the BOLA family) mostly ignores the noisy bandwidth number and steers by the buffer level itself, which is smooth — great once the buffer is full, shaky at startup. Real players blend them: throughput while the buffer is low, buffer-based once it's deep.

Reading that is one thing; feeling it is another. Drag the bandwidth around and watch the player fight to keep playing — the ladder, the buffer, and the rebuffer are all live.

Step back to how all those renditions get made. Our tiny example produced three rungs of three segments — nine independent encode jobs. Nothing about segment 2 of the 480p rung depends on segment 1, or on the 1080p rung, or on anything else. That independence is a gift: the entire transcoding stage is embarrassingly parallel. Split the master on keyframe boundaries, drop every (segment × rendition × codec) cell onto a queue, and let a fleet of workers chew through them in any order, on any machine.

At real scale the numbers get big fast. A two-hour film at four-second segments is 1,800 segments; times six rungs, times three codecs, is on the order of 32,000 independent encode jobs for a single title — and a big platform ingests thousands of hours of new video every minute. This is why transcoding is run as a batch pipeline of queues and workers, not a monolith: a job queue feeds a horizontally-scaled worker pool, each job is idempotent and retryable (a crashed worker just means the segment gets re-encoded — same input, same output), and a coordinator assembles the manifests once the cells for a title are done. The generic machinery for that — durable queues, worker pools, status tracking, idempotent retries — is a pattern you'll meet head-on in Long-Running Tasks; here the point is just that per-segment independence is what lets a two-hour film finish transcoding in minutes instead of hours.

Which codec should those encodes use? A codec is the compression scheme itself, and the choice is a three-way trade between compression efficiency, encode cost, and device support — you rarely get all three.

• H.264 / AVC (2003) is the universal floor: every device made in the last fifteen years decodes it, and it encodes cheaply. It's also the least efficient of the modern options.
• H.265 / HEVC compresses roughly 50% better than H.264 at the same quality — but it carries patent-licensing baggage that slowed adoption.
• AV1 (2018, royalty-free) is the current efficiency leader: 30–50% smaller than H.264 and about 20% smaller than HEVC at equal quality — at the cost of an encoder that can be tens of times slower, and decode support only on newer devices.

A quick real comparison — the same clip, H.264 versus AV1:

H.264 (libx264, crf23):    0.85 MB
AV1   (libaom-av1, crf32):  0.77 MB   → 10% smaller here

Only 10%? Because our source is a flat synthetic test pattern — already easy for any codec. That's the tell: codec gains depend on the content. On grainy, high-motion film, AV1's advantage opens up to that 30–50%, which is exactly why Netflix and YouTube pour CPU into AV1 for their most-watched titles. And because no single codec is both universal and efficient, big platforms ship several — H.264 so everything plays, AV1 for the modern devices that can save the bandwidth — and let each client pick the best one it can decode. More encodes, more storage, better delivery.

That content-dependence isn't just a codec footnote — it upends the ladder itself. A fixed ladder (everybody gets 1080p@4.5 Mbps, 720p@2.8 Mbps, and so on) overspends on simple content and starves complex content. A flat cartoon looks perfect at a fraction of the bitrate an action scene needs; giving them the same rungs wastes storage and CDN money on the cartoon and under-serves the action.

Netflix's answer, per-title encoding, is one of the prettier optimizations in the field. For each title, trial-encode it at many (resolution, quality) points and measure perceptual quality — VMAF — against bitrate. Plot quality versus bitrate and the best-achievable points trace a convex hull; the ideal ladder is the set of rungs sitting on that hull, and it's different for every title. Push it further — per-shot encoding (the 'dynamic optimizer') cuts the video into shots and optimizes each one, because a dark, still dialogue scene and a confetti explosion in the same movie want wildly different bitrates. The payoff is real: 20–50% less storage and bandwidth at the same quality. The ladder stops being a fixed table and becomes a per-title, even per-shot, decision.

• ⭐ One master → many renditions. Transcode the upload into a bitrate ladder (resolution × bitrate × codec). Real: one 12 s clip spanned 6.43 MB at 1080p to 0.58 MB at 240p — the low rungs keep bad networks playing.
• ⭐ Segments + manifests make it switchable. Each rung is chopped into 2–6 s segments, each starting on a keyframe, listed in an HLS/DASH manifest. Keyframes must align across rungs (closed GOP, scene-cut off) or the switch glitches. CMAF shares fMP4 segments so you encode once, package for HLS and DASH both.
• ⭐⭐ Adaptive bitrate streaming is the player's job. Using throughput + buffer, the player picks a rung per segment — climbing when the network is good, dropping before the buffer empties. Rebuffering (buffer → 0) is the worst outcome; ABR exists to avoid it.
• The build pipeline is embarrassingly parallel. Segment × rendition × codec = thousands of independent jobs (≈32k for a 2-hour title) → a queue-and-workers batch farm (the infra of Long-Running Tasks).
• Codecs and ladders are trade-offs, not defaults. H.264 for reach, AV1 for efficiency (30–50% smaller on real content); per-title / per-shot encoding tunes the ladder to the content and saves 20–50%.

The renditions you just built don't serve themselves to a million people from one server — they're pushed to the edge, close to viewers, by a CDN (CDNs: Caching at the Edge), and the older, colder titles eventually age into cheaper storage (Tiered & Cold Storage, next). Put the whole chain together — upload, chunk, transcode, package, cache, adapt — and you've essentially designed the spine of YouTube.