youtube

Starting Point

Starting point: The simplest video platform. Creators upload videos to a single server, which stores the raw file on disk. Viewers request the video, and the server streams it directly back. No transcoding, no CDN, no processing.

How it works:

1. Creator uploads the raw video file to the Video Server via HTTP.
2. Video Server writes the file to local disk.
3. Video Server records metadata (title, description, file path) in the database.
4. Viewers request the video → server reads from disk and streams the raw file.

Four critical flaws:

The most fundamental limitation: we're streaming the original file in its original format. We need to transform it into multiple formats and distribute it globally.

Solving Upload Reliability and Server Bottleneck

Problem being solved: The application server is the bottleneck for uploads. Streaming multi-GB files through the application server wastes bandwidth, ties up server threads, and makes resumable uploads difficult to implement.

Solution: Instead of uploading through the application server, the creator uploads directly to object storage (S3/GCS) using a signed URL. The application server only generates the signed URL (lightweight) and records metadata.

How it works:

1. Creator calls POST /api/videos/upload/initiate with video metadata.
2. Upload Service generates a signed URL with a cryptographic signature, granting permission to upload to a specific path in object storage, with expiration (typically 15-60 minutes).
3. Creator uses the signed URL to upload the raw video directly to object storage (S3 multipart upload or GCS resumable upload).
4. Object storage notifies the Upload Service (via webhook/event) when the upload completes.
5. Upload Service records the video metadata and triggers the processing pipeline.

Why this matters at scale:

• The application server processes ~12 upload requests/sec at peak — each is a lightweight JSON API call (~1KB). Without signed URLs, it would need to handle 12 concurrent multi-GB streams — requiring massive bandwidth and server resources.
• Object storage services (S3, GCS) are designed for exactly this pattern — they scale horizontally to handle thousands of concurrent uploads natively.

What we've solved:

• ✅ Upload reliability: Object storage provides chunked, resumable uploads natively. A 10GB upload that fails at 80% resumes from the last completed chunk.
• ✅ Server decoupling: Application servers handle only lightweight API calls, not file streams.

What's still broken:

• ❌ No transcoding: Raw 4K video in creator's codec served to all viewers.
• ❌ No CDN: All viewers fetch from the same origin — global latency.
• ❌ No content safety: Pirated or harmful content uploaded without any check.

Solving Device Compatibility and Content Quality

Problem being solved: The raw uploaded video is in a single format and resolution. Viewers on different devices and network speeds need different versions. Content must be checked for copyright and safety before publishing.

Solution: After upload completion, trigger a video processing pipeline — a DAG of tasks that transforms the raw video into multiple formats, resolutions, and codecs, while simultaneously running content safety checks and generating thumbnails.

The processing DAG:

The pipeline is structured as a Directed Acyclic Graph (DAG) — tasks that can run in parallel do, while some tasks have dependencies:

Upload Complete
     │
     ├──→ Content Safety Check ────────────────┐
     │      (copyright, nudity, violence)       │
     │                                          │
     ├──→ Transcode to 2160p (H.264 + VP9) ───→│
     ├──→ Transcode to 1080p (H.264 + VP9) ───→├──→ Pipeline Orchestrator
     ├──→ Transcode to 720p  (H.264 + VP9) ───→│      │
     ├──→ Transcode to 480p  (H.264)       ───→│      ├──→ Update metadata (PUBLISHED)
     ├──→ Transcode to 360p  (H.264)       ───→│      ├──→ Push to CDN
     │                                          │      └──→ Notify creator
     └──→ Generate thumbnails ─────────────────┘

Transcoding details:

• Each resolution is transcoded into multiple codecs: H.264 (universal), VP9 (better compression, used by Chrome), and optionally AV1 (best compression, but slow to encode).
• Each transcoded file is further split into segments (2-6 seconds each) for adaptive bitrate streaming. A 10-minute video at 1080p/H.264 produces ~100-300 segments.
• A DASH manifest (.mpd) or HLS playlist (.m3u8) is generated listing all available quality levels and their segment URLs.

What we've solved:

• ✅ Device compatibility: Multiple codecs ensure playback on any device and browser.
• ✅ Adaptive streaming: Multiple resolutions + segments enable ABR.
• ✅ Content safety: Copyright + community guideline checks before publishing.
• ✅ Thumbnails: Auto-generated from video frames.

What's still broken:

• ❌ Global delivery: All transcoded segments sit in origin storage. A viewer in Tokyo still fetches from Virginia — high latency, buffering.
• ❌ Metadata management: No database design for tracking videos, stats, and processing state.

Solving Global Latency and Bandwidth

Problem being solved: Transcoded video segments sit in origin storage (S3/GCS in a single region). Viewers worldwide experience high latency and buffering because every segment request travels to the origin.

Solution: Distribute transcoded segments to a Content Delivery Network (CDN) with edge locations worldwide. Popular videos are cached at edge nodes closest to viewers. Origin is only contacted on cache misses.

How CDN caching works for video:

The 80/20 rule at YouTube scale:

• ~20% of videos account for ~80% of views (viral content, popular creators).
• These videos are proactively pushed to all CDN edge locations.
• The remaining 80% of videos (long tail) are pulled on demand — the first viewer triggers a cache fill; subsequent viewers in that region are served from cache.

What we've solved:

• ✅ Global low latency: Sub-50ms to nearest CDN edge.
• ✅ Bandwidth distribution: Origin only handles the initial cache fill; edges handle all viewer traffic.
• ✅ Scalability for viral content: CDN absorbs traffic spikes for popular videos.

What's still remaining:

• ❌ Database design: No structured system for video metadata, stats, and processing state.
• ❌ Metadata scalability: How to handle 5,800 QPS of metadata reads?

Final Design

The final architecture adds the metadata layer, database design, and caching to complete the system.

Database design rationale:

Why PostgreSQL for metadata over NoSQL? The relationships are inherently relational (videos belong to users, jobs belong to videos), and we need ACID for status transitions (uploading → processing → published). The read volume (~5,800 QPS) is well within PostgreSQL's capability with read replicas and Redis caching for hot data.

NFR Scorecard — All Requirements Met

Deep Dive #1

Video transcoding converts a raw uploaded video into multiple formats (codecs) and resolutions to ensure compatibility across all devices and network conditions. This is the most compute-intensive operation in YouTube's entire infrastructure.

What is transcoding?

Transcoding involves three steps per output rendition:

1. Decode the original compressed video frame-by-frame.
2. Scale each frame to the target resolution (e.g., 4K → 720p).
3. Re-encode with the target codec at the target bitrate.

The Quality Ladder

YouTube generates multiple renditions, known as the quality ladder:

Why multiple codecs?

YouTube prioritizes: VP9 for Chrome/Android viewers (majority), H.264 as fallback, AV1 for high-traffic videos where the encoding cost is amortized over millions of views.

Deep Dive #2

The video processing pipeline is not a simple linear sequence — it's a Directed Acyclic Graph (DAG) where some tasks run in parallel (transcoding multiple resolutions) while others have dependencies (DASH manifest generation depends on all transcoding jobs completing).

Pipeline stages:

DAG Orchestration

The pipeline is orchestrated by a system like Apache Airflow, Temporal, or a custom DAG engine. The orchestrator:

• Tracks the state of each task (pending, running, completed, failed).
• Triggers sequential tasks only when their dependencies complete.
• Retries failed tasks with exponential backoff.
• Reports status back to the Metadata DB (so the Upload Service can show progress to creators).

Distributed transcoding optimization: For very long videos, a single rendition can be parallelized by splitting the video into time segments (e.g., 1-minute chunks), transcoding each chunk on a separate worker, and concatenating the outputs. This reduces wall-clock transcoding time from 30 minutes to ~5 minutes.

Deep Dive #3

Signed URLs are a foundational security pattern for handling file uploads at scale. Understanding why they exist and how they work is critical for any system that handles user-generated binary content.

The Problem Without Signed URLs

Without signed URLs, the video upload flow would be:

1. Creator sends the entire video file to the application server.
2. Application server streams the file to object storage.
3. Application server is a bottleneck — handling multi-GB streams ties up threads, consumes bandwidth, and requires massive server capacity.

At YouTube scale (100K uploads/day, avg 500MB each = 50 TB/day of upload bandwidth):

• Routing 50 TB/day through application servers would require hundreds of high-bandwidth servers just for proxying files.
• Every network hop doubles the latency and halves the throughput.

How Signed URLs Solve This

The cryptographic signature:

A signed URL encodes:

• Bucket + object path — where the file will be stored.
• HTTP method — typically PUT (upload only, not download).
• Expiration time — URL is valid for a limited window (e.g., 15 minutes).
• HMAC signature — HMAC-SHA256(canonical_request, service_account_private_key).

Object storage validates the signature on every request. If the signature is invalid, expired, or the request doesn't match (wrong path, wrong method), the request is rejected.

Security properties:

• Only the intended creator can upload to the specified path.
• The URL expires — even if leaked, it becomes useless after the time window.
• The URL is scoped to a specific HTTP method — a PUT-signed URL can't be used for GET (download).
• Size limits can be enforced via Content-Length-Range conditions in the policy.

Deep Dive #4

Video content delivery is the most bandwidth-intensive operation on the internet. YouTube serves over 1 billion hours of video daily. Making this work requires a sophisticated CDN strategy with multiple caching tiers.

Tiered CDN Architecture

YouTube's CDN uses a three-tier caching hierarchy:

When a viewer requests a video segment:

1. Request goes to the nearest Edge POP (point of presence).
2. If cached → served immediately (cache hit). This handles ~60-70% of requests.
3. If not cached → request falls through to the Regional Hub.
4. If cached at Regional Hub → served from there (cumulative ~85-90% hit rate).
5. If not cached anywhere → fetched from Origin, cached at both Regional Hub and Edge POP for future requests.

Content Popularity and Caching Strategy

Edge cache eviction: LRU (Least Recently Used) eviction with a minimum TTL. Popular segments stay in cache; long-tail segments are evicted quickly. Edge storage is expensive (SSDs at 200+ locations), so cache efficiency matters enormously.

Deep Dive #5

For creators, upload speed directly impacts their workflow. A creator who waits 30 minutes for a 10GB upload will consider moving to a competitor. Here are the techniques to make uploads faster:

1. Parallel Chunk Uploads

Instead of uploading the file sequentially, split it into chunks (e.g., 5MB each) and upload multiple chunks simultaneously. With 8 parallel connections:

• Sequential: 10GB at 50 Mbps = ~27 minutes
• 8× parallel: ~3.4 minutes (if network is the bottleneck)

2. Client-Side Compression

For creators with fast CPUs but slow networks, client-side compression before upload can reduce file size by 30-50%. The YouTube app could re-encode using a fast preset (e.g., H.264 ultrafast) to optimize for upload speed, knowing the server will re-transcode anyway.

3. Upload from Near Edge

Instead of uploading to a centralized US-based storage, route uploads to the nearest edge/regional PoP. The signed URL can point to a regional storage endpoint: https://asia-southeast1.storage.googleapis.com/.... The file is then replicated to the primary origin asynchronously.

4. Progressive Processing

Don't wait for the entire upload to complete before starting processing. Begin transcoding as chunks arrive:

• After the first 60 seconds of video is uploaded, start transcoding that segment.
• By the time the full video is uploaded, the first several minutes are already transcoded.
• This overlaps upload and processing time, reducing end-to-end latency.

5. Bandwidth Feedback + Progress Bars

Show creators real-time upload progress with estimated time remaining. If the upload is going to take 30+ minutes, offer to send a notification when complete instead of requiring them to keep the app open.

Deep Dive #6

Adaptive Bitrate Streaming (ABR) is what makes video playback feel seamless even when network conditions change. The video player dynamically switches between quality levels based on real-time network measurements.

How ABR Works

1. Manifest download: The player fetches the DASH MPD (or HLS M3U8) manifest, which lists all available renditions (quality × codec) and their segment URLs.
2. Segment-by-segment quality decision: For each 2-6 second segment, the ABR algorithm decides which quality to request based on:
   • Throughput estimate: Moving average of recent segment download speeds.
   • Buffer level: How many seconds of video are buffered ahead of playback.
   • Rebuffer risk: If the buffer is low, switch to lower quality to prevent stalling.
3. Seamless switching: Because segments are independently decodable, the player can switch quality between any two consecutive segments without rebuffering.

ABR Algorithm Comparison

YouTube uses a hybrid approach: primarily buffer-based with throughput as a secondary signal. The player prefers to maintain a target buffer (typically 20-40 seconds ahead) and selects the highest quality that the estimated throughput can sustain without depleting the buffer.

Deep Dive #7

YouTube's video metadata (titles, descriptions, stats, processing state) must handle ~5,800 read QPS average (29K peak) with strong consistency for writes and eventual consistency acceptable for stats counters.

Sharding Strategy

Why video_id over uploader_id for the videos table?

• Sharding by uploader_id would co-locate all of a creator's videos on one shard — convenient for "my videos" queries.
• But: a viral creator with 10M videos creates a hot shard. video_id distributes evenly.
• "My videos" queries use a secondary index or a separate creator-to-videos lookup table.

Replication Strategy

Video Stats: Write-Heavy Counter Pattern

video_stats is the hottest table — every view, like, and share is a write. Writing directly to PostgreSQL for every view would overwhelm the primary.

Solution: Redis counter + periodic flush

1. On each view: increment video_id:view_count in Redis (< 1ms, no DB write).
2. A background worker periodically (every 30-60 seconds) flushes accumulated counts to PostgreSQL in batch: UPDATE video_stats SET view_count = view_count + {delta} WHERE video_id = ?.
3. Read path: Redis has the real-time count; DB has the durable count.

This reduces DB writes by 100-1000× (thousands of Redis increments batch into one DB UPDATE).

These open-ended topics test architectural judgment and strategic thinking at the staff+ level.