twitter

Tweet Service handles writes at 21K QPS peak — it needs to be optimized for fast, durable inserts.

Timeline Service handles reads at 2.1M QPS peak — it needs to be optimized for ultra-fast cache lookups.

Follow Service handles social graph mutations (~relatively rare, bursty when a celebrity account goes viral) — it has different scaling characteristics.

Each service can scale horizontally and independently based on its own traffic patterns. A spike in timeline reads doesn't need to scale the tweet write path, and vice versa.

When a tweet is posted, multiple things need to happen: fan-out to followers' timelines, indexing for search, analytics ingestion, and (for certain tweets) notification dispatch.

If the Tweet Service called each downstream service synchronously:

• Write latency would be 200-400ms (4 sequential calls).
• If fan-out or search indexing is down, tweet posting fails.
• Adding new consumers (e.g., a new recommendation signals pipeline) requires changing the Tweet Service.

Kafka decouples all of this: Tweet Service publishes once, each downstream consumer processes independently. Adding a new consumer is zero-change to the producer. If a consumer falls behind, events queue up in Kafka — no data loss, no back-pressure on the write path.

Consider an average user with 200 followers. When they post a tweet:

• Fan-out-on-write: 200 Redis LPUSH operations (extremely fast — Redis does 100K+ ops/sec per node).
• Fan-out-on-read: Every time the 200 followers open their timeline, the system queries all 200 accounts' tweets, merges them, and sorts — 200× more reads per timeline load.

With a 100:1 read:write ratio, the math overwhelmingly favors pre-computing timelines at write time. The write cost per tweet (200 cache inserts) is amortized across 100 reads that each cost O(1) instead of O(200).

Lady Gaga has 84 million followers. A single tweet triggers 84M cache writes. Even at 100K writes/sec per Redis node, that's 840 seconds (14 minutes) across one node — or 84 seconds with 10 nodes working in parallel. During that time, the fan-out queue for her followers backs up, delaying timeline updates for millions of users.

The hybrid approach:

1. Define a "celebrity threshold" (e.g., users with > 100K followers).
2. For non-celebrities: fan-out-on-write (push). Tweet IDs are eagerly written to all followers' caches.
3. For celebrities: fan-out-on-read (pull). The tweet is NOT pushed to follower caches. Instead, when a follower loads their timeline, the Timeline Service:
   • Reads the pre-computed timeline from their cache (non-celebrity tweets).
   • Queries the recent tweets of each celebrity they follow (typically < 10 celebrities).
   • Merges the two lists and returns the combined timeline.

This hybrid approach is what the real Twitter implemented, as described by Raffi Krikorian (Twitter's VP of Engineering) in his 2013 architecture talk.

Cassandra's partition-key model maps perfectly to our access patterns:

• tweets partitioned by author_id → "get all tweets by user X" is a single partition scan.
• followers partitioned by user_id → "get all followers of user X" is a single partition scan (needed for fan-out).

The real Twitter originally used MySQL with sharding, then moved to a custom storage layer called Manhattan (essentially a distributed key-value store similar to Cassandra's model).

The Fan-out Service is the most write-intensive component. For each tweet from a non-celebrity:

1. Look up the author's follower list from Cassandra (or a cached copy).
2. For each follower, do LPUSH timeline:{follower_id} tweet_id in Redis.
3. Do LTRIM timeline:{follower_id} 0 199 to cap the list at 200.

To handle 1.4M writes/sec:

• Parallel fan-out workers: 50 worker instances consuming from Kafka. Each processes ~28K writes/sec.
• Redis pipelining: Batch 100 LPUSH commands into a single Redis pipeline — reduces round-trip overhead by 100×.
• Redis cluster: 10 Redis nodes with consistent hashing. Each node handles ~140K ops/sec — well within Redis's capability of 500K+ ops/sec per node.

Each tweet needs a unique ID. Using a centralized auto-incrementing counter is a bottleneck. Twitter invented Snowflake — a distributed ID generation service that produces 64-bit IDs containing:

• 41 bits: Timestamp (millisecond precision) — makes IDs time-sorted.
• 10 bits: Machine ID — supports 1024 generator instances.
• 12 bits: Sequence number — supports 4096 IDs per millisecond per machine.

This means each machine can generate 4096 × 1000 = 4 million IDs/sec, and the total cluster can handle trillions of IDs/sec. Since IDs embed the timestamp, sorting by ID is equivalent to sorting by time — no need for a separate timestamp column for ordering.

For the hybrid fan-out strategy, a follower's timeline is assembled from two sources:

1. Pre-computed cache (fan-out-on-write tweets from non-celebrities):
   LRANGE timeline:{user_id} 0 199 → 200 tweet IDs.

2. On-demand queries (fan-out-on-read tweets from celebrities the user follows):
   For each celebrity (say 5 celebrities), query their latest tweets from Cassandra:
   SELECT * FROM tweets WHERE author_id = ? LIMIT 20.

The Timeline Service merges these two lists, deduplicates, scores them through the ML ranker, and returns the top 20.

Performance impact: The celebrity queries add ~20-40ms to the read path (5 parallel Cassandra queries × ~5ms each). Compared to fanning out 84M writes, this is an excellent trade-off.

Not all 300M DAU are active at the same time. Many users haven't opened the app in days. Fanning out tweets to inactive users' caches wastes writes and memory.

Optimization: Only fan out to users who have been active in the last 3 days. For users who haven't been active:

• When they open the app, compute their timeline on-the-fly (fan-out-on-read) from the tweets of users they follow.
• Cache the result so subsequent reads are fast.
• Resume normal fan-out for this user going forward.

This reduces the fan-out write volume by 30-50% (since a large fraction of registered users are not daily active).

When a user follows a new account:

1. The Follow Service writes the edge to the social graph (Cassandra).
2. An async job fetches the followed user's latest 20 tweets from Cassandra.
3. These tweet IDs are merged into the follower's timeline cache (Redis).
4. The list is re-sorted by tweet_id (which is time-sorted via Snowflake) and trimmed to 200.

When a user unfollows an account:

1. The Follow Service deletes the edge from the social graph.
2. An async job removes all tweet IDs authored by the unfollowed user from the follower's timeline cache.

Both operations are eventually consistent — the user might see a brief period where the followed/unfollowed user's tweets appear or disappear.

Memory estimation:

• 300M users × 200 tweet IDs × 8 bytes per ID = 480 GB of timeline data.
• Tweet content cache: 600M tweets/day × 5 days hot window × 480 bytes = ~1.44 TB.
• Total: ~2 TB of Redis memory.

Cluster sizing:

• Redis maxes at ~25 GB of useful data per node (leaving headroom for overhead).
• 2 TB ÷ 25 GB = 80 Redis nodes minimum.
• With replication factor 2 (for HA): 160 nodes.

QPS capacity:

• Each Redis node handles ~100K-500K ops/sec depending on operation type.
• 2.1M read QPS ÷ 100K/node = 21 nodes minimum for reads.
• Combined with 1.4M write (fan-out) QPS: need at least 35 nodes for throughput.

So we're memory-bound (80 nodes) rather than QPS-bound (35 nodes). 80 primary + 80 replica = 160 nodes.

With a 5-year retention policy:

Hot tier (0-30 days): Tweets in the last 30 days are most frequently accessed. These live in Cassandra with SSD storage and are cached in Redis for the timeline.

Warm tier (30 days - 1 year): Older tweets are migrated to a cheaper Cassandra cluster with HDD storage. Still queryable for user profile pages.

Cold tier (1-5 years): Archived to object storage (S3) for compliance. Queryable via batch jobs (Spark/Presto) but not real-time.

Beyond 5 years: Deleted per retention policy. Cassandra TTLs can handle this automatically.

Cassandra TTL on the tweets table:

INSERT INTO tweets (...) VALUES (...) USING TTL 157680000;  // 5 years in seconds

Twitter serves users globally. For low latency, deploy in 3+ regions (e.g., US-East, EU-West, AP-Southeast).

• Cassandra: Native multi-DC replication with tunable consistency. Writes go to the local DC and replicate asynchronously to other DCs. Reads are served locally.
• Redis: Each region has its own Redis cluster with its own copy of timeline data. Fan-out happens locally in each region.
• Kafka: Cross-region replication with MirrorMaker 2. Tweet events published in US-East are replicated to EU-West and AP-Southeast, triggering local fan-out.

User-follows-user across regions: the Follow Service writes are replicated to all regions within seconds via Cassandra's multi-DC replication.

When a user deletes a tweet, it must be removed from:

1. Cassandra: soft-delete (mark as deleted) for compliance, then hard-delete after grace period.
2. Redis (author's profile cache): remove tweet_id.
3. Redis (every follower's timeline cache): scanning and removing a specific tweet_id from 200,000+ lists is expensive.

Practical approach: Don't eagerly remove from follower caches. Instead:

• Mark the tweet as deleted in the tweet content cache.
• When the Timeline Service hydrates tweet IDs, it skips deleted tweets.
• Over time, new tweets push deleted tweet IDs off the end of the 200-entry list.

This is a lazy cleanup approach — simple, efficient, and eventually consistent.

To prevent spam and abuse:

1. Per-user tweet rate limit: Max 300 tweets/day, max 1 tweet per 15 seconds. Enforced via Redis counter with TTL: INCR tweet_count:{user_id}:{date}, check < 300.
2. Per-user follow rate limit: Max 400 follows/day. Prevents mass-follow spam.
3. Global rate limit: If total write QPS exceeds 3× steady-state, shed traffic via HTTP 429 with Retry-After header.
4. Spam detection: Content analysis on tweets (URL blacklists, ML-based spam classifier). Flagged tweets are hidden from timelines pending review.
5. Bot detection: CAPTCHA challenges for accounts that exhibit bot-like patterns (rapid-fire tweets, immediate follower-unfollower cycles).

Currently, timelines are loaded on app open and paginated on scroll. For a more engaging experience, Twitter shows a "New tweets" indicator when fresh tweets arrive. How is this delivered?

Options:

• WebSocket/SSE: Server pushes "new tweets available" events to connected clients.
• Short polling: Client polls every 30 seconds for timeline updates.
• Long polling: Client holds an HTTP connection open; server responds when new data is available.

Twitter search requires indexing 600M tweets/day in near-real-time. A search for "Super Bowl" during the game should show tweets posted seconds ago.

Key challenges:

• Indexing throughput: 7K tweets/sec must be tokenized, analyzed, and added to the search index.
• Inverted index storage: 600M tweets/day × 30 tokens/tweet = 18 billion index entries/day.
• Query latency: Search results must return in < 200ms.
• Relevance ranking: Balance recency with engagement signals.

Twitter's "For You" feed uses a recommendation algorithm that surfaces tweets from accounts the user doesn't follow. This is a fundamentally different pipeline from the follow-based timeline.

The staff challenge: designing a recommendation pipeline that processes billions of candidate tweets across 300M users, scores them in real time, and blends them with the follow-based timeline.