comment-system

A naive approach: a single service backed by a MongoDB replica set. MongoDB can handle ~30K reads/sec per node with secondaryPreferred reads. To reach 3.45M QPS, you'd need 115+ MongoDB nodes just for reads — and each node stores the full dataset.

The problem isn't just capacity — it's cost and operational complexity. A Redis cluster serving cached comment pages costs 10× less per read than a MongoDB query. By putting a cache in front of the database, we reduce DB load by 95%+.

When a comment is posted, multiple things need to happen: cache invalidation, spam check, notification, analytics. If the write service called each of these synchronously:

1. Latency: 4 sequential service calls add 200-400ms to the write path.
2. Coupling: If the spam service is down, comment posting fails.
3. Reliability: If any service call fails mid-way, we'd need distributed transactions.

Kafka decouples all of this: the write service publishes a single CommentCreated event, and each downstream consumer processes it independently. If the spam service is down, the event waits in Kafka until it recovers — no data loss.

The author expects to see their comment immediately after posting. But other readers are served from a cache that's seconds stale. How do we reconcile?

Two approaches:

1. Client-side optimistic update: After the POST returns 201 with the comment, the client inserts it into the local comment list without waiting for a server refresh. This is the simplest and most common approach (used by YouTube, Reddit, etc.).

2. Write-through cache: After writing to MongoDB, the write service also writes the updated comment page to Redis. This ensures even a page refresh shows the new comment. However, this adds complexity and potential race conditions.

Recommendation: Client-side optimistic update. It's simpler, requires no cache coordination, and covers 99% of cases.

Both are viable, but MongoDB has advantages for this workload:

PostgreSQL with TimescaleDB or Citus is also a strong choice if the team values SQL and transactional guarantees. YouTube uses a sharded MySQL setup; Reddit uses PostgreSQL.

Every time we display a comment, we need the vote count. If we computed it by counting rows in the votes collection (db.votes.count({ comment_id: X })), that's O(N) per comment — and with millions of votes on popular comments, it's far too slow.

Instead, we denormalize: store upvote_count and downvote_count directly on the comment document. When a user votes, we:

1. Insert/update the vote record in the votes collection.
2. Atomically increment/decrement the counter on the comment: db.comments.updateOne({ _id: id }, { $inc: { upvote_count: 1 } }).

The $inc operator is atomic in MongoDB — no race condition even with concurrent votes.

Write-through means: on every write, update both the DB and the cache atomically. For a comment system, this is problematic:

1. Cache key complexity: A single new comment invalidates multiple cache keys (all pages for that topic, both sort orders). Computing which cache entries to update is complex.
2. Race conditions: Two concurrent writes to the same topic could produce inconsistent cache states if not coordinated.
3. Over-engineering: With a 5-second TTL, the worst case is 5 seconds of staleness. Write-through adds complexity to save those 5 seconds — not worth it.

Recommendation: Cache-aside with event-driven invalidation. The Kafka consumer proactively invalidates on writes, and the TTL acts as a safety net.

A viral post gets millions of simultaneous readers. When the cache expires, thousands of concurrent requests all miss the cache and hit MongoDB — a "thundering herd".

Mitigations:

1. Request coalescing (singleflight): If a cache key is already being refreshed, queue new requests and serve them all from the single DB query result.
2. Stale-while-revalidate: Serve the stale cache entry while asynchronously refreshing it from MongoDB. Users see slightly stale data for one cache cycle.
3. Lock-based refresh: Only one request acquires a Redis lock to refresh the cache; others wait on the lock or serve stale.

For recent sort: cursor = created_at timestamp of the last comment on the current page.

For popular sort: cursor = (upvote_count, created_at) compound cursor. Since upvote_count isn't unique, we use created_at as a tiebreaker:

WHERE topic_id = X AND (upvote_count, created_at) < (cursor_votes, cursor_ts)
ORDER BY upvote_count DESC, created_at DESC
LIMIT 20

The compound cursor is encoded as a single opaque token (base64(votes:timestamp)) to prevent clients from manipulating cursor values.

When listing top-level comments, should we also fetch their replies?

Eager loading (embed replies in response):

• Fetch all replies for all 20 comments on the page in a single query.
• Pro: One API call, no loading spinners for replies.
• Con: If some comments have 500 replies, the response is huge.

Lazy loading (separate API call):

• Return only top-level comments with reply_count. Client shows "5 replies" link.
• When clicked, fetch replies: GET /api/comments/{id}/replies?cursor=...&size=10.
• Pro: Fast initial load, predictable response size.
• Con: Additional roundtrip for replies.

Hybrid (recommended): Embed the first 3 replies inline; show "Load more replies" for the rest. This is what YouTube and Reddit do.

Each MongoDB shard is a replica set (typically 3 nodes: 1 primary + 2 secondaries):

• Writes go to the primary.
• Reads use secondaryPreferred — reading from secondaries distributes read load and keeps the primary free for writes.
• Failover: If the primary dies, an automatic election promotes a secondary to primary within seconds. MongoDB's replica set protocol handles this transparently.

For cross-region disaster recovery, deploy one secondary in a different availability zone.

MongoDB supports TTL (Time-To-Live) indexes that automatically delete documents after a specified period:

// Auto-delete comments older than 10 years (315,360,000 seconds)
db.comments.createIndex(
  { created_at: 1 },
  { expireAfterSeconds: 315360000 }
);

MongoDB runs a background thread every 60 seconds that scans the TTL index and removes expired documents. This automatically enforces the 10-year retention policy without manual cleanup jobs.

For vote records, apply the same TTL — once the parent comment is deleted, orphaned votes serve no purpose.

Two schools of thought:

Pre-write (synchronous): Check before saving. If spam, reject the POST.

• Pro: Spam never enters the database.
• Con: Adds 50-200ms to the write path. A slow ML model blocks all comment submission.

Post-write (asynchronous): Save first, check later. If spam, soft-delete.

• Pro: Write path stays fast (~50ms). ML model doesn't block the user experience.
• Con: Spam is briefly visible (typically 1-5 seconds) before being removed.

Recommendation: Post-write async. The brief visibility window is acceptable because:

• Other users are reading from a cache that doesn't include the spam comment yet (the comment was just written; cache hasn't refreshed).
• Only the spammer sees their own comment immediately (and it's their own spam).

A viral event (e.g., Super Bowl, product launch) can spike comment volume 10-100×. Protection strategies:

1. Per-user rate limit: Max 1 comment per 10 seconds per user (Redis counter with TTL). Prevents spam bots and overly fast humans.
2. Per-topic rate limit: If a single topic exceeds 10K comments/minute, enable a write queue (Kafka) and process at a sustainable rate. New comments are accepted with a 202 Accepted + "Your comment will appear shortly" message.
3. Global circuit breaker: If total write QPS exceeds 5× steady-state, shed 50% of traffic via HTTP 429 (Too Many Requests) with a Retry-After header.
4. Auto-scaling: The Comment Service (both read and write) runs on Kubernetes with horizontal pod autoscaling triggered by CPU and QPS metrics.

For extreme spikes, we can decouple the write API from the database entirely:

1. The write service receives the comment and publishes it to a Kafka topic (instead of writing to MongoDB directly).
2. A batch consumer drains the topic and writes to MongoDB at a controlled rate.
3. The API returns 202 Accepted — the comment is guaranteed to be saved (Kafka durability) but not yet visible.

This adds write latency (seconds instead of milliseconds) but prevents MongoDB from being overwhelmed. The trade-off is acceptable during extreme spikes — users can tolerate a few seconds of delay when millions of people are commenting simultaneously.

Currently, users must refresh or paginate to see new comments. For a live discussion (e.g., during a sports event), users expect comments to appear in real time.

Staff candidates should discuss:

• WebSocket connections per topic — maintaining millions of persistent connections during a viral event requires dedicated WebSocket gateway servers.
• Server-Sent Events (SSE) — simpler than WebSocket for unidirectional pushes, but each connection holds an HTTP thread.
• Fan-out architecture: New comments are published to a Kafka topic → a fan-out service pushes them to all connected clients for that topic.
• Scaling concern: A topic with 100K concurrent viewers means 100K WebSocket connections for a single piece of content.

Disqus provides comments to 750K+ websites. This introduces multi-tenancy challenges:

• Tenant isolation: Should each website (tenant) have its own database/shard, or share infrastructure with logical separation?
• Noisy neighbor: One viral website could consume disproportionate resources. How do you implement per-tenant rate limiting and resource quotas?
• Custom branding: Each website customizes the comment widget's appearance. The API must return tenant-specific CSS/config.
• Cross-origin embedding: The comment widget runs in an iframe on third-party sites. CORS, CSP, and cookie policies (third-party cookie deprecation) affect the design.

Pure chronological sorting buries high-quality comments under noise. Platforms like Reddit and Hacker News use sophisticated ranking algorithms:

• Reddit's Best sort: Uses Wilson score confidence interval (considers both upvotes and total votes to rank reliably even with few votes).
• Hacker News: Combines upvotes, time decay, and comment author reputation.
• YouTube: Uses ML to rank by engagement signals (replies, likes, sentiment).

The staff challenge: computing a real-time ranking over millions of comments per topic while serving 3.45M QPS reads. Pre-compute or compute on-the-fly?