Rate Limiter: Interview Guide

The following topics contain open-ended architectural questions without one right answer. They are for Staff-level conversations, where you show systems thinking, trade-off analysis and strategic judgment.

Achieving Global Fairness Across Regions

Context: Your API runs in 5 AWS regions. A tenant with a global limit of 10,000 requests an hour sends traffic to all of them. If each region enforces the limit on its own, the tenant can get 5 times its limit: 10,000 an hour in each region.

Discussion Points:

• Accuracy against latency: one global store adds a cross-region round trip — tens to hundreds of milliseconds — to every request, which the hot path cannot afford
• Splitting the budget: give each region a share (2,000 an hour each) and move unused share between regions as traffic shifts
• Eventual consistency: each region enforces locally, and a background job adds up the regions and adjusts the shares for the next period
• When to enforce synchronously and when asynchronously
• Edge cases: when one region is down, should the others take over its share?

Cost-Based vs Count-Based Rate Limiting

Context: Different endpoints cost the backend very different amounts. A search takes 100 ms of CPU and touches three databases; a profile lookup takes 5 ms and is served from cache. Should they count the same against a limit?

Discussion Points:

• Weighted tokens: charge each endpoint its cost (search = 10 tokens, profile = 1); the script already takes a cost
• Fairness against simplicity: costs are fairer but harder for customers to predict and to understand
• How to set and update the weights in production: measured profiles or hand tuning
• Gaming: can a client use cheap endpoints to get around the limits on expensive ones?
• The alternative: separate limits per class of endpoint (search = 100 an hour, profile = 10,000 an hour)

Distributed Quota Reservation System

Context: You're building a batch processing API where users submit jobs that consume quota over time — "process 1 million records" uses 1 million of a daily quota. How do you prevent double-spending when requests arrive on many gateways?

Discussion Points:

• A reservation protocol: take the quota up front, release the unused part when the job finishes or times out
• Partial failures: a job crashes after using 500,000 of the 1 million it reserved — how is that recovered?
• Optimistic (assume success) against pessimistic (reserve the maximum) reservation
• Idempotency: how is a retried job kept from being charged twice?
• How does this interact with per-request rate limits — should batch jobs bypass them?

Challenging Constraints: Accuracy vs Throughput

Context: The requirements ask for exact limits at 1 million requests a second. Exactness means every check of a key is an atomic step on the one shard that holds it. What if 1% inaccuracy were acceptable — how would the architecture change?

Discussion Points:

• An exact check of one key runs on one Redis process: in our test about 130,000 to 200,000 script calls a second
• Sampling: check 1 request in 10 at 10 tokens each, with a local block — a tenth of the calls, but in a simulation about 1.9 times the limit for a client far over it
• What "99% accurate" means: 1% over the limit on average — acceptable for which limits?
• Where exactness matters: billing, security-sensitive endpoints
• Where throughput matters more: general protection of the API, absorbing floods

Soft Limits and Dynamic Quotas

Context: Your API has 80% of its capacity unused at night and hits its limits during business hours. How would you encourage more use off-peak while protecting peak capacity?

Discussion Points:

• Soft limits: allow bursts above the quota while traffic is low, warn but do not block
• Dynamic quotas: lower the limits when the system is under load (say, when CPU passes 70%)
• Time-based policies: higher limits at night, stricter at peak hours
• Credits: unused quota rolls over, like mobile data, rewarding well-behaved clients
• Fairness: how to keep one client from taking all of the off-peak capacity

Rate Limiting as a Product Feature: Tiered Pricing and Fair Usage

Context: Product wants three pricing tiers: Free (100 requests a day), Pro (10,000 a minute) and Enterprise (custom limits). Engineering needs rate limiting that supports per-customer limits that change the moment a customer upgrades.

Discussion Points:

• The moment of upgrade: if a customer was limited at 100 a day and upgrades to Pro at noon, should their bucket be refilled?
• Downgrades: enforce the lower limit at once, or give a grace period?
• Abuse of the free tier: many accounts, rotating keys
• Soft limits as a product: warn at 80%, throttle at 100%, block at 120%
• Who decides what "fair use" is: product, billing and engineering together

The following table summarizes what interviewers typically expect at each seniority level when discussing the rate limiter design.

Dimension	Mid-Level (L4)	Senior (L5)	Staff (L6)
Requirements	Identify the core needs (reject over the limit, tell the client, bursts) and do the volume math	Set the latency budget, the failure behaviour and how exact the limits must be	Challenge the constraints: do limits need to be exact at 1M requests a second? What changes if not?
Placement	Explain why the check runs in the gateway, before any backend work	Compare the gateway filter, a rate-limit service and local limits: round trips, accuracy, operations	Combine local and global limits, and say what each guarantees when the other fails
Algorithm	Explain the token bucket and size a bucket for a requirement	Compare fixed window, sliding log, sliding counter, leaky bucket and token bucket with numbers: memory, bursts, accuracy	Pick per limit: windows for long quotas, buckets for bursts; know GCRA and the leaky bucket's two meanings
State and Concurrency	Use Redis with one key per client	Make check-and-update atomic with a script; use Redis's clock; set the expiry to the refill time; several rules per request	Hot keys (rejection cache, leasing, splitting), cluster slots, what failover loses
Failure Handling	Know that Redis can fail and what the client sees	Timeouts, circuit breaker, fail open or closed, local fallback; recovery	Degraded-mode guarantees: how far over a limit a client can get, and for how long
Configuration	A rule format: scope, capacity, rate	Push and pull with versions, changes live within 60 s, shadow mode	Per-customer overrides, rollout by client, tiers and upgrades as a product

Core Architecture in 60 Seconds

"The check runs in the gateway, before any backend work." A per-IP rule before authentication, per-key and per-tenant rules after it. A rejected request costs one Redis round trip and gets a 429 with Retry-After.

"The counts live in a shared Redis Cluster, one bucket per rule and identity." Every gateway sees one count per limit. A token bucket allows bursts up to its capacity and holds the long-term rate; the bucket is two fields, gone once it would be full again.

"One script per check makes it atomic." Redis runs it on one thread with nothing in between; it uses Redis's clock, so gateway clocks do not matter. A request's three checks go together, in parallel to their shards.

"When Redis fails, the API does not." Timeouts in milliseconds, a circuit breaker, fail open to coarse local limits. Hot keys get a local rejection cache, leased tokens or a split bucket.

Key Trade-offs to Mention

Trade-off	Option A	Option B
Where the check runs	Gateway filter (one round trip)	Rate-limit service (one more hop; shared logic, separate deploys)
Where counts live	Shared Redis (one count per limit)	Local memory (no hop; N gateways, N counts)
Algorithm	Token bucket (bursts up to the capacity)	Fixed window (simplest; 2× at a window boundary; fine for long quotas)
Atomicity	Script (one round trip)	WATCH/MULTI/EXEC (several round trips, retries on busy keys)
Redis down	Fail open with local limits (API up)	Fail closed (no abuse, no API)
Several rules	Send together, charge all (one round trip)	In order, stop at the first rejection (no charge, more round trips)

Numbers to Remember

What	Number
A Redis round trip in a data center	~200 µs (1 Gbit/s network)
The token-bucket script on the server	~3 µs (INCR: 0.3 µs)
Script calls per Redis process	~120,000 to 180,000 a second (our test)
One token bucket in Redis	~132 bytes (GCRA: 84)
A sliding log of 1,000 requests	~103 KB
Fixed window at a boundary	up to 2× the limit
Cloudflare's sliding window counter	0.003% of decisions wrong
This design at peak	3M checks a second on 50 Redis primaries

Common Mistakes to Avoid

1. Starting with the algorithm instead of the placement. Where the check runs and where the counts live shape everything after.
2. Ignoring failure modes. "We'll use Redis" without saying what happens when it is slow or down is a mid-level answer.
3. Trusting the gateways' clocks. Pass the time into the script from each gateway and clocks that disagree add or remove tokens; use Redis's TIME.
4. Forgetting hot keys. One key lives on one shard; a cluster cannot split it.
5. Numbers without a source. "Redis adds 1–2 ms" or "the script adds 0.1 ms" sound precise and are wrong by a factor of five to thirty. Say what you measured, or what the documentation says.