Rate Limiting & Quotas
Introduction
In L244 (Designing Your AI API) we ended on a promise: every AI endpoint needs auth and rate limits. This is that lesson. Rate limiting is how you keep an AI service affordable, fair, and abuse-resistant — and it has a twist that trips up everyone coming from normal web APIs.
Why meter at all? Because an AI call is uniquely costly to leave open:
- Cost — you're billed per token, so an unmetered client (or a bug, or an attacker) can run up a five-figure bill overnight.
- Fairness / capacity — one heavy user must not be able to starve everyone else of throughput.
- Abuse / DoS — unbounded consumption is OWASP LLM10 (from L239 — The OWASP LLM Top 10); mass querying can also be used to extract your model.
- Your own ceiling — you have provider RPM/TPM caps too, so you must shape traffic to stay under them.
The AI-specific twist — and the heart of this lesson: you limit on two dimensions at once, not one. A normal API counts requests. An LLM API also counts tokens, because tokens are the real unit of cost and capacity — and the token limit usually bites first.
Scope: this is you rate-limiting your clients (server-side enforcement). It's the mirror of L234 (Fallbacks, Retries & Circuit Breakers), which was about handling being rate-limited by an upstream provider. Same headers, opposite side of the wire.
The Twist — Limit Tokens, Not Just Requests
Here's the mistake that causes mysterious throttling and deadlocked queues: counting only requests. LLM providers — and you, in your own service — limit on both:
- RPM (requests per minute) — how many calls you accept.
- TPM (tokens per minute) — how many tokens (prompt + completion) you process.
Either can throttle independently, and TPM usually binds first, because requests are wildly uneven in size. A single call with a 200,000-token context costs as much as fifty 4,000-token calls. So a service that's nowhere near its request limit can be maxed out on tokens — and a naive request-only limiter lets those huge calls through until the token budget (and your bill) explodes.
The 2025 best practice is a dual-bucket design — two token-buckets running in parallel, one for requests and one for tokens — and a request is admitted only if both have room. And critically, pre-estimate the tokens before the call (with tiktoken / count_tokens), or you hit the worst failure mode: you queue 500 requests that all fit RPM, none fit TPM, and the queue deadlocks.
The lab makes this visceral: hold RPM fixed and drag tokens-per-request — watch the binding limit flip from RPM to TPM, and your served throughput collapse, with the request count never changing. In LLM land, tokens are the budget.
The Algorithms — Token Bucket & Friends
How do you actually enforce a limit? Four classic algorithms, each a different trade between burst-tolerance and accuracy:
- Token bucket (the default) — a bucket holds up to N tokens and refills at a steady rate; each request spends a token. It allows short bursts (spend the whole bucket) then settles to the refill rate. Flexible and the most common choice — and note the name overload: here a "token" is a request permit, separate from LLM tokens (for TPM, the bucket literally holds LLM tokens).
- Leaky bucket — requests drain at a constant rate regardless of arrival; smooths bursts into a steady stream. Stricter, less bursty.
- Fixed window — count requests per clock-minute. Dead simple, but allows a double burst at the boundary (max at 11:59:59 + max at 12:00:00).
- Sliding window — a rolling 60-second count (log or weighted counter). Accurate, fixes the boundary burst, slightly more state.
In production these run in a shared, atomic store — typically Redis — so the count is correct across all your server instances (a per-process counter is useless behind a load balancer). And the cleanest place to enforce them is the model gateway (L219 — The Model Gateway & Router), so one policy covers every service and entry point.
For most AI services: token bucket (for bursts) on both the RPM and TPM dimensions, in Redis, at the gateway. Reach for sliding-window when you need strict, boundary-accurate counting.
# Dual token-bucket: a request is admitted only if BOTH the request- and token-buckets have room.
# (Atomic in Redis in production; simplified here.)
class DualLimiter:
def __init__(self, rpm, tpm):
self.req = TokenBucket(capacity=rpm, refill_per_sec=rpm / 60)
self.tok = TokenBucket(capacity=tpm, refill_per_sec=tpm / 60)
def admit(self, est_tokens: int): # est_tokens = PRE-FLIGHT estimate (tiktoken/count_tokens)
if not self.req.has(1):
return Reject(limit="RPM", retry_after=self.req.seconds_until(1))
if not self.tok.has(est_tokens): # the big-context call fails HERE, long before RPM
return Reject(limit="TPM", retry_after=self.tok.seconds_until(est_tokens))
self.req.take(1); self.tok.take(est_tokens) # spend from BOTH buckets
return Admit()
# after the call, reconcile self.tok with the ACTUAL tokens used (completion length is unknown up front)Rate Limits vs. Quotas — and Tiering Them
Two words people use interchangeably and shouldn't, because they solve different problems:
- A rate limit is a short-window control — "you may do X per minute." It smooths bursts and protects capacity moment-to-moment. Exceed it and you get throttled right now (a 429), but you can try again in seconds.
- A quota is a long-window budget — "you may do Y per day / month" (often in tokens or dollars). It caps total spend / allowance. Exceed it and you're done until the period resets (or you upgrade) — it's about billing and entitlement, not instantaneous capacity.
You need both: a rate limit so a burst can't overwhelm you, and a quota so a (legitimate or runaway) user can't blow the monthly budget. And you tier them by plan — free vs pro vs enterprise get different RPM/TPM and monthly token/$ allowances (exactly how the providers do it: OpenAI scales from Tier 1's ~500 RPM to Tier 5's ~10,000). Enforce across three layers so the blast radius of any one actor is bounded:
- Global — protect the whole service / your provider ceiling.
- Per tenant — one customer org can't starve the others.
- Per user / key — one user (or a leaked key) can't starve their own org.
Rule of thumb: rate limit = "slow down now," quota = "you're out of allowance." Tier both, enforce at every layer, and tie the quota to your cost controls (the cost-optimization section) so limits and billing agree.
When You Throttle — the 429 Contract
Throttling isn't just blocking — it's a conversation with the client, and being a good API citizen makes the difference between a client that backs off gracefully and one that hammers you into a worse outage:
- Return
429 Too Many Requests— the standard status for rate/quota exceeded. - Include
Retry-After— tell the client exactly how long to wait. This is the single most useful header; a client that honors it recovers cleanly. - Include
RateLimit-*headers — the standardRateLimit-Limit/-Remaining/-Reset(and providers'x-ratelimit-*) so well-behaved clients can self-throttle before they hit the wall. - Distinguish rate-limit from quota — a retryable "slow down" (429 + short Retry-After) is different from "you're out of monthly quota" (429 with an
insufficient_quota-style code — don't tell them to retry in 2 seconds; tell them to upgrade).
And the mirror image, from L234 (Fallbacks, Retries & Circuit Breakers): when you are the client being throttled by a provider, honor their Retry-After instead of out-clevering it — retrying sooner isn't smart, it just counts against your limit and makes things worse.
Finally, degrade gracefully rather than hard-failing the user: queue the request, shed load, or fall back to a cheaper/smaller model that's within budget. A throttle should feel like "a moment, please," not a brick wall.
The whole contract: say 429, say when to come back (Retry-After), expose the budget (RateLimit-*), and soften the landing (degrade).
from fastapi import Request
from fastapi.responses import JSONResponse
@app.middleware("http")
async def rate_limit(request: Request, call_next):
dec = limiter.admit_for(request.state.api_key, est_tokens=estimate_tokens(request))
if not dec.ok:
return JSONResponse(status_code=429, content={"error": dec.reason}, headers={
"Retry-After": str(dec.retry_after), # exactly how long to wait
"RateLimit-Limit": str(dec.limit), "RateLimit-Remaining": "0",
"RateLimit-Reset": str(dec.reset), # standard headers → clients self-throttle
})
return await call_next(request)
# quota exceeded? → 429 with an "insufficient_quota" code: do NOT advise a short retry — advise an upgrade.See It — The Rate-Limit Lab
Meter a service yourself and watch the two limits fight. Pick a tier, then drag the traffic and tokens-per-request — and find the ceiling:
The reveal: at a normal request size you're already TPM-bound while RPM sits half-empty — so raising your request limit would do nothing. Shrink the token size and RPM takes over; grow it and TPM chokes you at a fraction of the request cap. The binding limit moves with token size — which is exactly why request-only limiting silently fails for LLM apps.
🧪 Try It Yourself
Use the Rate-Limit Lab and what you've learned:
- On Pro with the default token size, which limit is binding — and why is that surprising if you only think in requests?
- Drag tokens/request down to the minimum. What becomes the binding limit, and why?
- Drag tokens/request up high. Why does served throughput collapse even though RPM has tons of headroom?
- Why must you pre-estimate tokens before admitting a request, not just count them after?
- A user hits the limit. Which HTTP status and which header do you return, and how does that differ if they're out of their monthly quota?
→ (1) TPM is binding while RPM is half-empty — surprising because the request count looks fine, but each request carries thousands of tokens, so the token budget runs out first. (2) RPM — tiny requests sip tokens, so the token budget is huge relative to the request cap, and the request limit binds. (3) Because each request now eats a big slice of the TPM budget, so only a handful fit per minute — TPM throttles you at a fraction of the RPM cap. (4) Because if you admit on RPM and only count tokens after, you can admit a queue that all fits RPM but busts TPM — the queue deadlocks (none can actually run). Estimate up front (tiktoken/count_tokens) and reconcile after. (5) 429 + Retry-After (plus RateLimit-*), advising a short wait. For an exhausted monthly quota it's still 429 but with an insufficient_quota-style code — don't advise a 2-second retry; advise an upgrade (the allowance won't reset for the period).
Mental-Model Corrections
- “Limit requests per minute and you're done.” No — TPM usually binds first. Run dual buckets (RPM and TPM); a few big-context calls blow the token budget while RPM looks fine.
- “Count tokens after the call.” Too late — you'll admit a queue that fits RPM but deadlocks on TPM. Pre-estimate tokens before admitting.
- “Rate limit and quota are the same thing.” Rate limit = per-minute burst control ("slow down now"); quota = per-day/month budget ("out of allowance"). You need both.
- “An in-process counter is fine.” Behind a load balancer it's wrong — use a shared atomic store (Redis) so the count is global.
- “Just block over-limit requests.” Return 429 + Retry-After + RateLimit- headers* so clients back off precisely, and degrade gracefully (queue / cheaper model) instead of a brick wall.
- “When a provider 429s me, retry fast.” Honor their Retry-After — retrying sooner counts against your limit and worsens the outage (L234).
- “One global limit is enough.” Enforce global + per-tenant + per-user so a single actor (or leaked key) can't starve the pool.
Key Takeaways
- Meter AI calls to protect cost, fairness, and against abuse (unbounded consumption, LLM10) — and to stay under your own provider caps.
- The LLM twist — limit on two axes: RPM and TPM, with dual token-buckets, because tokens are the real cost/capacity unit and TPM usually binds first. Pre-estimate tokens before admitting, or the queue deadlocks.
- Know the algorithms: token bucket (bursty, the default) · sliding window (accurate) · leaky bucket / fixed window — run them in a shared store (Redis) at the gateway (L219).
- Rate limit ≠ quota: rate = per-minute "slow down now"; quota = per-day/month "out of allowance" ($/tokens). Tier both (free/pro/enterprise) and enforce per global / tenant / user.
- Be a good citizen on a throttle: 429 + Retry-After + RateLimit-* headers, distinguish rate-vs-quota, and degrade gracefully; conversely honor a provider's Retry-After (L234). Next: CI/CD for Prompts & Evals.