Designing Your AI API (Sync, Async, Streaming)
Introduction
Welcome to Deploying & Scaling — turning a working AI feature into a production service. And it starts with the most basic decision: the shape of your API. How does a client call your AI, and how does it get the answer back?
For a normal web service this is boring — request in, JSON out, done in milliseconds. For an AI service it's the whole ballgame, because an LLM call is slow and variable: a quick classification is sub-second, a chat reply is several seconds, a deep-research or agent run is minutes, a batch job is hours. A single synchronous "send request, wait for the full answer" endpoint — the obvious default — breaks across that range: it makes a chat user stare at a frozen spinner, and it flat-out times out on anything long.
So the core skill is matching the API shape to the workload. There are five to know:
- Synchronous — request → wait → full response (short, fast).
- Streaming (SSE) — request → tokens stream back (user-facing text).
- WebSocket — a bidirectional channel (interactive: interrupts, tools, voice).
- Async job — submit → get a job id → poll or webhook (long-running).
- Batch API — upload many → results later (high volume, cheap).
Scope: this is the server-side API contract — the interface your service exposes. It's distinct from L225 (Streaming UX: TTFT, Token-by-Token & Markdown Buffering), which is how you render a stream nicely on the client, and from L219 (The Model Gateway & Router), which is about where calls are routed. Here: what shape is the endpoint itself.
The Latency Problem — Why Sync Breaks
Everything follows from one fact: you cannot predict how long an LLM call takes, and it's often long. Synchronous request/response — the client opens a connection and blocks until the full answer arrives — fails two ways:
- The dead-spinner problem (UX). Even a "fast" 4-second chat reply is an eternity to stare at a blank screen. Synchronous gives the user nothing until everything is ready — the opposite of the perceived-latency wins from L224/L225.
- The timeout cliff (it literally fails). Connections don't stay open forever. Load balancers, reverse proxies, API gateways, and browsers cut idle/long connections after ~30–60 seconds. A deep-research task that takes 3 minutes will have its connection severed mid-flight — the work may even complete on the server, but the client gets a 504 and never sees the result. You can't just "raise the timeout" past every hop you don't control.
So the question for every AI endpoint is: how long might this take, and is a human waiting on it? Those two answers pick your shape. Short + machine → sync is fine. Longer + a human watching → stream it. Minutes+ → don't hold a connection at all; hand back a job. Let's take them in turn.
Synchronous — Simple, for Short Calls
The simplest shape, and still the right one for a real slice of workloads: the client POSTs, the server calls the model, and returns the complete response in the same HTTP call.
- Use it when the call is fast and bounded (well under the timeout) and the consumer is a machine, not a waiting human watching text appear: a classification, an extraction, a short structured answer, an internal step in a larger pipeline.
- Why it's good here: nothing is simpler to build, test, and reason about — plain request/response, easy retries (L234), trivial client code. Don't add streaming or job queues to something that returns in 300ms — that's over-engineering.
- Where it breaks: the moment a human is waiting on multi-second generated text (bad UX), or the task can exceed the timeout (it fails). That's the next two shapes.
Rule of thumb: synchronous is the default for short, machine-to-machine calls — and the wrong default for everything a person watches or anything that runs long.
Streaming — SSE for Text, WebSocket for Interaction
When a human is reading generated text, you don't make them wait for the whole thing — you stream it as it's produced, so the first token shows in a fraction of a second (the TTFT win from L225 — Streaming UX). The server-side question is which streaming protocol.
- Server-Sent Events (SSE) — the default. The client sends one request and the server pushes a one-way stream of token events until done. It's plain HTTP, so every proxy, load balancer, CDN, and firewall already supports it — which is exactly why both OpenAI and Anthropic use SSE for their streaming APIs. For chat and text generation, reach for SSE first.
- WebSocket — when you need bidirectional. SSE is one-way: after the request, the client can only receive (or close). The moment the user needs to send data mid-stream — interrupt/barge-in, a follow-up that steers generation, tool calls negotiated in real time, or voice — you need a two-way channel, and that's a WebSocket. It's heavier (stateful connections, more infra), so use it for interactive/agentic/voice apps, not plain chat.
Streaming has production gotchas that bite everyone once:
- Proxy buffering — nginx buffers responses by default, so the user sees nothing for seconds then the whole thing at once. Disable it (
proxy_buffering off,X-Accel-Buffering: no). - CDN/caching — mark SSE uncacheable (
Cache-Control: no-cache, no-transform) and disable on-the-fly gzip (Content-Encoding: identity). - Keepalives — send a comment line every 15–30s so idle-timeout proxies don't drop the connection.
The decision in one line: stream generated text over SSE; upgrade to a WebSocket only when the user must talk back mid-stream.
# Streaming with SSE (FastAPI). One-way token stream — the default for chat/text.
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
@app.post("/chat")
def chat(req: ChatReq):
def gen():
for chunk in client.chat(model="gpt-5.5", messages=req.messages, stream=True):
yield f"data: {json.dumps({'delta': chunk.delta})}\n\n" # SSE event per token
yield "data: [DONE]\n\n"
return StreamingResponse(gen(), media_type="text/event-stream", headers={
"Cache-Control": "no-cache, no-transform", # don't let a CDN buffer/cache the stream
"X-Accel-Buffering": "no", # disable nginx proxy buffering
"Connection": "keep-alive",
})
# Need interrupts / tool calls / voice mid-stream? SSE can't carry client→server data — use a WebSocket.Async Jobs & Batch — for Work That Runs Long
When a task takes minutes to hours — a deep-research report, a long agent run, processing a big document — don't hold a connection at all. You'll hit the timeout cliff, and you'll tie up a server worker for the whole duration. Instead, decouple the request from the result with the async job pattern:
- The client
POSTs the request; the server validates it, enqueues the work, and immediately returns202 Acceptedwith ajob_id— in milliseconds. The connection closes. - The client gets the result one of two ways:
- Polling —
GET /jobs/{id}every few seconds until status isdone. Dead simple and robust (works even if the client can't receive inbound connections). - Webhook — the server
POSTs to a callback URL the moment the job finishes. Efficient and event-driven, but the client must expose an endpoint and handle retries + verify the signature.
- Polling —
This is exactly how OpenAI's background mode, Batch API, and Deep Research work — long tasks return a handle, and you poll or get a webhook (the webhook-id doubles as an idempotency key since events can be delivered more than once).
And for high-volume, latency-tolerant work — "summarize these 50,000 documents by tomorrow" — there's a specialized async shape: the Batch API. Upload all the requests as one file, and the provider processes them within a window (typically ~24 hours) at roughly half the cost of real-time calls. It's the right tool when you have scale and don't need it now; the wrong tool for anything a user is waiting on.
The mental model: async jobs trade immediacy for the ability to run as long as you need — and they're the only shape that survives multi-minute work without timing out.
# Async job pattern: accept fast, return a handle, deliver the result later.
@app.post("/reports", status_code=202)
def submit_report(req: ReportReq, idempotency_key: str = Header(...)):
if existing := jobs.find(idempotency_key): # a retried POST must NOT start a 2nd job
return {"job_id": existing.id, "status": existing.status}
job = jobs.create(idempotency_key, req)
queue.enqueue(run_report, job.id) # the minutes-long work runs OFF the request path
return {"job_id": job.id, "status": "queued"} # 202 — returns in ms, no timeout risk
@app.get("/reports/{job_id}") # OPTION A: client polls until done
def status(job_id):
j = jobs.get(job_id)
return {"status": j.status, "result": j.result if j.status == "done" else None}
def on_complete(job): # OPTION B: webhook — push when finished
post(job.callback_url, signed({"job_id": job.id, "status": "done", "result": job.result}))See It — The API Shape Lab
Now match shapes to workloads yourself. Pick a workload, pick an API shape, and see the request lifecycle and how well they fit:
Run the wrong pairings on purpose — they're the lesson: chat on synchronous (a dead spinner), deep-research on synchronous (times out), classify on WebSocket (absurd over-engineering), voice agent on SSE (can't carry interrupts). Each workload has a shape that fits and several that make the feature feel broken. The shape is a design decision, not a default.
The Decision Framework & Cross-Cutting Concerns
The whole lesson compresses into two questions per endpoint — how long might it take, and is a human watching? — which pick the shape:
| Workload | Shape |
|---|---|
| short, machine consumes it | Synchronous |
| a user reads generated text | Streaming (SSE) |
| interactive — interrupts, tools, voice | WebSocket |
| runs for minutes+ (research, agent) | Async job (poll / webhook) |
| huge volume, latency-tolerant | Batch API |
And regardless of shape, four things are non-negotiable:
- Timeouts — bound every model call (L234 — Fallbacks, Retries & Circuit Breakers); for long work, async is the timeout strategy.
- Idempotency keys — a client retry (or a duplicate webhook) must not start a second job or double-charge. Key the request so the second one is a no-op.
- A versioned request/response schema — your API is a contract; define it (typed), and version it so you can evolve without breaking callers (the focus of L247 — Versioning Prompts, Models & Datasets).
- Auth & rate limits — every endpoint needs them, and AI endpoints especially (they're expensive and abusable) — which is the very next lesson, L245 — Rate Limiting & Quotas.
Wrap it all behind the gateway (L219) so these concerns — shape, timeout, auth, limits, observability — are enforced in one place for every model call.
🧪 Try It Yourself
Use the API Shape Lab and what you've learned:
- Put Chat assistant on Synchronous. What's the user experience, and what's the fix?
- Put Deep-research report (2–5 min) on Synchronous. Why doesn't it just feel slow — why does it fail?
- Put Interactive voice agent on Streaming (SSE). Why isn't SSE enough — what does voice need that SSE can't give?
- Put Quick classify on Async job. It works — so why is it the wrong choice?
- You're building a feature that summarizes a user's uploaded 80-page PDF (~90s). Which shape, and why?
→ (1) A dead spinner for the whole multi-second generation — terrible UX. Fix: stream over SSE so the first token appears in ~300ms. (2) Because the connection gets cut by a load balancer/proxy at ~30–60s — the timeout cliff — so the client gets a 504 and never sees the result, even if the server finishes. Use an async job. (3) SSE is one-way — the user can't interrupt or feed input mid-turn; voice needs bidirectional real-time, i.e. a WebSocket. (4) A sub-second call doesn't need a job queue + poll loop — it's over-engineered; just use synchronous. (5) An async job (202 + job id → poll or webhook): ~90s exceeds the timeout cliff, so don't hold a connection — accept the upload, return a handle, and notify when the summary is ready (stream the final summary if a user is watching it render).
Mental-Model Corrections
- “One synchronous endpoint is fine.” It breaks across the LLM latency range — dead spinner for humans, timeout for long tasks. Match the shape to the workload.
- “Just raise the timeout for long jobs.” You don't control every hop (LB, proxy, CDN, browser), and holding a worker for minutes doesn't scale. Use an async job.
- “Streaming = WebSockets.” SSE is the default for one-way token streaming (plain HTTP, what OpenAI/Anthropic use). WebSocket is only for bidirectional (interrupts, tools, voice).
- “Stream everything for nice UX.” Streaming a one-token classifier is pointless overhead; a machine consumer wants a plain synchronous response.
- “My stream works locally, so it's done.” Proxy buffering, CDN caching, and idle timeouts silently break SSE in production — disable buffering, mark uncacheable, send keepalives.
- “Polling is primitive; always use webhooks.” Polling is simpler and more robust (no inbound endpoint, no retry/signature handling). Webhooks are efficient but operationally heavier — pick per client.
- “Retries are harmless.” Without an idempotency key, a retried submit starts a second (expensive) job or double-acts. Key every mutating request.
Key Takeaways
- An LLM call is slow & variable, so the API shape is a design decision — a naive synchronous endpoint either shows a dead spinner or times out. Ask: how long might it take, and is a human watching?
- Synchronous for short, machine-consumed calls (don't over-engineer them). Streaming (SSE) for user-facing generated text — one-way, plain HTTP, the OpenAI/Anthropic default.
- WebSocket only when bidirectional — interrupts, tool calls, voice. SSE can't carry client→server data mid-stream.
- Async job (202 + job id → poll / webhook) for long-running work (minutes+) — the only shape that survives the timeout cliff; Batch API for high-volume, latency-tolerant, ~50%-cheaper work.
- Every shape needs: a timeout (L234), an idempotency key (retries don't double-act), a versioned schema (L247), and auth & rate limits (next: L245 — Rate Limiting & Quotas) — enforced at the gateway (L219). And mind the SSE production gotchas (proxy buffering, caching, keepalives).