Streaming for Perceived Latency

Introduction

This is the finale of the Inference & Latency section — and it's the lesson that makes the other five pay off for your users. We spent the section grinding down the real numbers: TTFT via prefill and the KV cache (L207-L208), TPOT via batching, speculative decoding, and a tuned server (L209-L211). Streaming is the final, almost-free move that converts those numbers into a product that feels fast.

Here's the surprise: streaming does not make generation finish any sooner. The total time is identical. What it changes is perceived latency — the wait the user actually feels — by showing the first token immediately and letting them read along, instead of staring at a blank screen until the whole answer is ready. It collapses the felt wait from E2E all the way down to TTFT.

In this lesson:

The principle — why a streamed answer feels far faster than an identical non-streamed one
How it works on the wire — Server-Sent Events (SSE) and the OpenAI / Anthropic streaming protocols
Streaming in code — the client loop and the server-side SSE relay (FastAPI)
The tradeoffs — mid-stream errors, validation, and the proxy-buffering trap that silently kills streaming

Scope: this is the mechanism and the principle. The product side — treating latency as a product feature (L224) and the deeper streaming UX like token-by-token rendering and markdown buffering (L225) — lives in the UX section; we'll point there, not cover it. Output guardrails on a streamed response are L232. Next, the course turns to cost — Token Economics (L213).

Infographic titled 'Streaming for Perceived Latency'. The big idea: streaming does NOT make generation finish sooner — it makes the wait FELT shrink from the whole end-to-end time down to just the time-to-first-token. CENTER, two timelines of the SAME response, both finishing at the same moment. NON-STREAMING: a long blank 'generating…' bar for the entire end-to-end time (the user stares at a blank screen, unsure it's even working), then the full answer appears all at once at the end; the FELT wait is the entire bar (E2E). STREAMING: a short red silence segment until the first token (TTFT), then green tokens flow in faster than anyone can read until the end; the FELT wait is only that short TTFT segment. Both finish together, but streaming's perceived wait is many times smaller — for example 300 milliseconds felt instead of 5 seconds. This is the payoff of the whole section: it spent five lessons cutting TTFT (prefill, L208) and TPOT (decode, L209-L211); streaming is what turns those metrics into a product that FEELS fast (L207). HOW IT WORKS ON THE WIRE: Server-Sent Events (SSE) — the server holds one HTTP connection open with content-type text/event-stream and pushes 'data:' chunks as each token is ready; OpenAI sends choices delta content chunks ending with data [DONE], Anthropic sends typed events (message_start, content_block_delta with a text_delta, message_stop). SSE is one-way server-to-client and works through firewalls, load balancers and CDNs; use WebSockets only when you need two-way. THREE CARDS. Card 1, the principle: humans perceive a feedback-less wait as longer than it is, and studies rate streaming responses 'faster' even when the total time is identical; users tolerate a slow total far better than a slow first token. Card 2, the wire: SSE text/event-stream, data chunks, the client appends each delta; server-side it is a FastAPI StreamingResponse, and you must disable proxy buffering or streaming silently degrades back to batch. Card 3, the tradeoffs: once you send HTTP 200 and start streaming you can't signal an error with a status code — mid-stream errors must be sent as stream events; you can't validate or guardrail the output until it is complete; and a mid-stream provider failure leaves the client holding partial data. Section roadmap strip: metrics, KV cache, batching, speculative decoding, serving with vLLM/TGI, streaming. Takeaway banner: streaming is the cheapest latency win — same total time, but the felt wait collapses from E2E to TTFT; just don't let a buffering proxy quietly turn it back into batch.

The Principle — Perceived Latency Beats Actual Latency

Recall the metrics from L207: TTFT (time to first token), TPOT (time per output token after that), and E2E (the whole response). The non-obvious truth is that users don't experience E2E — they experience TTFT plus the reading pace. Two facts from UX research make this concrete:

A wait without feedback feels longer than it is. A blank screen creates uncertainty — "is it even working?" — and uncertainty amplifies perceived delay. Classic response-time thresholds: under 1 second keeps a user's train of thought; past 1 second they start to wonder if it's broken; 10 seconds is the outer edge of holding attention. A 5-second blank screen blows past all of that.
Progress feedback resets the clock. Studies find users rate a streaming response as faster than a batch one even when the total time is identical. Watching words appear is proof the system is working, so the felt wait shrinks to the moment the first token lands.

So the single most important number for perceived speed is TTFT — and a response that streams at faster than reading speed (recall ~10 tokens/sec beats human reading, L207) means the user is never actually waiting after that first token; they're reading. A 30-second report that streams with a 400 ms TTFT feels responsive; the same 30 seconds delivered as one silent block feels broken.

This is why every major chat product streams. It's the highest-leverage latency win available — it costs almost nothing to implement and it changes the entire feel of the app, because it attacks the number the user actually feels.

On the Wire — Server-Sent Events (SSE)

How does a token "appear" on the client the instant the model produces it? The transport is Server-Sent Events (SSE) — and it's simpler than it sounds. The server responds with Content-Type: text/event-stream and holds the HTTP connection open, pushing a small data: chunk for each token as it's generated, until it sends a final "done" marker. It's "HTTP with the door held open." Every major provider — OpenAI, Anthropic, Google — uses SSE. Here's what's actually on the wire:

# OpenAI-style stream (stream=True): each chunk carries a `delta`, ending with a sentinel
Content-Type: text/event-stream

data: {"choices":[{"delta":{"content":"Stream"}}]}

data: {"choices":[{"delta":{"content":"ing"}}]}

data: [DONE]

# Anthropic-style stream: TYPED events (note the `event:` line)
event: message_start
data: {"type":"message_start", ...}

event: content_block_delta
data: {"type":"content_block_delta","delta":{"type":"text_delta","text":"Stream"}}

event: message_stop
data: {"type":"message_stop"}

A few things worth knowing:

It's one-way (server → client) over plain HTTP, which is exactly what token streaming needs. That's why providers use SSE, not WebSockets: SSE is browser-native, needs no special handshake, and sails through firewalls, load balancers, and CDNs. Reach for WebSockets only when you need two-way real-time (e.g. voice, live collaboration) — for one-way text streaming, SSE is the right, simpler tool.
The shapes differ but the idea is identical: OpenAI sends choices[0].delta.content and ends with data: [DONE]; Anthropic sends typed events (message_start → many content_block_delta with a text_delta → message_stop). The SDKs hide this — you usually iterate a clean token stream — but knowing the wire format is what lets you relay it yourself (next).

Streaming in Code — Client & Server

On the client, the SDK turns that wire format into a simple loop. With Anthropic you open a stream and iterate text_stream (this is the stream=True you saw served by vLLM in L211, in its Claude form):

import anthropic

client = anthropic.Anthropic()

with client.messages.stream(
    model="claude-opus-4-8",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain streaming in two sentences."}],
) as stream:
    for text in stream.text_stream:          # one chunk per token group, as it arrives
        print(text, end="", flush=True)      # <-- flush! or your terminal/buffer hides the stream
    final = stream.get_final_message()        # full message + usage, once complete

But your users aren't calling Anthropic directly — your server is. So the real job is to relay the model stream out to the browser as your own SSE stream. In FastAPI that's a StreamingResponse that yields SSE frames:

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import anthropic, json

app = FastAPI()
client = anthropic.Anthropic()

@app.post("/chat")
def chat(prompt: str):
    def sse():                                            # a generator that yields SSE frames
        with client.messages.stream(
            model="claude-opus-4-8", max_tokens=1024,
            messages=[{"role": "user", "content": prompt}],
        ) as stream:
            for text in stream.text_stream:
                yield f"data: {json.dumps({'text': text})}\n\n"   # one SSE frame per token
        yield "data: [DONE]\n\n"                          # tell the client we're done
    return StreamingResponse(
        sse(),
        media_type="text/event-stream",
        headers={"X-Accel-Buffering": "no"},              # <-- disable proxy buffering (see Tradeoffs)
    )

That's the whole pattern: model stream → your generator → SSE frames → browser. The browser consumes it with the native EventSource API (or fetch + a stream reader) and appends each token to the UI. The mechanics of rendering it well — buffering partial markdown, the blinking cursor, smoothing — is L225; here, the point is the plumbing.

See It — The Streaming Lab

Feel the difference directly. Below is the same response delivered two ways, with one time scrubber. Drag the slider and watch what the user sees at each moment:

The same response, shown two ways, with one time scrubber. Drag the time slider: NON-STREAMING is a blank 'generating…' screen for the whole end-to-end time, then dumps the full answer; STREAMING shows the first word at TTFT and flows token-by-token, so mid-generation the user is already reading while the non-streaming panel is still blank. Both panels finish at the SAME moment — streaming doesn't lower total time, it collapses the *perceived* wait from E2E down to TTFT. Switch scenarios to feel it on a short reply vs a long report. Illustrative timing.

Drag it and the lesson lands viscerally:

Mid-generation, streaming already shows a usable answer while non-streaming is still a blank screen. That gap — for the entire duration — is the perceived-latency win.
Both panels finish at the exact same time. Streaming didn't speed up the model; it speeded up the experience.
The felt wait drops from E2E (the whole bar, non-streaming) to TTFT (the first token, streaming) — often a 10×+ reduction in perceived latency for zero change in actual compute.

The Tradeoffs — What Streaming Costs You

Streaming is a near-free perceived win, but it genuinely complicates your engineering. Four things to plan for:

You can't change your mind about the HTTP status. Once you've sent 200 OK and started streaming, the headers are gone — so an error that happens mid-generation can't be a 500. It has to be sent as a stream event (e.g. a data: {"error": …} frame), and your client has to distinguish a reported error from a dropped connection. Plan your error protocol up front.
You can't validate or guardrail an unfinished answer. Want to check the JSON is valid, run an output guardrail (L232), or verify a claim before showing it? You can't — half a response isn't checkable. So streaming is in tension with structured output and output validation: either stream the raw text and validate at the end, or buffer and lose the streaming benefit. (Incremental JSON parsers exist, but they're extra machinery.)
A mid-stream failure leaves the user holding half an answer. If the model or network dies at token 200 of 400, you can't transparently retry on another provider — the client already has partial data. The usual split: in a UI, keep the partial and let the user retry; in an agent pipeline, discard and retry from scratch (a half-answer is useless to the next step).
The silent killer: a buffering proxy. This one bites everyone. If a reverse proxy (nginx, a load balancer, a CDN) buffers the full response before forwarding, your streaming silently degrades back to batch — the code is perfect, but the user sees a blank screen then a dump. Disable upstream buffering (e.g. X-Accel-Buffering: no, proxy_buffering off) and raise the default proxy timeouts, which are usually too short for long generations.

None of these are reasons not to stream — every chat product streams. They're the checklist for streaming correctly: a stream-event error channel, validation deferred to completion (or buffered), a partial-failure policy, and buffering turned off end-to-end.

🧪 Try It Yourself

Reason through these, then confirm with the lab:

Predict: in the lab, does dragging to the end make the streaming and non-streaming panels finish at different times? What is different between them?
Your chat app streams perfectly in local dev, but in production users see a blank screen then the whole answer at once. What's the most likely cause?
You need the model to return valid JSON that your code parses. What's the tension with streaming, and how do you resolve it?
Mid-stream, your LLM provider 500s at token 150 of 300. What can you do in a chat UI vs in an agent pipeline, and why are they different?
Why do OpenAI and Anthropic use SSE instead of WebSockets for token streaming?

→ (1) No — both finish at the same total time; streaming doesn't lower E2E. What differs is perceived latency: streaming showed a usable answer the whole way (felt wait ≈ TTFT), non-streaming was blank until the end (felt wait = E2E). (2) A buffering reverse proxy / CDN is buffering the full response before forwarding — disable upstream buffering (X-Accel-Buffering: no / proxy_buffering off) and check proxy timeouts. (3) You can't validate JSON until it's complete, so either stream the text and parse/validate at the end (losing nothing if the UI just shows text), or buffer the response and forgo streaming for that call — pick per use-case. (4) In a UI, keep the partial answer and offer retry (a half-answer is still readable). In an agent pipeline, the partial is useless to the next step, so discard and retry from scratch. You can't transparently failover mid-stream because the client already received partial data. (5) Token streaming is one-way (server → client) over plain HTTP; SSE is browser-native and passes through firewalls/LBs/CDNs with no special setup. WebSockets add bidirectional complexity you don't need until you have a two-way use case.

Mental-Model Corrections

"Streaming makes the model faster." No — total time is unchanged. It lowers perceived latency by showing the first token at TTFT and letting the user read along, instead of waiting for E2E.
"Perceived latency is fluff; only real latency matters." Users only experience perceived latency. A blank screen feels broken; streamed tokens feel fast — at identical clock time. It's the highest-leverage UX win you have.
"Streaming needs WebSockets." No — it's one-way, so SSE (plain HTTP, text/event-stream) is the right tool. WebSockets are for bidirectional real-time.
"I'm streaming in code, so users get a stream." Not if a proxy buffers it — buffering silently turns streaming back into batch. Disable buffering end-to-end.
"I can validate / guardrail the response as it streams." Not a partial one — you can only validate a complete answer. Streaming trades away pre-display validation; defer it to completion or buffer.
"On a mid-stream error I'll just return a 500." You can't — the 200 headers already went out. Mid-stream errors must be stream events, and the client must handle them.
"Streaming and structured output go great together." They're in tension — you can't parse half a JSON object. Choose: stream raw text, or buffer for structured/validated output.

Key Takeaways

Streaming trades perceived latency, not actual. Same total time, but the felt wait collapses from E2E to TTFT — because users experience the first token + reading pace, not the whole response (L207).
It works because feedback beats silence. A blank screen feels broken; streamed tokens feel fast at identical clock time. It's the cheapest, highest-leverage latency win there is.
The transport is SSE (text/event-stream, data: chunks): OpenAI sends delta.content + [DONE]; Anthropic sends typed events (content_block_delta / text_delta). One-way, so SSE not WebSockets.
In code: iterate the SDK's token stream on the client; on the server, relay it as a StreamingResponse of SSE frames — and disable proxy buffering (X-Accel-Buffering: no) or it silently reverts to batch.
The tradeoffs: mid-stream errors must be stream events (no HTTP status); you can't validate/guardrail until complete (tension with structured output, L232); a mid-stream failure leaves partial output (UI: keep; agent: retry).
Defer the product/UX layer: latency as a product feature is L224; token-by-token rendering & markdown buffering is L225.
Section complete — Inference & Latency: metrics (L207) → KV cache (L208) → batching (L209) → speculative decoding (L210) → serving (L211) → streaming (L212). You can now make a model fast and make it feel fast.
Next — L213: Token Economics: Where the Money Goes — the section pivots from latency to cost: now that it's fast, make it cheap.