Treating Latency as a Product Feature

Welcome to a new section: Designing AI Product Experiences. The last two sections made the system fast (Inference & Latency) and gave you the architecture to run it. Now we make it feel great to use — and we start with the most fundamental UX fact of AI products: they're slow, and the wait is variable. A chatbot thinks for seconds; an agent runs for minutes. You can't always fix that with engineering. So you do the next best thing — you design the wait.

The reframe that defines this lesson: latency is a product feature, not just an engineering metric. Container 1 (L207) taught you to measure latency (TTFT, TPOT, E2E) and minimize it. This lesson is about the other half — the part you control even when the model is slow: perceived latency. Two products with the identical 3-second response time can feel like a snappy tool or a broken one, depending entirely on how you show the wait.

In this lesson:

• The human thresholds — the hard perceptual limits (0.4s, 1s, 10s) you design against
• Perceived vs actual — the gap you exploit, and why it's where the UX lives
• The waiting toolkit — optimistic UI, skeletons, progress, streaming — matched to the wait length
• Latency budgets per surface — a product decision, not a global number
• Consistency beats speed, and why AI makes all of this a first-class problem

Scope: this is the product/UX view of latency. The engineering side — how to cut TTFT/TPOT (KV cache, batching, speculative decoding, serving) — was Inference & Latency (L207–L212). Streaming UX in depth (token-by-token rendering, markdown buffering) is the very next lesson (L225) — here we place streaming as one tool in the kit.

Perceived performance isn't vibes — it's grounded in fixed limits of human cognition that have held since Nielsen synthesized the HCI research in 1993. Three numbers are the goalposts for every interface, AI or not:

Add one more, the Doherty Threshold (~400ms): keep the system's response under ~400ms and you don't just feel fast — you keep the user in flow and measurably increase their productivity (the IBM finding that started it). Sub-50ms is the bar for typing/cursor feeling real-time.

These aren't preferences you can train away — they come from working-memory decay and attention span, so they're universal. The whole game of latency UX is: get the user some meaningful response under the threshold that matters for the surface — even if the complete answer takes much longer. Which is only possible because of the next idea.

Here's the lever. There are two latencies, and users only feel one:

• Actual latency — the real time from request to complete response. Engineering owns this (and often can't make it sub-second for an LLM).
• Perceived latency — how long the wait feels. Design owns this — and it can be a fraction of the actual, or a multiple of it.

The gap is huge and it cuts both ways. The same multi-second wait:

• behind a blank screen feels ~35% longer — and worse, users hit the "is it broken?" instinct and refresh or leave;
• behind a skeleton screen feels ~40% shorter (a measured result) and the abandonment instinct disappears;
• behind streaming, the felt wait collapses to TTFT — the moment the first token appears — so a 5-second answer can feel like a 400ms one.

This is the core skill of the section. You usually can't move actual latency much from the product side — but you can routinely move perceived latency by 2–10×. So the question is never just "how do we make it faster?" It's "how do we make the wait feel like it's under the threshold?" The Perceived-Latency Lab below makes this gap visceral; first, the toolkit that creates it.

There's a small, well-worn toolkit for shrinking perceived latency. The art is matching the tool to the wait length (the thresholds from above):

• Instant acknowledgment — always. The millisecond the user acts, show you heard them (the message appears, the button depresses, a cursor blinks). Never let an action vanish into a blank moment.
• Optimistic UI — for actions you're confident will succeed: show the result immediately, before the server confirms, and silently reconcile (revert + explain on the rare failure). Makes a 300ms round-trip feel like 0ms.
• Skeleton screens — for 1–10s content loads: render the shape of the result (greyed boxes matching the final layout). ~40% less perceived wait than a spinner, and it kills the "is it broken?" refresh. (Match the real dimensions; animate subtly; reveal progressively.)
• Streaming — the AI superpower: render tokens as they generate, so the felt wait = TTFT, not E2E. The single highest-leverage latency-UX move for chat (its own lesson, L225).
• Determinate progress + steps — for >10s tasks (agents, batch): show real progress, the current step, and an ETA. A labeled, moving bar turns an unbearable blank wait into a tolerable, legible one — and lets the user step away.

Roughly, by wait length:

The anti-pattern this kills: the blank screen with a centered spinner for a multi-second AI call. It's the default, it's the worst option, and it's why so many AI features feel broken even when they work.

Take one wait and show it five ways. Drag the actual latency and watch the felt wait — and the abandonment risk — change completely, while the real time stays the same:

The takeaway in one screen: the bars are all the same actual latency, yet the blank screen bar is multiples of the streaming bar. You didn't make the model faster — you changed what the user experiences. Notice streaming pins the felt wait to TTFT regardless of how long the full answer takes; that's why it's the default for chat, and why TTFT (L207) is the metric that actually maps to perceived speed.

"Fast enough" isn't one number — it depends entirely on the surface. Different interactions have different perceptual contracts, so you set a latency budget per surface as a deliberate product decision:

Two consequences for how you build:

• The budget picks the tool. A <100ms autocomplete can't stream-and-wait — it needs a tiny/cached model (routing, L216). A 2-minute agent must show progress. The budget is an input to your architecture, not an afterthought.
• TTFT is usually the budget, not E2E. For anything conversational, the number that governs the experience is time-to-first-token, because streaming hides the total. Optimize and SLO TTFT first.

Write the budget down per surface, then design and measure to it. "The chatbot feels slow" is unactionable; "TTFT p95 is 1.2s against a 500ms budget on the chat surface" tells you exactly what to fix.

A counterintuitive but crucial point: a predictable wait beats a sometimes-faster, sometimes-slower one. Users build a mental model of how long your product takes; variance breaks that model and reads as unreliable — even if your average is great. A response that's usually 1s but occasionally 6s feels worse than one that's reliably 2s.

This is why you design and SLO to the tail, not the average:

• Use P95/P99, not the mean. In 2026 production, LLM P95 latency runs ~1.6–3.2× the P50 — so your typical user might be happy while a large minority has a terrible time. The average hides them.
• The outliers ruin perceived performance. One 8-second stall in a session of 1-second replies is what the user remembers. Streaming helps (TTFT is more stable than E2E), but a slow first token is still felt.
• Cap and degrade gracefully. Better a fast, slightly-worse answer (a cheaper model, a cached result, a timeout-and-fallback) than a long unpredictable stall. Tie this to the guardrail/fallback loop (L222).

Reframe your latency goal: not "make the average fast" but "make the p95 stay under the budget, and make the wait predictable." Consistency is a feature.

Classic apps mostly hit Nielsen's limits by being fast — a CRUD request is 50ms and you're done. AI is different in three ways that force latency into the design layer:

• It's often long. A good answer can take seconds; an agent, minutes. You frequently cannot get under 1s of actual latency, so perceived-latency design isn't optional polish — it's the only lever left.
• It's highly variable. The same prompt can take 800ms or 6s depending on output length, load, and routing. High variance is the norm, not the exception — so predictability must be designed in.
• It arrives token-by-token. Unlike a CRUD response that's all-or-nothing, an LLM streams — which is a gift: you can show the first token fast and let the rest flow, collapsing perceived latency to TTFT. The medium itself hands you the best perceived-latency tool.

Put together: AI products are slow, jittery, and streamed — the exact conditions where perceived-latency design matters most and actual-latency optimization hits a floor. That's why "treat latency as a product feature" is the first lesson of AI UX: every other experience decision (streaming, citations, progress, defensive states) is, at bottom, about making the wait feel good. The rest of this section is that toolkit.

Reason through these, then confirm with the Perceived-Latency Lab:

1. Predict: in the lab, set actual latency to 3s. Which treatment makes it feel the longest, which the shortest, and roughly what's the ratio between them?
2. Your chat feature has a 2.5s average response and users say it "feels slow / broken." You can't make the model faster this quarter. Name two changes that fix the feel without touching the model.
3. Why is TTFT the number to put an SLO on for a chat product, rather than end-to-end latency?
4. You're building inline autocomplete and a research agent. Why can't they share one latency budget — and what's the right UX for each?
5. Your p50 latency is a great 900ms, but users still complain. You check and p95 is 5.5s. What's going on, and what should you optimize?

→ (1) Blank screen feels longest (~4s — ~35% over actual + abandonment); streaming feels shortest (~0.45s = TTFT). Roughly a ~9× difference in felt wait for the same 3s. (2) Stream the response (felt wait → TTFT) and/or show a skeleton instead of a spinner (~40% less perceived wait) + acknowledge the send instantly. Pure perceived-latency wins. (3) Because streaming hides the total — once the first token is on screen the user is reading, so the experience is governed by time-to-first-token, not when the last token lands. (4) Different perceptual contracts: autocomplete races keystrokes (<100ms, needs a tiny/cached model, no spinner), while a research agent runs for minutes (acceptable only with progress + steps + the ability to leave). One budget can't serve both. (5) Variance — the p95 (5.5s) is what a large minority feels, and the average hides it; one bad stall is what people remember. Optimize the tail (p95/p99) and make latency predictable, not just the mean.

• "Latency is an engineering metric — minimize the number." It's also a product feature — design the perceived wait. You often can't move actual latency, but you can routinely move felt latency 2–10×.
• "Just add a spinner." A blank-screen spinner is the worst common option for multi-second AI waits. Use a skeleton (~40% less felt wait) or stream; acknowledge instantly.
• "Faster is always the goal." Consistency beats raw speed — a predictable 2s feels better than a jittery "usually 1s, sometimes 6s." Design to p95, not the average.
• "Optimize end-to-end latency." For conversational surfaces, TTFT is the number that maps to perceived speed (streaming hides the total). SLO TTFT first.
• "One latency target for the app." No — set a budget per surface: autocomplete <100ms, chat TTFT <300–500ms, voice <300ms, agents minutes-with-progress.
• "Perceived latency is a hack / fake." It's grounded in fixed human thresholds (0.4s/1s/10s) and measured effects (skeletons cut felt wait ~40%). Managing perception is the job.
• "AI is slow, nothing to do but wait for faster models." AI being slow, variable, and streamed is exactly why perceived-latency design is the biggest lever you have right now.

• Latency is a product feature — design the perceived wait, not just the actual number. The same 3s can feel like 4s (blank screen) or 0.45s (streaming).
• Design against the human thresholds: ~0.4s (Doherty — instant, in-flow), 1s (flow uninterrupted), 10s (attention limit). Get a meaningful response under the line that matters — even if the full answer takes longer.
• The waiting toolkit, matched to the wait: <1s → optimistic UI / instant result; 1–10s → skeleton or stream; >10s → progress + steps + notify; always acknowledge the action in <100ms. Kill the blank-screen spinner.
• Set a latency budget per surface (a product decision): autocomplete <100ms, chat TTFT <300–500ms, voice <300ms, agents minutes with progress. The budget picks the architecture; TTFT is usually the real budget, not E2E.
• Consistency beats raw speed — variance reads as unreliable. Design and SLO to p95/p99 (which runs ~1.6–3.2× p50), not the average; cap and degrade gracefully.
• AI forces this: responses are slow, variable, and streamed, so actual-latency optimization hits a floor and perceived-latency design becomes your biggest lever.
• Next — L225: Streaming UX — the highest-leverage tool in the kit, in depth: TTFT, token-by-token rendering, and the markdown-buffering gotchas that make streaming actually feel good.