Latency vs Cost vs Quality — The Iron Triangle

Every model decision you'll ever make is a tug-of-war between three things: how good the output is (quality), how fast it arrives (latency), and how much it costs. These form an Iron Triangle — pull toward one corner and you usually move away from another. There is no model that is simultaneously the smartest, the fastest, and the cheapest.

As one apt summary puts it: "speed costs money, intelligence costs latency, and verbosity costs both." The engineers who ship great products aren't the ones who find a magic model with no trade-offs — they're the ones who know, for their use case, which corner they can compromise on.

You'll learn:

• The three corners — quality, latency, cost — and how they pull against each other
• Latency precisely: TTFT and TPS (and the reasoning-model landmine)
• The levers to optimize each corner
• Why streaming is a near-free latency win
• A practical way to choose by use case

Name the three forces clearly:

• Quality — how capable/accurate the model is on your task. Bigger and reasoning models are generally smarter.
• Cost — dollars per token (input + output), from the pricing lesson. Smaller models are far cheaper.
• Latency — how fast the response is. This one has two parts, so let's be precise about it.

The whole lesson lives in the picture below: a triangle where every model and setting sits somewhere inside, and moving toward one corner trades off the others.

"Latency" is really two numbers, and confusing them leads to bad decisions:

• TTFT — Time To First Token. How long until the first token appears. This drives perceived responsiveness — the wait before anything happens.
• TPS — Tokens Per Second. How fast tokens stream after the first. This drives total duration for long outputs.

A rough model of total time:

total_time  ≈  TTFT  +  (output_tokens ÷ TPS)

Real numbers (2026): Claude Opus ~78 TPS / ~0.85s TTFT; GPT-5.5 ~92 TPS / ~1.1s TTFT; small/fast models like Haiku or Gemini Flash get TTFT under 600ms; specialized inference hardware (Groq, Cerebras) pushes 400–500+ TPS. Bigger, smarter models tend to have lower TPS and higher TTFT — the quality↔latency tension, made of numbers.

One trap deserves its own warning. Reasoning models (from the reasoning lesson) generate a long internal chain-of-thought before the visible answer — and from the user's perspective, all that thinking shows up as TTFT.

The effect is dramatic: enabling extended reasoning can inflate TTFT 5–30×, with frontier reasoning modes hitting tens of seconds (in the worst cases, over a minute) before the first visible word. That's fine for an offline 'solve this hard problem' job — and unacceptable for a real-time chat or voice assistant.

So reasoning trades latency (and cost — those thinking tokens bill as output) for quality on hard problems. Reach for it deliberately, and never put a heavy reasoning model on a latency-critical path.

You're not stuck where a model lands by default — each corner has levers:

Notice how many levers help two corners at once: a smaller model is faster and cheaper; shorter output cuts latency and cost. The painful trades are the ones that touch quality — that's the corner you protect or sacrifice most deliberately.

One technique improves the experience for free: streaming (from the chatbot lesson). It doesn't make generation any faster — total time is unchanged — but it shows tokens as they arrive, so the user sees progress in ~1 second instead of staring at a spinner for ten.

This matters because humans judge speed by TTFT-like perceived responsiveness, not total duration. A streamed 10-second answer feels far snappier than a non-streamed 4-second one. Always stream anything user-facing — it's the cheapest latency improvement you'll ever make.

Spec-sheet numbers are best-case; real systems are messier:

• Latency degrades under load. With concurrent users, queuing, and cold starts, real-world throughput can drop 30–60% versus a quiet benchmark. Test at your expected load.
• TTFT grows with input size. A 100-token prompt might start in ~200ms; a 50,000-token prompt can take 1–2 seconds just to begin — so long context costs you on latency too (yet another reason to curate, from the long-context lesson).
• Verbosity taxes both corners. Long outputs cost more and take longer (more tokens to generate). Prompting for concise answers is a latency and cost win.
• Measure on your own task. Published TTFT/TPS rarely match your prompt sizes, region, and load. Instrument real requests.

The decision becomes easy once you ask: which corner can't I compromise?

A powerful pattern is to mix: route easy requests to a fast/cheap model and escalate only the hard ones to a slow/smart model — getting most of the quality at much of the speed and savings (we'll build exactly this router later in the course).

You can't sit in all three corners at once — so choose. Drag the dot below toward what your use case can't compromise and watch the priority weights shift and a model recommendation appear:

• Drag to Quality → it suggests a frontier/reasoning model (accept the cost & latency).
• Drag to Latency → a small, fast model + streaming.
• Drag to Cost → a cheap model + Batch API.
• Sit in the middle → a balanced mid-tier model.

That single decision — which corner you give up — is the whole lesson.

• "There's a best model." No — there's a best model for a given priority. The Iron Triangle says you choose, not win all three.
• "Latency is one number." No — TTFT (responsiveness) and TPS (throughput) are different; optimize the one your UX needs.
• "Streaming makes it faster." It makes it feel faster — total time is the same; it's perceived latency you're improving.
• "Reasoning models are just better." They're better on hard tasks but a latency/cost landmine (TTFT 5–30×) — wrong for real-time.
• "Benchmarked speed = my speed." No — load, prompt size, and region change everything; measure your own.

• Model choice is an Iron Triangle of quality, latency, cost — you optimize for a priority, you don't max all three.
• Latency = TTFT (responsiveness) + TPS (throughput); total ≈ TTFT + output/TPS. Bigger models are slower; reasoning inflates TTFT 5–30×.
• Levers: quality (bigger/reasoning, RAG, fine-tune) · latency (smaller model, stream, shorter output, trim prompt) · cost (smaller model, caching, batch). Many levers help two corners; quality is the deliberate trade.
• Stream everything user-facing — a free perceived-latency win.
• Choose by use case: real-time → latency; batch → cost; high-stakes → quality; everyday → a balanced mid-tier — and route to get the best of both.

Next: quality is the slipperiest corner to measure — so we learn to read benchmarks critically (MMLU, GPQA, SWE-bench) and not be fooled by a leaderboard.