Token Economics: Where the Money Goes

We just finished making inference fast (L207-L212). Now we make it cheap — and this is where AI engineering meets the CFO. This lesson opens the Cost Optimization section by answering the question every team eventually asks in a panic: "why is our LLM bill so high, and where is the money actually going?" Get the economics right and the next four lessons (the levers) make sense; get it wrong and you'll optimize the thing that doesn't matter.

The unit of cost is the token (recall L36's token math, and L207's token metrics). But three facts surprise almost everyone: output tokens cost several times more than input, you re-pay for the entire prompt on every single call, and agents multiply that into bills that become a top line item. Understanding those three is 90% of cost control.

In this lesson:

• The cost formula — what you actually pay for, with current (2026) pricing
• Surprise #1: output costs ~3-5× input — and why that flips your optimization priorities
• Surprise #2: you pay the whole prompt every call — system prompt, context, and growing history
• The cost bomb: agent loops — why agents burn ~50× a chat, and how a weekend run hit $4,200
• The levers — a map of how the rest of this section cuts the bill

Scope: this is the economics — where the money goes and how to measure it. The levers to cut it each get their own lesson: prompt caching (L214), semantic caching (L215), model routing & cascades (L216), and batch APIs & distillation (L217). We'll preview them here; we go deep there.

Every hosted LLM bills by the token, and crucially in two directions: the tokens you send (input) and the tokens it generates (output), each at its own price. The cost of a single call is just:

cost = (input_tokens × input_price) + (output_tokens × output_price)

Prices are quoted per million tokens. Current flagship-to-cheap pricing (2026):

Two things jump out before we even do any math. First, the model tier swings cost ~5× — Haiku vs Opus is the same call for one-fifth the price (the whole point of routing, L216). Second, output is consistently ~5-6× the input price in every row. That's not a rounding detail; it's the single most important fact in LLM cost — so it gets its own section.

Look down the last column: output tokens cost about 5× input tokens, everywhere (industry-wide it ranges 3-10×). Why? It's the exact lesson from the inference section. Input is processed in one parallel prefill pass (compute-bound, efficient — L207/L208), so it's cheap per token. Output is generated one token at a time in the decode loop (memory-bandwidth-bound, the GPU mostly waiting — L208), so it's expensive per token. The price tag literally reflects the compute physics we studied.

This flips your optimization instincts. The shape of your workload decides where the money is:

• Short prompt → long answer = OUTPUT-dominated. A 200-token prompt that yields a 2,000-token essay: ~98% of the cost is output. Here, trimming the answer (ask for a summary, cap max_tokens, stop over-explaining) is your biggest lever.
• Big context → short answer = INPUT-dominated. A 20,000-token RAG prompt that yields a 200-token answer: ~95% of the cost is input. Here, trimming the context (retrieve less, cache the prefix) is the lever.

The instinct "my prompt is short, so this is cheap" is often backwards — a short prompt that triggers a long generation is output-heavy and pricey. Always look at the input/output split, not just the total. (The interactive below makes this split vivid.)

Here's the one that quietly wrecks budgets: the API is stateless. The model remembers nothing between calls, so you must re-send the entire input on every single request — and you're billed for all of it, every time. That "input" is rarely just the user's question. It's:

• the system prompt (instructions, few-shot examples, tool definitions) — sent every call,
• the retrieved context (RAG documents) — sent every call,
• and in a conversation, the entire history — which grows with every turn.

That last point compounds. In a chat, turn 1 re-sends 1 message; turn 10 re-sends all 10; turn 50 re-sends all 50. The per-turn input cost grows linearly with the conversation length, so the total cost of a long chat grows roughly quadratically. A 4,000-token system prompt sent on every one of a million daily calls is 4 billion input tokens/day of pure overhead — before the user types a word.

This is why prompt caching (L214) exists and is the highest-ROI cost lever: if the same large prefix (system prompt + context) is re-sent every call, you can cache it and pay ~0.1× for the cached portion instead of full price. But you can't fix what you don't see — first, notice that you're re-paying for the prompt every call.

Now combine the two surprises with agents (the whole of Container 3) and you get the real story of LLM bills in 2026. A chatbot sends one message and gets one response. An agent runs a loop — think, call a tool, read the result, think again, call another tool, re-check — and every step re-sends the entire accumulated context (the system prompt + every tool call and result so far). So:

• The context grows on every step, and you re-pay for all of it each step. By step 20, a single call's input can exceed 50,000 tokens. At Sonnet's $3/M, that's ~**$0.15 for one late-loop step** — and a task is many steps.
• The result: agents burn roughly 50× the tokens of a chat for the same user-visible work. A single 20-step task can cost $1-2 by itself.
• At scale this is brutal: teams report their AI bill becoming the second-largest line item after salaries within 90 days of enabling coding agents — and one developer racking up $4,200 in API fees over a single weekend during an autonomous refactoring run.

The cost drivers of an agent aren't mysterious: (number of steps) × (context length per step). Both grow together as the loop runs. That's why agent cost control is about fewer steps and smaller context per step — context management / compaction (recall L208's memory framing), prompt caching the stable prefix (L214), and routing cheap sub-steps to small models (L216). An agent that's correct but unbounded in steps is a financial liability.

Stop guessing and compute it. Pick a model and drag the sliders — watch the per-request cost, the monthly bill, and (most importantly) the input-vs-output split:

Three experiments to run:

• Push output up (long answer, short prompt): the split bar goes green (output) — you're output-dominated, so cap the answer length.
• Push input up (big RAG context, short answer): the bar goes violet (input) — you're input-dominated, so trim/cache the context.
• Switch Opus → Haiku at the same workload: the monthly bill drops ~5× for the same call — the single biggest lever, if a cheaper model clears your quality bar (that's routing, L216).

Now that you can see where the money goes, here's the toolkit — each gets its own lesson next:

Plus the two "free" wins you can apply today, straight from this lesson: trim the output (cap max_tokens, ask for concise answers) when you're output-dominated, and trim the context / fewer agent steps when you're input-dominated. Stacked, these levers routinely cut a bill by well over half — but you apply them in order of where your money actually is, which is why this lesson came first. Measure, then optimize the dominant term.

Reason through these, then check with the calculator (and the runnable snippet above):

1. Predict: a request with a 200-token prompt and a 2,000-token answer on Sonnet — is it input- or output-dominated, and roughly what share is output?
2. Your RAG app sends a 15,000-token retrieved context for a 150-token answer, on every call. Where's the money, and which two levers help most?
3. Your chatbot's per-message cost keeps rising as a conversation goes on, even though messages are short. Why?
4. A coding agent runs 30-step tasks and your bill exploded. Name the two cost drivers of an agent loop and one fix for each.
5. You're choosing between Opus 4.8 and Haiku 4.5 for a high-volume classification task. What's the cost difference, and what's the one question that decides it?

→ (1) Output-dominated — at $3/$15, output is 2000/1e6×15 = $0.030 vs input 200/1e6×3 =$0.0006, so ~98% output. Cap max_tokens / ask for concision. (2) Input-dominated (~95% input: 15000×$3 ≫ 150×$15). Best levers: prompt caching the context if it repeats (L214), and retrieving less / smaller chunks (RAG). (3) The API is stateless, so each turn re-sends the whole growing history as input — per-turn input cost rises with conversation length (total ≈ quadratic). Fix with caching (L214) + history trimming/compaction. (4) Number of steps × context length per step (the context is re-sent and grows each step). Fixes: bound the steps (recovery/step budget, L131) and shrink per-step context (compaction + prompt caching). (5) Haiku is ~5× cheaper ($1/$5 vs $5/$25). The deciding question: does Haiku clear your quality bar on this task? If yes, route to it (L216); if no, the cheap price is irrelevant.

• "Tokens are tokens — input and output cost the same." No — output costs ~3-5× input (the decode loop is memory-bound and expensive). Always optimize the dominant direction.
• "My prompt is short, so the call is cheap." Not if it triggers a long answer — that's output-dominated and pricey. Look at the split, not the prompt length.
• "The model remembers the conversation, so I only pay for new messages." The API is stateless — you re-send and re-pay for the entire history every turn. Long chats cost ~quadratically.
• "An agent is just a few LLM calls." An agent is a loop that re-sends a growing context each step — ~50× a chat. Steps × context-per-step is the bill.
• "Cost ≈ number of requests." Cost ≈ tokens, weighted by the output premium, the re-sent prompt, and the agent multiplier. A million cheap classifications can cost less than a thousand long agent tasks.
• "Optimize cost by switching to the cheapest model." Only if it clears your quality bar — and routing per query (cheap for easy, expensive for hard, L216) usually beats one blanket choice.
• "We'll deal with cost later." Agent bills become a top line item in ~90 days. Measure cost/request and cost/task before you ship, and set budgets/limits.

• You pay per token, in two directions: cost = in_tok × in_price + out_tok × out_price, quoted per 1M tokens. The model tier swings cost ~5×.
• Output costs ~3-5× input (decode is memory-bound, L208) — so short-prompt/long-answer = output-dominated (trim the answer), big-context/short-answer = input-dominated (trim/cache the context). Optimize the split, not the total.
• You re-pay the whole prompt every call (stateless API): system prompt + context + growing history. Long chats cost ~quadratically.
• Agent loops are the cost bomb: context is re-sent and grows every step, so agents burn ~50× a chat ($4,200-in-a-weekend territory). Drivers = steps × context-per-step; control both.
• Measure first. Compute cost/request and cost/task (and the input/output split) before shipping; set budgets. You can't cut what you haven't located.
• The levers (next): prompt caching (L214, ~90% on the prefix), semantic caching (L215), routing & cascades (L216, ~5×), batch + distillation (L217, 50%+). Apply them where your money actually is.
• Next — L214: Prompt Caching — the highest-ROI lever, attacking Surprise #2 head-on: stop re-paying full price for the same prompt prefix on every call.