Batch APIs & Distillation for Cost

You've now got four cost levers: you can see where the money goes (token economics, L213), reuse the prefix (prompt caching, L214), reuse the answer (semantic caching, L215), and pick a cheaper model per query (routing, L216). All four are runtime levers — they change how you call the model, request by request. This lesson adds the last two, and they're different in kind — they're structural:

• Batch APIs — submit work asynchronously and get it back within 24h (usually <1h) at exactly 50% off. Same model, same prompt — the only price is that you wait. It's the single biggest discount any major provider offers, and it's almost free to adopt.
• Distillation — use a big teacher model to teach a small student model your task, then run the cheap student in production. You trade an upfront cost for the lowest per-call price — and you end up owning a model tuned to your workload.

Why "structural." Caching and routing optimize each call as it happens. Batch changes when the work runs; distillation changes which model you own. They sit underneath the runtime levers — and crucially, all six stack.

Because this is the last lesson in Cost Optimization, we'll finish by assembling the whole toolkit into one cost-optimization playbook — the order to apply the levers, and why they multiply rather than add.

In this lesson:

• Batch APIs — the 50%-off deal, the mechanism (submit → poll → retrieve), real provider limits, and when not to use it
• Distillation — first principles (Hinton's soft labels), how it works with today's APIs, the upfront-vs-volume economics, and the failure modes (incl. the Terms-of-Service trap)
• The cost-optimization stack — all six levers, runtime vs structural, layered

Scope: this closes the Cost section. Latency optimizations (batching on the GPU, i.e. continuous batching) were a different thing back in L209 — don't confuse it with batch APIs here (we'll flag the difference). Fine-tuning in depth and full LLMOps/deployment come in later sections; here we use distillation purely as a cost lever.

Here's a deal that sounds too good: take the exact same model and the exact same prompts, submit them a different way, and pay half price on both input and output tokens. No quality change, no prompt change. The only catch: you get the results later (within 24 hours, usually under one).

That's the Batch API, and every major provider offers it at the same ~50% discount:

Why can they halve the price? Because async work is easy to schedule. Real-time requests force the provider to keep GPUs on standby for your spike; batch requests can be slotted into spare capacity whenever it frees up (overnight, between traffic peaks). You're selling them scheduling flexibility and they're paying you 50% for it. It's the same trade as off-peak electricity.

Don't confuse this with L209. Continuous batching in L209 was the serving engine packing many users' tokens into one GPU step to raise throughput — invisible to you, happening in real time. A Batch API is a product: you explicitly hand over a pile of non-urgent requests and accept delayed delivery for a discount. Same word, different layer.

The shape is the same everywhere: submit a batch → poll until it's done → stream the results. Each request carries a custom_id so you can match results back to inputs (they don't come back in order). Here's the full loop on Anthropic's Message Batches API:

Three things that matter in production:

• Failures are per-request, not per-batch. One bad request doesn't sink the others; you get succeeded / errored / expired per custom_id. expired (didn't finish in 24h) is not billed — just requeue it.
• It's not a conversation. Each request is independent — you can't do multi-turn within a batch. (You can still batch the first turn of 50,000 conversations.)
• No streaming, and the job is immutable once submitted. Batch is for fire-and-forget volume.

It stacks with prompt caching (L214). A cache hit is billed at 10% of base input, and that figure is then halved by the batch discount — providers quote up to ~75% off when you combine them. Two levers, one request.

The decision rule is one question: does a human need this answer right now? If no → batch it and pocket the 50%.

Batch shines for offline, high-volume work:

• Evals & benchmarks — running thousands of test cases (you'll do a lot of this in the Guardrails section).
• Classification & data enrichment — labeling, tagging, sentiment, extraction over a whole corpus.
• Synthetic-data generation — including generating the training set for distillation (next!).
• Embeddings backfills — re-embedding a document store after a model change.
• Nightly/periodic jobs — reports, summaries, digests, content moderation queues.

Don't batch when latency is the product:

• Anything interactive — chat, copilots, agents in the loop (an agent waiting 20 minutes per step is useless).
• Tight SLAs — if a user or downstream system is blocked on the response.
• Tiny volumes — the submit/poll/retrieve overhead isn't worth it for a handful of calls.

The payoff is bigger than it looks. Reporting teams find that when half their workload is batchable, the Batch API alone cuts 35–48% off the total monthly bill — for essentially an afternoon of plumbing. A useful habit: default new offline pipelines to batch, and only promote a workload to the real-time API when you've confirmed it actually needs to be.

Routing (L216) picked the cheapest model that already exists for each query. Distillation goes further: it creates a new cheap model that's good at your task — by having a powerful teacher model train a small student model. You then run the student in production at a fraction of the cost, and you own it.

First principles (Hinton, Vinyals & Dean, 2015 — the paper that named it). The insight: a trained teacher's output is richer than the label alone. Ask a teacher "is this a cat?" and it says 97% cat, 2% dog, 1% fox — those small numbers (Hinton called them the "dark knowledge") encode how the teacher sees the world (cats look more like dogs than foxes). So you train the student to match the teacher's full probability distribution (the "soft labels"), not just the hard answer — a much richer signal per example. A temperature knob softens the teacher's over-confident output so those subtle relationships are visible (Hinton used T from 1–20). The student's loss combines matching the teacher with getting the ground-truth right.

The classic proof it works: DistilBERT (2019). Distilling BERT into a 6-layer student gave a model 40% smaller and 60% faster that kept 97% of BERT's language understanding. That ratio — most of the quality at a fraction of the size/cost — is the whole pitch of distillation.

Distillation vs the other levers. Caching reuses a past answer; routing picks an existing model; distillation bakes the capability into the weights of a model you own. It's the only lever that produces a new artifact — and the only one with a real upfront cost.

Hinton's method needs the teacher's logits (the raw probabilities). With a closed API you can't see those — so the LLM-era version is response (or "data") distillation: just collect the teacher's outputs and fine-tune the student on (input → teacher output) pairs. Less information per example than true soft-label distillation, but it works remarkably well and needs only API access. The pipeline:

Providers automate every step:

• OpenAI Model Distillation — add store: true to your normal frontier calls to capture them (tag with metadata), filter the good ones in the dashboard, hit "Distill", and it fine-tunes a small model (e.g. a gpt-mini). A few hundred samples can be enough; thousands of diverse ones do better.
• Amazon Bedrock Model Distillation — pick a teacher (e.g. Claude Sonnet) and a student (Claude Haiku or Nova); Bedrock generates the synthetic data and runs the fine-tune for you. Reported results: a distilled Nova Micro ran >95% cheaper on tokens, and distillation boosted function-calling accuracy while cutting cost and latency.

The economics — why distillation is a volume play. Unlike the runtime levers (≈ free to switch on), distillation has a real upfront cost: teacher generation (pay frontier prices to label the training set — batch it!) + the fine-tuning job. That cost is fixed; the savings are per call. So:

$\text{break-even volume} \approx \frac{\text{upfront cost}}{\text{per-call saving}}$

Below that volume you never recoup the investment; far above it, distillation is the cheapest possible way to serve the task. It's a bet that pays off only at scale.

Distillation is powerful but narrow — it's the right tool only when several things line up:

Use it when: the task is well-defined and stable (classification, extraction, a specific style/format, a fixed tool-calling pattern), the volume is high (so you clear break-even), and a frontier model already does it well (the student can only mimic a teacher that's right). On-device or self-hosting needs push you here too.

Avoid it when:

• The task is broad or open-ended — a small student can't absorb general reasoning; it'll mimic the surface and miss the edge cases. (Reach for routing instead.)
• Volume is low — you'll never earn back the upfront cost.
• The task changes often — every change means re-distilling.
• Cheaper levers already suffice — try caching/routing/batch first; distillation is the heaviest lift.

The failure modes (all require an eval to catch):

• Teacher errors propagate. The student inherits the teacher's mistakes, biases, and hallucinations — garbage in, garbage out, baked into weights.
• Silent degradation & distribution shift. The student is only as good as your training set's coverage; it quietly fails on inputs you under-sampled, and as real traffic drifts from the training distribution, quality decays. Keep evaluating in production.
• Homogenization. If everyone distills from the same few teachers, all the students share the same blind spots — a fragility worth knowing about at the ecosystem level.

⚠️ The Terms-of-Service trap (do not skip this). Most frontier providers' terms forbid using their model's outputs to train a competing model. Distilling within a provider's own offering is explicitly supported (OpenAI→OpenAI, Bedrock Claude→Claude/Nova). Using one vendor's outputs to build a rival model can violate their ToS — a legal/compliance risk, not just an engineering one. Check the terms before you distill across vendors.

Here's the whole section in one widget. Start with a workload at full frontier price, then layer the levers and watch the bill fall — the three runtime levers (caching, routing) are on; add this lesson's two structural ones (batch, distill):

The point the waterfall makes: cost levers multiply, they don't add. Each lever cuts a percentage of what's left after the previous one — so three modest levers (−20%, −40%, −35%) already compound past −68%, and adding batch and distillation pushes the stack to ~−88%. Notice the character of each: caching and routing are free runtime wins (flip them on, no downside); batch costs you latency; distillation costs you an upfront investment. That's exactly the order you should adopt them in — which is the playbook next.

Six lessons, one coherent stack. Apply the levers cheapest-effort first — bank the free wins before the heavy lifts:

The mental model: runtime levers change how you call the model (same model, smarter calls — caching, routing); structural levers change when you run it and which model you own (batch, distillation). Runtime levers are nearly free and stack instantly — do them first. Structural levers ask for something in return (latency, or an upfront investment) — reach for them once the easy wins are banked and the volume justifies it.

And above all (L213's rule): measure first. Every lever here trades something — latency, complexity, an eval burden, an upfront cost. Instrument your spend, find the workload that actually dominates the bill, and apply the lever that targets it. Optimizing a workload that's 3% of your cost is wasted effort. The cheapest token is the one you understood before you spent it.

Reason through these, then confirm with the stack widget:

1. Predict: in the widget, turn on the three runtime levers, note the %, then add batch and distill. Why does the total saving end up near −88% and not the simple sum of the individual percentages?
2. You have a live customer-support chat and a nightly job that summarizes the day's 80,000 tickets. Which one goes on the Batch API, and why not the other?
3. A startup doing 800 classification calls/day wants to distill a frontier model into a small one to save money. Good idea? What's the number you'd check first?
4. Your distilled student model scored great at launch; three months later users complain about wrong answers on a new product line. What happened, and what process should have caught it?
5. You want to distill GPT-class outputs into an open-weights Llama model you'll self-host and sell. What do you check before writing any code?

→ (1) Because the levers multiply, not add — each one cuts a fraction of what's left after the previous (0.80 × 0.60 × 0.65 × 0.70 × 0.55 ≈ 0.12, i.e. −88%). Stacking is compounding. (2) The nightly summary job → Batch (it's offline, high-volume, nobody's waiting → 50% off). The chat must stay real-time — a 24-hour window would destroy it. (3) Probably not yet — check the break-even volume (upfront cost ÷ per-call saving). At ~800 calls/day the fixed cost of teacher-generation + fine-tuning likely never gets repaid; use routing instead until volume grows. (4) Distribution shift — the new product line wasn't in the training set, so the student silently degrades on it. Continuous evaluation in production (sampling + checking quality per segment) should have flagged it; then re-distill with fresh data. (5) The Terms of Service — using one vendor's outputs to train a competing model typically violates their terms. It's a legal/compliance question to resolve before the engineering.

• "The Batch API uses a worse/cheaper model." No — it's the same model and prompt, just processed asynchronously. You pay 50% less purely for accepting a delay.
• "Batch API = the continuous batching from L209." Different layers. L209 was the serving engine packing tokens onto a GPU in real time (a throughput trick). Batch API is a product for handing over non-urgent work for a discount.
• "Distillation is just fine-tuning." It is fine-tuning — but the training data comes from a teacher model (synthetic), not human labels. The teacher is the source of the knowledge.
• "Distillation always saves money." Only above break-even. It has a real upfront cost (teacher generation + training); at low volume you never recoup it.
• "A distilled student is as smart as the teacher." Only on the narrow task it was trained for — and only as good as the teacher and your data coverage. It inherits the teacher's errors and degrades silently off-distribution.
• "I can distill any model's outputs into my own." Often a ToS violation — providers forbid training a competing model on their outputs. Distill within a provider's own offering, or check the terms.
• "Pick the one best cost lever." The levers stack — caching and routing and batch and distillation, layered. A mature system runs several at once.
• "Optimize cost everywhere." Measure first (L213). Optimize the workload that dominates the bill; cutting 50% off a 3%-of-spend job is busywork.

• Batch APIs trade latency for 50% off — same model, same prompt, processed async (≤24h, usually <1h) at half price on input and output. Submit (≤100k req / 256MB) → poll processing_status → stream results by custom_id. Stacks with prompt caching for up to ~75% off.
• Batch the offline work — evals, classification/enrichment, synthetic data, embeddings backfills, nightly jobs. Never batch real-time/interactive/agent-loop traffic. ~35–48% off the monthly bill when half the workload qualifies.
• Distillation makes a cheap model that mimics the frontier — a teacher generates outputs on your task, you fine-tune a small student on them (response distillation, since closed APIs hide logits). Hinton's soft labels + temperature; DistilBERT = 40% smaller, 60% faster, 97% quality. Student runs ~10–30× cheaper.
• Distillation is a volume bet — real upfront cost (teacher generation + fine-tune) repaid only above break-even. Use it for high-volume, narrow, stable tasks; OpenAI & Bedrock automate it (Claude Sonnet→Haiku).
• Mind the failure modes — teacher errors propagate, the student degrades silently off-distribution (keep an eval!), and ToS forbid distilling a competing model from a vendor's outputs.
• Cost levers MULTIPLY, not add — measure (L213) → prompt cache (L214) → semantic cache (L215) → route (L216) → batch + distill (L217). Runtime levers change how you call; structural levers change when you run & which model you own. Free wins first, structural finishers last.
• Next — a new section: AI Application Architecture. You've made the pieces fast (latency) and cheap (cost); now we assemble them into a reference architecture for a real LLM application — the full diagram, end to end.