A Repeatable Model-Selection Workflow

You now know the pieces: model capabilities, long-context limits, multimodality, the Iron Triangle, and how to read benchmarks without being fooled. This lesson assembles them into a repeatable process — so 'which model should I use?' stops being a vibe and becomes a quick, evidence-based decision you can run for every project.

The whole thing is a funnel: start with the dozens of available models, apply a few filters to shrink the field fast, then let a small evaluation on your own data pick the winner. Five steps, reusable forever.

You'll learn:

• The 5-step funnel from 'all models' to 'your choice'
• How to filter on non-negotiables and classify the task tier
• How to shortlist with benchmarks (skeptically) and decide with your own eval
• Why selection is continuous, and how to keep a simple scorecard

Five steps, each narrowing the field:

1. Filter by non-negotiables (privacy, latency cap, budget, modality).
2. Classify the task by difficulty tier.
3. Shortlist 3–5 candidates from the right benchmarks (read skeptically).
4. Evaluate the shortlist on your own data — this is what actually decides.
5. Pilot, monitor, and re-evaluate — because models and prices keep changing.

The golden rule running through all of it: benchmarks shortlist; your eval decides. Here's the funnel:

Before comparing quality, eliminate anything that's simply disqualified. These are hard constraints, not preferences — and they shrink the field dramatically in one pass:

• Data privacy / residency: must the data stay on-prem or in-region? If so, you're limited to self-hostable/open models (Ollama, vLLM) or providers with the right guarantees.
• Latency ceiling: a real-time voice agent rules out heavy reasoning models (the TTFT landmine).
• Budget: a high-volume, cost-sensitive feature rules out the priciest frontier models.
• Modality: need audio/video? That alone narrows you to specific models (Gemini-class).
• Context size & compliance: required window, certifications (SOC2/HIPAA), region.

Write these down first. It's pointless to evaluate a model you can never ship.

Next, ask how hard the task actually is — because that decides which class of model (and which benchmarks) you should even consider. Recall the three flavors lesson:

• Simple / structured (classification, extraction, routing, short rewrites) → a small, cheap, fast model is usually enough.
• Moderate (general chat, summarization, standard coding) → a solid mid-tier model (Sonnet-class).
• Hard / multi-step (complex reasoning, tough math, agentic coding, deep analysis) → a frontier or reasoning model.

Most teams overshoot here — reaching for the biggest model out of caution and overpaying for capability the task doesn't need. Match the tier honestly; you can always escalate the genuinely hard cases (that's the router, next lesson).

Now use public benchmarks for what they're good at: narrowing the field, not making the final call (last lesson's whole point).

• Pick the benchmarks that match your task tier — e.g. SWE-bench for coding, GPQA for hard reasoning, an arena for general chat. Ignore saturated ones (MMLU) when comparing top models.
• Triangulate across a static eval + a human-preference arena + an agentic suite; favor fresh, contamination-resistant benchmarks.
• Narrow to 3–5 candidates (5–8 max) — more than that just creates evaluation work without payoff.

The output of this step is a small, defensible shortlist — not a decision. The decision comes next.

This is where the winner is actually chosen. Public scores can't tell you which model is best at your task — only your own evaluation can.

1. Build a small eval set: 100–200 real examples drawn from your actual workload — including edge cases and known failure modes, and cases where you know the correct answer. (For high-stakes apps, 500+.)
2. Run every shortlisted model on it.
3. Score what you care about: correctness/quality, plus cost and latency (the Iron Triangle) — and format, tone, or whatever defines 'good' for you.
4. Scoring method: exact-match/rules for objective tasks; LLM-as-judge (a strong model grading outputs) for subjective ones — first calibrate the judge against ~50–100 human ratings so you trust it.

This modest investment — an afternoon of work — routinely overturns the leaderboard pick. A model ranked #3 publicly may be #1 on your data. (We build evaluation properly in its own container; this lightweight version is enough to choose.)

A model that wins your offline eval still has to survive production. So:

• Pilot the top 1–2 on real traffic (a small % of users, or shadow mode), watching real quality, cost, and latency — not just your test set.
• Instrument it: log tokens/cost, latency (TTFT/TPS), and failures so you can see how it behaves at scale.
• Re-evaluate periodically. This is the part everyone forgets: model selection is not one-and-done. New models and price cuts land almost monthly — a choice that was optimal in March may be beaten (or undercut) by June. Re-run your eval when something notable ships.

And rather than betting everything on one model, you can route between them — fast/cheap by default, escalate the hard cases — which is exactly the next lesson.

Make the decision legible with a simple scorecard — candidates as rows, what matters as columns:

Then pick by your weighted priorities from the Iron Triangle: latency-first for real-time, cost-first for batch, quality-first for high-stakes. The scorecard turns a fuzzy debate into an obvious choice — and documents why you chose, so the decision is revisitable when the landscape shifts.

Run the funnel on a real task. Pick something you'd actually build (e.g. 'summarize legal contracts'). Walk the 5 steps in your head: ① non-negotiables (privacy? latency?), ② task tier (hard → frontier), ③ shortlist 3 from the right benchmarks, ④ the decisive step — what 20 real examples would you test on? ⑤ how you'd pilot. The output isn't 'the top-ranked model' — it's your top-2, chosen on your data.

• "Just use the top-ranked model." That's the shortlist, not the decision — your own eval decides, and the best model is often overkill (cost/latency).
• "Pick once and move on." No — selection is continuous; models and prices change monthly. Re-evaluate.
• "I need a huge eval set first." No — 100–200 representative examples are enough to choose; deep eval comes later.
• "Bigger model = safer choice." Usually overshoot — match the task tier; you can route hard cases to a bigger model.
• "Offline eval is enough." Validate in a pilot on real traffic — production surfaces issues your test set won't.

• Model selection is a repeatable 5-step funnel: filter non-negotiables → classify the task tier → shortlist from benchmarks (skeptically) → evaluate on your data → pilot & re-evaluate.
• Filter first (privacy, latency, budget, modality) — never evaluate a model you can't ship.
• Benchmarks shortlist; your own eval (100–200 real examples) decides — use LLM-as-judge for subjective scoring, calibrated against humans.
• Score candidates on a scorecard of quality + cost + latency, and pick by your Iron Triangle priority.
• Re-evaluate over time — the best choice is a moving target as models and prices change.

Next: the final piece of choosing — build vs buy, and the model router that lets you stop betting on a single model and instead send each request to the model that fits it best.