Skip to main content
CodePeetCodePeet
LLM Evaluation & Observability

Statistical Rigor: Sample Size & Confidence

Introduction

You've built a toolbox of metrics and learned to match each to its task. This final lesson of the section adds the layer that makes those numbers trustworthy — because here's the uncomfortable truth: a metric is a number computed from a sample, and samples are noisy.

Your eval said the new prompt scores 86% and the old one 84%. Did you improve — or did you just watch a coin land heads twice? This is the #1 way teams fool themselves after they have evals: reading noise as signal and shipping a “win” that was random. This lesson is the statistical rigor that separates a real improvement from a lucky bounce — confidence intervals, sample size, and significance — the difference between “I think it's better” and “it's better, p < 0.05.”

An infographic titled 'Statistical Rigor: Sample Size & Confidence' explaining how to tell a real improvement from noise when evaluating LLMs. The central idea is that an eval score is an estimate from a finite sample, not the truth, so a pass rate of eighty-five percent on two hundred examples comes with a margin of error, and the same prompt re-run on a different day moves by a few points due to non-determinism; with only a hundred examples, four flipping is inside the noise floor, so a small move is a coin flip, not progress. Every score should be reported as a confidence interval, an error bar around the number, computed for a proportion as the score plus or minus one point nine six times the square root of p times one minus p divided by N; because the margin shrinks with the square root of N, quadrupling the examples halves the error bar, and roughly two hundred fifty to five hundred samples are needed to stabilize a pass rate. To decide whether a new version B is really better than A, the heuristic is whether their ninety-five percent confidence intervals overlap: non-overlapping intervals are strong evidence of a real difference, while heavy overlap means the gap could be random variation. The rigorous method is a paired test on the same eval set, which has more statistical power than two separate runs, using McNemar's test for pass-fail metrics on the discordant pairs or a paired bootstrap whose confidence interval for the difference must exclude zero. Two practical traps are highlighted: noise has two sources, the data sampled and the model's non-determinism, so for small sets you run the eval multiple times and average, since detecting a two percent gain can require about nine runs; and the multiple-comparisons trap means that tuning twenty prompts will produce one apparent winner by chance, because testing twenty hypotheses at a five percent level gives a sixty-four percent chance of a false positive, fixable with a held-out test or a Bonferroni correction. A rigor checklist says to report confidence intervals not point estimates, compare on the same set with a paired test, size the eval for the smallest effect you care about, run multiple times for non-deterministic systems, and treat any move within the noise band as no change. The takeaway banner reads: an eval score is a noisy estimate, so attach a confidence interval, only believe a difference when the intervals separate, and never declare victory on a move that lives inside the noise.

Your Eval Score Is an Estimate, Not the Truth

When your eval reports 85%, that is not “the model's performance.” It's an estimate of performance, measured on the particular N examples you happened to test — and like any estimate from a sample, it has uncertainty baked in.

Two things make it wobble. First, which examples you sampled: a different 200 would give a slightly different number. Second, non-determinism: re-run the same prompt on the same model tomorrow and the score moves a point or two on its own. Put those together and you get a noise floor — a band the score bounces around in for free.

The consequence is brutal for small evals: “with only 100 examples, four results flipping is well inside the noise — the same prompt re-run on a different day moves by more than that.” So a 2-point improvement on a 100-example set isn't an improvement at all — it's the measurement equivalent of a coin flip. Before you trust any score, you have to know how wide its noise band is.

Confidence Intervals: the Error Bar Around Your Number

The tool for the noise band is the confidence interval (CI) — the error bar that should accompany every eval score. Instead of “85%,” you report “85% (95% CI: 80–90%),” meaning you're 95% confident the true pass rate lies in that range.

For a pass rate (a proportion), the 95% margin is wonderfully simple:

import math

# Your eval gave 170/200 PASS (graded by the claude-opus-4-8 judge). That 85% is an
# ESTIMATE from a sample -- report it with a 95% confidence interval, never bare.
def ci(passes, n, z=1.96):
    p = passes / n
    margin = z * math.sqrt(p * (1 - p) / n)        # normal approx (use Wilson for small n)
    return p, p - margin, p + margin

p, lo, hi = ci(170, 200)
print(f"{p:.0%}  (95% CI: {lo:.0%}-{hi:.0%})")     # -> 85%  (95% CI: 80%-90%)
# The +/- margin shrinks with sqrt(N): 4x the examples => half the error bar.

Two facts to burn in. One: the margin shrinks with √N. To halve your error bar you need 4× the data — precision is expensive. Two: always report the interval, not the bare point estimate. A naked “85%” hides whether the true value could be 80% or 90% — and that range is exactly what decides whether your next “improvement” is real. A point estimate without a CI is a number pretending to be a fact.

See It: Is B Really Better, or Noise?

Here's the whole lesson in one widget. Version A scores 85%; a new version B scores higher. Drag the sample size and watch both 95% confidence intervals shrink — and drag B's score to set the true gap. The verdict tells you the only thing that matters: do the intervals overlap (could be noise) or separate (a real win)?

Interactive: two eval scores — A (85%) and a draggable B — each shown with its 95% confidence interval as an error bar. Drag the sample size N (10–1000) and the intervals shrink with sqrt(N); drag B's true score to set the gap. A live verdict says whether the CIs overlap (a small gap on a small eval is inside the noise — not yet a win) or separate (B is significantly better). Shows why a 3-point gain needs a big eval, while a 10-point gain is obvious on a small one.

Play with it and the rules become intuition: at N = 20 the error bars are enormous and a 3-point gap is hopeless to call; push N up and they tighten until the gap finally clears the noise. And the bigger the true gap, the less data you need — a 10-point jump is obvious on a tiny set; a 2-point one may never separate without thousands of examples. The interval, not the point, tells you whether to believe the win.

Sample Size: How Many Examples Do You Need?

"How big should my eval set be?" has a precise answer, and it isn't “feels like enough.” It depends on the smallest effect you want to detect — the minimum detectable effect. Detecting a 10-point improvement takes far fewer examples than detecting a 3-point one, because the smaller the signal, the more you must shrink the noise to see it.

The sample-size formula falls right out of the margin: to get a margin of error E, you need

N ≈ z² · p(1−p) / E²

Plug in 95% confidence (z = 1.96), an expected pass rate, and your target margin. As a rule of thumb, a single pass rate stabilizes around 250–500 examples (the standard error drops sharply past ~250), and a 5% margin near an 80% pass rate wants ~250. The discipline: size the set for the difference you actually need to detect — if you're chasing 2-point gains on 50 examples, no amount of staring at the number will help; the math says it can't tell.

Is B Really Better Than A? Comparing Systems

Usually you don't care about a score in isolation — you care whether the new version beat the old one. Two ways to answer it, from quick to rigorous:

The eyeball heuristic (CI overlap). Put both scores' 95% CIs side by side. If they don't overlap, that's strong evidence of a genuine difference. If they overlap heavily, the gap could easily be random — you need more data, or the difference is too small to measure. (That's exactly what the widget above shows.)

The rigorous way (a paired test on the same set). Run both versions on the same eval examples and compare them pairwise — far more powerful than two separate runs, because it cancels the “which examples” noise. For pass/fail metrics, use McNemar's test, which looks only at the discordant pairs (cases where the two versions disagree). Or use a paired bootstrap: resample the examples thousands of times and check that the difference's 95% CI excludes zero.

# New prompt scores 88% vs the old 85% -- both on the SAME 200 examples. Real or noise?
# Quick check: do the 95% CIs overlap?   old 80-90% , new 83-93%  ->  they OVERLAP.
# A 3-point gain on 200 examples sits inside the noise band. Don't ship the "win" yet.

# The rigorous way is a PAIRED test on the SAME set (more power than two separate runs):
from statsmodels.stats.contingency_tables import mcnemar
result = mcnemar(disagreement_table, exact=True)   # uses only the DISCORDANT pairs
ship = result.pvalue < 0.05                         # only NOW is the improvement real
# Tuning 20 prompts? One will "win" by luck -- at a=0.05 that's a 64% false-positive shot.

The takeaway: “B scored higher” is not evidence. “B beat A on the same set, and the difference's CI excludes zero (p < 0.05)” — that is evidence. Same set, paired test, interval on the difference.

Two Noise Sources & the Many-Prompts Trap

Two practical gotchas catch even careful teams:

  • Noise has two sources. There's data noise (which examples you sampled) and prediction noise (the model's non-determinism — different answers on re-runs). For small eval sets, the fix for prediction noise is to run the eval multiple times and average — it directly buys statistical power. (Detecting a 2% gain at p < 0.05 can take ~9 runs; a 1% gain, ~36. This is why small benchmarks are run 10× while huge ones are run once.)
  • The many-prompts trap (multiple comparisons). Tune 20 prompt variants, pick the best, and declare a winner — and you've almost certainly fooled yourself. Testing 20 hypotheses at α = 0.05 gives a 64% chance that at least one “wins” by pure luck. The more variants you try, the more likely your “best” is a fluke. Fixes: validate the winner on a fresh held-out set, or apply a Bonferroni correction (divide your threshold by the number of tests). The prompt that looked best on the set you tuned on is exactly the one most likely to regress.

The Rigor Checklist

Boiled down to what to actually do, every time you read an eval number:

  1. Report a confidence interval, not a point estimate. “85%” → “85% (80–90%).”
  2. Compare on the same set with a paired test (McNemar / paired bootstrap), not two separate runs.
  3. Size the eval for the smallest effect you care about (N ≈ z²p(1−p)/E²) — not by vibes.
  4. Run non-deterministic systems multiple times and average to crush prediction noise.
  5. Treat any move inside the noise band as no change — don't ship it, don't celebrate it.
  6. Beware many-prompt “winners” — validate on held-out data or correct for multiple comparisons.

That's the whole discipline. It's what lets you say, honestly, “this is better” — and be right. Without it, you're navigating by a number that's half signal and half dice.

🧪 Try It Yourself

Audit your most recent “improvement” — it takes five minutes and is often humbling:

  1. Find the numbers. What was your eval size N, and what was the score delta (old → new)?
  2. Compute the margin. Plug into 1.96 * sqrt(p*(1-p)/N). For N = 50 at ~85%, that's about ±10 points. Now compare: is your delta bigger than the margin? If your “win” was 84% → 88% on 50 examples, it's inside the noise — you haven't shown anything yet.
  3. Measure your noise floor directly. Re-run the eval on the same old version twice (or three times). How much does the score bounce between identical runs? That bounce is your noise floor — and a real improvement has to clearly exceed it.

If step 2 or 3 just deflated a result you'd believed, good — you've internalized the most important habit in evaluation: demand that the signal beat the noise before you trust it.

Mental-Model Corrections

  • “85% is the model's score.” No — it's an estimate from a sample, with a margin of error. Report it as 85% (CI 80–90%); the true value could be anywhere in that band.
  • “84% → 86% means we improved.” On a small set, that's almost certainly noise. A 2-point move on 100 examples is the measurement equivalent of a coin flip — check whether it clears the CI.
  • “Just get more examples; one run is fine.” More examples cut data noise, but non-determinism is a second source — for small sets, run multiple times and average.
  • “I tried 20 prompts and #7 won, so ship #7.” Multiple comparisons: with 20 tries there's a 64% chance one wins by luck. Re-validate #7 on fresh data.
  • “Compare by running each version separately.” A paired test on the same set is far more powerful — it removes the which-examples noise. Use McNemar's or a paired bootstrap.
  • “Report the number.” Report the interval. A point estimate without a CI is a number pretending to be a fact.

Key Takeaways

  • An eval score is a noisy estimate, not the truth — it carries a margin of error from which examples you sampled and from the model's non-determinism.
  • Report a confidence interval: for a pass rate, ±1.96·√(p(1−p)/N). The margin shrinks with √N, so 4× the data halves it; a bare point estimate hides the uncertainty that decides everything.
  • Size the eval for the effect you want to detect (N ≈ z²p(1−p)/E²) — ~250–500 stabilizes a pass rate; small effects need big sets. Don't size by vibes.
  • To compare A vs B, use the same set and a paired test (McNemar / paired bootstrap): the difference's CI must exclude zero. Non-overlapping CIs ≈ real; heavy overlap ≈ noise.
  • Two traps: non-determinism (run multiple times and average — detecting 2% can need ~9 runs) and multiple comparisons (20 prompts → 64% chance of a fluke “winner”; hold out or Bonferroni).
  • The one rule: make the signal beat the noise before you believe it. Anything inside the noise band is no change — “it's better, p < 0.05” beats “I think it's better” every time.