Catching Regressions Before They Ship

Introduction

You built validated evals (L173 (From Failures to Targeted Evals)) and you run them in a loop (L174 (The Analyze → Measure → Improve Loop)). This lesson is the safety net that keeps your hard-won quality from quietly sliding backwards: you wire that suite into CI as a regression gate that runs on every change and blocks the merge before a regression can ship.

Here's the uncomfortable truth that makes this non-optional: AI systems regress silently and non-locally. A wording tweak that improves one kind of query degrades another. A provider model upgrade you didn't write a single line of code for — opus-4-7 → opus-4-8 — can change behavior just enough to break JSON parsing or weaken a safety refusal. Without an automated gate, you find out from users, in production, days later. With one, the bad change never leaves the pull request. As the saying in this field goes: evals are the CI of the LLM lifecycle. Let's build that gate.

An infographic titled 'Catching Regressions Before They Ship' about turning your validated eval suite into an automated continuous-integration regression gate that blocks bad changes before they reach production. The central idea is a pipeline with arrows: any behavior-changing change, which includes a reworded prompt, a model or provider version upgrade you did not write code for, a re-embedded retrieval index, a tool change, or a dependency bump, triggers continuous integration on the pull request; the eval suite re-runs every case in the regression set; the new scores are compared against the production baseline; and a quality gate either blocks the merge with a non-zero exit status or lets it pass and deploy. The key danger is that AI regressions are silent and non-local: a wording change that helps one type of query quietly degrades another, and a provider model upgrade can change behavior such as wrapping output in a metadata object so JSON parsing breaks, all with zero code change, so you must re-run the suite on changes you did not even make. The right unit of analysis is the per-case diff, not the aggregate score: you look at which specific test cases went from pass to fail, which is a regression, and which went from fail to pass, which is an improvement, because a change can be net positive in aggregate while still regressing a case you care about. The regression set is sourced from production logs, twenty-five to one hundred cases mixing happy paths, edge cases, and adversarial inputs, and it grows by a ratchet rule: every production failure is added as a permanent test case so the same bug can never silently return, and the baseline only ever climbs. Because models are non-deterministic, a score wobble is not necessarily a regression, so you reduce noise by setting temperature to zero and averaging over k trials, and you only count a drop as real if it falls below the baseline by more than the confidence band, since prediction noise dominates and averaging k runs shrinks it by roughly one over the square root of k. Tolerance is risk-tiered: a small similarity dip might prompt investigation rather than a block, but a safety or toxicity regression is a zero-tolerance hard block, and the suite is layered so cheap deterministic and heuristic checks plus key judges run on every pull request with caching to control cost while the full suite and red-team run nightly. The takeaway banner reads: re-run your validated suite on every change, read the per-case diff, gate safety at zero tolerance, average over trials so you do not block on noise, and pin every caught failure as a permanent test so the baseline only climbs.

Why AI Regressions Are Different

In traditional software, a regression is usually local and loud: you change function f, a unit test for f goes red, you see it instantly. LLM regressions break both of those comforts:

They're non-local. Behavior comes from one tangled artifact — the prompt + model + context — so a change aimed at failure mode A can silently move failure mode B. The research literature even has a name for it: “When Better Prompts Hurt.” A reword that fixes date handling can bury your budget constraint three sentences deeper and break it.
They're silent. There's no stack trace when an answer becomes subtly worse. The output still looks plausible — it's just wrong now.
They happen without you changing anything. Providers ship new model versions; “latest” tags move under you. A documented pattern: an app “passed all tests with one model, then failed the next day” — same code, new model behavior (one model started wrapping outputs in a metadata object, breaking JSON parsing; another answered an off-topic request it used to refuse).

Manual spot-checking can't catch this — it doesn't scale, and the human eye glides right over a plausible-but-degraded answer. The only defense that works is automatically re-running a comprehensive eval suite on every change, including the changes you didn't make.

Your Eval Suite Is the Regression Gate

You already have the instrument. The validated, binary evals from L173 plus their gold sets are a regression suite — “the same prompts, datasets, and evaluator thresholds, re-run after every model, prompt, retriever, or tool change.” Turning it into a gate is a small, well-trodden bit of plumbing:

Treat prompts and config as source code — store them in git (prompts/, model_config.yaml, the RAG index manifest).
Trigger CI on any behavior-changing diff — a pull_request with path filters on those files.
Re-run the suite against the regression set.
Gate the merge — exit non-zero if the thresholds aren't met, which trips GitHub's status check and blocks the merge button. A failed eval is a failed build.

The result: a quality change can't reach main — let alone production — without passing the same bar you validated by hand. Here's the gate:

# Your validated eval suite (L173) IS your regression gate. Re-run it on EVERY change
# before merge — and gate the merge on the result.

# .github/workflows/evals.yml — the TRIGGER (prompts & data are source code):
#   on:
#     pull_request:
#       paths: ["prompts/**", "rag_index/**", "model_config.yaml", "requirements.txt"]
#   jobs: { gate: { steps: ["python regression_gate.py"] } }

import json, sys
from statistics import mean

GOLD     = load_cases("regression_set.jsonl")   # 25-100 cases from PROD + every past bug
BASELINE = json.load(open("baseline.json"))     # per-case scores of the SHIPPED version
NOISE    = json.load(open("noise.json"))         # per-case noise band (see Code 2)

def regression_gate(app, suite, k=5):
    regressed, improved = [], []
    for c in GOLD:
        score = mean(suite.run(app, c) for _ in range(k))      # avg k trials -> de-noise
        b = BASELINE[c.id]
        if   score < b - NOISE[c.id]: regressed.append(c.id)   # real drop, beyond noise
        elif score > b + NOISE[c.id]: improved.append(c.id)
    crit  = [c for c in regressed if GOLD_BY_ID[c].critical]    # safety = ZERO tolerance
    block = bool(crit) or len(regressed) > 0                    # strict: ANY regression blocks
    print(f"improved={improved}  regressed={regressed}  critical={crit}")
    sys.exit(1 if block else 0)        # non-zero exit -> merge-blocking GitHub status check
# A new model version, a reworded prompt, a re-embedded index — all run this gate. Nothing
# ships if a case regressed. When a fix lands, the baseline RATCHETS up; it never drops.

Tools like promptfoo (used by OpenAI and Anthropic), Braintrust, and LangSmith package this exact flow — a GitHub Action that runs the suite and posts the per-case improved/regressed diff as a PR comment. But the logic is just: re-run the suite, compare to baseline, exit non-zero on a regression. The gate doesn't fix anything — it catches, so a human decides before it ships.

The Regression Set & the Ratchet

A regression gate is only as good as the cases it runs. Two rules build a great regression set:

Source it from reality. Start with 25–100 cases pulled from production logs, tagged by difficulty and failure type, mixing happy paths, edge cases, adversarial inputs, and off-topic requests. Real traffic reveals patterns synthetic examples miss (this is the data from your error analysis, L171 (Look at Your Data — The #1 Rule)).

Then ratchet. This is the single most important habit in the lesson:

Every time a failure reaches production, add it to the regression set as a permanent test — before you ship the fix.

That one discipline converts each bug into a vaccine: the same failure can never silently return, because the gate now checks for it forever. Your baseline becomes a floor you never drop below — historical performance ratchets up, never down. A suite that grows one production incident at a time becomes, within months, an institutional memory of every way your system has ever broken — and the thing that stops it from breaking that way again.

Read the Per-Case Diff, Not the Aggregate

Here's the mistake that lets regressions through even with a suite: gating on the aggregate score alone. A change can be net-positive overall while regressing a specific case you care about — it fixes three cases and breaks one, the average ticks up, and you ship a broken behavior.

The correct unit of analysis is the per-case diff — for every case in the set, did it go:

pass → fail? That's a regression (the dangerous one).
fail → pass? That's an improvement.
unchanged? Fine.

Good tooling surfaces exactly this: “which test cases improved, which regressed, and by how much.” You read the regressed list every time, regardless of what the average did. Especially: a regression on a critical case — a safety refusal, a PII check, a legal disclaimer — is a hard block at zero tolerance, even if the overall score went up. Net-positive is not good enough when one of the things that broke is the thing that must never break.

See It: Run the Gate

Push a change through the gate yourself. Pick a change to ship — a clean tool fix, a “better” prompt reword, a silent model upgrade, a RAG re-index — tune the CI run, and hit run. Watch the per-case diff and the gate's verdict. Try the model upgrade (you “changed no code”) and watch it regress a critical safety case. Then play with trials and temperature and see a “regression” turn out to be noise.

Interactive: a playable CI regression gate on a leasing assistant with an 8-case regression set (baseline 6/8 passing; 'Refuse unsafe / off-topic' is a CRITICAL case). The user (1) picks ONE change to ship — Add a date-parsing tool / Reword the system prompt to fix dates / Upgrade model opus-4-7→4-8 (no code change) / Update the RAG index / Add an output guardrail; (2) tunes the CI run with three segmented controls — trials k (1/5/20), temperature (0/0.7), and tolerance (any-regression-blocks / net-score); (3) runs the gate. A pipeline strip with arrow connectors animates: CHANGE ▸ RUN SUITE ×k ▸ COMPARE BASELINE ▸ GATE PASS/BLOCK. Results show the aggregate score vs baseline (with a ± noise band), counts of improved↑ and regressed↓ cases, and a PER-CASE DIFF grid where each case shows baseline-dot → new-dot colored green (FIXED, fail→pass) / red (REGRESSED, pass→fail) / grey (same), with flaky flips tagged ⚡FLAKY?. The gate BLOCKS the merge on any regression (or any critical-safety regression at zero tolerance) with a non-zero-exit explanation, or PASSES. The teaching: the date tool is a clean win (ships); the reworded prompt fixes dates but regresses budget + JSON non-locally (blocked); the model upgrade fixes some cases but breaks the CRITICAL safety case with zero code change (blocked — why you re-run on model bumps you didn't make); the RAG update is net-zero but regresses citations (blocked under strict, ships under net-score — the tolerance tradeoff). With temp 0.7 and k=1 a borderline case flips as ⚡FLAKY noise and can cause a false block; setting temp 0 or k=20 averages it out — real-regression-vs-noise. Blocked changes offer Revert and '＋ pin the regressed case to the set' (the ratchet); shipped changes ratchet the baseline up. Lets the reader run all 5 changes × the CI settings and feel why you read the per-case diff, gate safety at zero tolerance, and de-noise before trusting a verdict.

Two things should jump out. The silent model upgrade — where you changed nothing — regresses a critical safety case and gets blocked: that's the gate earning its keep against a change you'd never have eval'd by hand. And the “better” prompt that fixes dates but non-locally breaks budget and JSON: net-positive aggregate, two real regressions, blocked. The gate sees what your eyes wouldn't.

Real Regression, or Just Noise?

Now the subtle part that separates a robust gate from a flaky one. LLMs are non-deterministic — the same input can produce different outputs — so a score that dips from 92% to 90% between two runs might be a real regression or might be pure noise. Gate naively and you'll either block good changes (false alarms) or, worse, learn to ignore the gate.

Three disciplines tame it:

Pin down what you can. Run the judge at temperature=0 for reproducible scoring, and never gate free-form text on exact-match — gate on assertions and thresholds (L160 (Functional Correctness & Exact Match)) that tolerate benign variation.
Average over k trials. Prediction noise (same question, different answer) dominates data noise, and averaging k runs shrinks it by roughly 1/√k. More trials → a tighter band → power to detect a small but real regression.
Compare confidence bands, not point estimates. Put a CI on the score (L165 (Statistical Rigor: Sample Size & Confidence)) and only count a drop as a regression if the new mean falls below baseline by more than the noise band.

And watch for flaky cases: a test that flips more than ~30% of the time is too noisy to gate on — stabilize it (temp 0, tighten the assertion) before it blocks merges on coin-flips.

# A score wobble is NOT a regression. LLMs are non-deterministic, so a drop only counts if
# it clears the NOISE band. Two moves: temperature=0 + average over k trials.
from statistics import mean, stdev

def noise_band(suite, app, case, k=20, z=2.0):
    runs = [suite.run(app, case) for _ in range(k)]   # k samples (judge at temperature=0)
    sd   = stdev(runs) if k > 1 else 0.0
    return z * sd / (k ** 0.5)            # 95% CI half-width on the mean (the band; L165)

def is_real_regression(new_mean, baseline_mean, band):
    return new_mean < baseline_mean - band      # below baseline by MORE than the noise band
# Prediction noise (same input, different output) dominates data noise, and averaging k runs
# cuts it ~1/sqrt(k) -> more trials = a tighter band = power to see a SMALL real regression.
# A case that flips > ~30% of the time is FLAKY: stabilize it (temp=0, tighten the assertion,
# don't exact-match free-form text) before you let it gate a merge.

In the widget, run the same change at temp 0.7, k=1 a few times: a borderline case flips as ⚡FLAKY? and can falsely block the merge. Switch to temp 0 or k=20 and the noise averages out — the spurious “regression” disappears, and the verdict is one you can trust. De-noise before you decide.

Thresholds, Tolerance & What Runs in CI

Define the gate's rules in advance, so there's no arguing at merge time:

Absolute floors — a minimum the suite must clear (e.g. “pass rate ≥ 90%”, “accuracy ≥ 0.85”).
Relative tolerance — “no regression beyond X vs. the production baseline.” For most metrics a tiny dip warrants investigation, not a block; for safety, PII, toxicity, and other critical cases, tolerance is zero — any regression is an immediate block (and in production, a rollback).
Risk-tier the approvals — low-risk changes merge on a green gate alone; medium-risk needs a human reviewer; high-stakes domains (legal, medical, financial) need two independent reviewers before going live.

Then manage cost and latency with a layered, tiered suite (the cheap-first ladder from L173):

Per-PR (fast, blocking): deterministic checks (regex/JSON), heuristic scores (embedding similarity), and the key judges — with caching (don't re-query the model when inputs are unchanged) so the gate stays quick and cheap.
Nightly / scheduled (deep): the full suite, expensive judges, and red-team / adversarial runs (jailbreak, prompt-injection, PII).
Production (live): continuous monitoring that feeds new failures back into the regression set — closing the loop to L174.

The gate is a decision-support instrument, not a tyrant: it makes the quality consequence of every change visible and explicit before it ships.

🧪 Try It Yourself

Stand up a real regression gate this week — it's an afternoon of plumbing:

Put your prompts in git. Move your system prompt, model config, and RAG manifest into version-controlled files.
Make a regression set. Pull 25–50 real cases from your logs — happy paths, edge cases, and every bug you've fixed. Save expected behavior / pass criteria with each.
Write the gate. A script (or promptfoo / Braintrust / LangSmith) that runs your suite and exits non-zero if pass rate drops below your floor or any case regresses vs. a saved baseline.json.
Wire it to CI. A GitHub Action on pull_request with path filters on prompts/** and your config — so it runs automatically and blocks the merge on red.
Add the ratchet. Make a team rule: a production bug isn't closed until its case is in the regression set. Then re-run the gate against opus-4-8 vs your pinned model — see if the upgrade is safe before you take it.

When you're done, quality can only go up: every change is checked against every past failure, and nothing that regresses can ship.

Mental-Model Corrections

“Tests passed once, so we're safe.” Re-run the suite on every change — including model/provider upgrades you didn't make and RAG/data updates. Behavior drifts silently underneath you.
“The aggregate score went up, ship it.” Read the per-case diff. A net-positive change can regress a specific case — and if that case is safety, net-positive is irrelevant. Gate critical cases at zero tolerance.
“The score dropped 2% — that's a regression.” Maybe not. LLMs are non-deterministic; a drop counts only if it clears the noise band. Use temp 0, average over k trials, compare confidence intervals.
“Exact-match the output in CI.” Free-form output varies benignly run to run; exact-match makes the gate flaky. Gate on assertions and thresholds (L160), not string equality.
“Manual spot-checks before each release are enough.” They don't scale and they miss non-local, plausible-looking regressions. Automate the gate.
“Once it's caught, the bug is handled.” Only if you pin it as a permanent test (the ratchet). Otherwise the same regression returns the next time someone edits the prompt.

Key Takeaways

Wire your validated suite into CI as a regression gate — it re-runs on every behavior-changing diff (prompt, RAG data, deps, and model upgrades you didn't make) and blocks the merge (non-zero exit) on a regression. Evals are the CI of the LLM lifecycle.
AI regressions are silent and non-local: a “better” prompt degrades another case; a provider model bump breaks behavior with zero code change. Only an automated gate catches them.
Read the per-case diff, not the aggregate — pass→fail is a regression even when the average rises. Critical cases (safety/PII) gate at zero tolerance.
Build the set from production and ratchet it: 25–100 real cases, and every production failure becomes a permanent test — the baseline only ever climbs (L171).
Distinguish a real regression from noise: LLMs are non-deterministic, so use temp 0, average over k trials (noise shrinks ~1/√k), and compare confidence bands (L165), not point estimates. Flaky (>~30%) cases can't gate.
Tier the suite for CI: cheap deterministic + key judges per-PR (cached, fast); full suite + red-team nightly; feed production failures back into the set. Next: a full case study of debugging a failing feature with exactly these tools (L176 (Case Study: Debugging a Failing AI Feature)).