Detecting Drift & Degradation

Introduction

You're now measuring quality continuously on live traffic (L180 (Online Evals & Guardrail Metrics)). This lesson is about reading that trend for the most insidious production problem there is: drift — your system getting worse over time with no error, no crash, no alert. It just quietly degrades until users complain.

What makes LLM drift uniquely dangerous is that it's often not your fault and not in your code. The model your app was tested against in March may not be the model answering requests in April — providers update models silently, and “a carefully crafted prompt that worked perfectly yesterday might produce completely different outputs after a model update, even using the same version number.” Stanford and Berkeley documented this starkly: GPT-4's accuracy at identifying whether 17077 is prime fell from 97.6% in March 2023 to 2.4% in June — same task, same model name. This lesson is how you catch that decay early — what to monitor, the one alerting rule that separates signal from noise, and how to match the fix to the kind of drift.

An infographic titled 'Detecting Drift and Degradation' about how an AI system's quality silently decays in production and how to catch it. The danger is that drift is silent: there is no error or crash, the system just gets gradually or suddenly worse, and it is often not even your fault because the model can change underneath you. The canonical evidence is the Stanford study on how ChatGPT's behavior changed over time, where GPT-4's accuracy at identifying a prime number fell from ninety-seven point six percent in March to two point four percent in June on the exact same task and the same model name. There are four kinds of drift. Data or input drift is when the distribution of inputs shifts, for example a new user segment, language, or topic the system was never tuned for. Concept drift is when the relationship between input and the correct output changes because the world moved, such as a new product line or policy, so the right answer changed but your knowledge did not. Model drift is when the provider silently updates the model behind the API, so a prompt that worked yesterday behaves differently today even under the same version string. Embedding or semantic drift is when the vector space or the RAG corpus shifts so retrieval degrades. You detect drift by watching three things: the quality trend from your online evals, cheap leading-indicator proxy metrics like refusal rate, response length, latency, and topic distribution, and statistical distribution tests like the Population Stability Index comparing a reference window to the current window, where PSI below zero point one is stable, zero point one to zero point two is a moderate shift, and above zero point two is significant drift. The single most important rule is to alert on the joint condition of input drift and a measurable eval drop together, because drift without an eval impact is a false alarm that burns on-call, and quality can crater with no input drift at all when the model is silently swapped. Finally you match the response to the type: for model drift pin the model version, re-evaluate against the new version, and roll back; for concept drift refresh the RAG corpus and knowledge and re-eval; for data drift expand the eval set with the new segment and add examples; for embedding drift re-embed and re-index the corpus. The takeaway banner reads: drift is silent and often external, so watch the quality trend plus proxies plus PSI, alert only on drift with an eval drop, and match the fix to the drift type.

The Four Kinds of Drift

“Drift” isn't one thing — and the type decides the fix, so name it precisely:

Data / input drift — the distribution of inputs changes. New users, a new language, a new topic, slang that didn't exist when you tuned the system. “Phrases that were once rarely used become mainstream.” The inputs moved; your system is now seeing things it wasn't built for.
Concept drift — the relationship between input and the correct output changes. The world moved: a new product launched, a policy changed, “best laptop” now means a different model. The question looks the same but the right answer is different — and your knowledge is stale.
Model drift — the model itself changes. For LLMs this is usually external and silent: the provider updates the model behind the API (the Stanford GPT-4 case). Your prompt, your data, your code — all unchanged — and behavior shifts anyway. “Prompt decay.”
Embedding / semantic drift — the vector space or RAG corpus shifts, so retrieval degrades: the embeddings “begin to shift relative to the space your system was calibrated against,” or your indexed corpus goes stale.

All four end in the same symptom — declining quality — but a fix for one does nothing for another. Pinning a model version won't help concept drift; re-indexing your corpus won't help a provider's model update. Diagnose before you treat.

How to Detect It: Trend, Proxies, PSI

Drift is silent, so you have to instrument for it — you won't get an exception. Three complementary signals:

The quality trend. Your online eval score (L180) over time, sliced by route/model/prompt_version, is the direct measure. A downward trend (gradual or a sudden step) is degradation. This is the signal that matters most.
Cheap proxy / leading indicators. Judge calls cost money, so watch free canary metrics that move early: refusal rate, retry/regenerate rate, response length, latency, token usage, topic distribution, sentiment, and user feedback (👍/👎 — L182 (Capturing & Using User Feedback)). A jump in “I can't help with that” or a 30% length change is a leading indicator that something shifted before your judge score fully reacts.
Distribution tests. Compare a reference window (a known-good baseline period) to the current window with a statistic. The workhorse is PSI (Population Stability Index) — bin both distributions and sum the weighted log-ratio of the per-bin shares:

# Population Stability Index: how far has a distribution moved from a REFERENCE window?
import numpy as np

def psi(reference, current, bins=10):
    edges = np.quantile(reference, np.linspace(0, 1, bins + 1))   # bin on reference quantiles
    edges[0], edges[-1] = -np.inf, np.inf
    ref_pct = np.histogram(reference, edges)[0] / len(reference) + 1e-6
    cur_pct = np.histogram(current,   edges)[0] / len(current)   + 1e-6
    return float(np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct)))   # the PSI sum
#   PSI < 0.1   -> stable (no real shift)
#   0.1 - 0.2   -> moderate shift (keep watching)
#   PSI > 0.2   -> SIGNIFICANT drift (investigate)
# Run it on a 1-D signal (response length, latency) OR a projection of INPUT EMBEDDINGS, comparing
# a fixed reference window to a rolling current window. (KS / KL / chi-square work too.)

The PSI thresholds are the standard rule of thumb: < 0.1 = stable, 0.1–0.2 = moderate shift (keep watching), > 0.2 = significant drift. Run it on input embeddings (input drift), on response-length or refusal-rate (output drift), and on retrieved-doc similarity (embedding drift). KS test, KL divergence, and chi-square do the same job; PSI is just the most common in production.

The One Rule: Alert on the Joint Condition

Here's the rule that separates a useful drift monitor from a pager that everyone mutes: alert on the joint condition — input drift AND a measurable eval drop. Not either one alone.

Why both? Because the two failure modes of naive alerting are equally bad:

Alert on input drift alone → false alarms. Your traffic mix shifts all the time — a marketing campaign, a new region, a seasonal topic. Input drift with no quality impact is benign, and paging on-call for it “burns trust” until people ignore the alerts. “Drift without eval impact is a false alarm and burns on-call.”
Alert on input drift alone → you also miss the scariest case. A silent model swap (the Stanford case) drops your quality with zero input drift — your inputs are identical, the model changed. An input-drift monitor sees nothing while quality craters. The eval score is the only thing that catches it.

So the logic is: drift + eval drop → real, investigate; drift, no drop → watch, don't page; no drift, eval drop → suspect a model change. Here's the joint-condition alert and the per-type response routing:

# DON'T page on drift alone. Alert on the JOINT condition: input drift AND an eval drop.
def classify(ref, cur):
    input_drift  = psi(ref.input_embeddings, cur.input_embeddings) > 0.2   # distribution moved a lot
    quality_drop = significant(ref.eval_scores, cur.eval_scores)           # AND quality fell (L165)
    if input_drift and not quality_drop:
        return "WATCH"            # benign shift -> do NOT burn on-call (drift w/o eval impact)
    if quality_drop and not input_drift:
        return "MODEL_DRIFT"      # quality fell with STABLE inputs -> the model changed under you
    if quality_drop and input_drift:
        return "DATA_OR_CONCEPT"  # new inputs your system can't handle yet
    return "HEALTHY"

RESPONSE = {                                       # match the fix to the TYPE
    "model":     ["pin the model version", "re-eval vs the new version", "rollback"],
    "concept":   ["refresh the RAG corpus / knowledge", "re-label & re-eval"],
    "data":      ["expand the eval set with the new segment", "add examples / RAG"],
    "embedding": ["re-embed & re-index the corpus"],
}

Notice the deep point: the input-drift signal and the quality signal are independent, and you need both. Quality is the thing you actually care about; input drift is the explanation. Alert on quality, use drift to diagnose — and never page on an explanation with no symptom.

See It: Investigate the Drift

Four real drift events, four investigations. Each shows the quality trend (your online eval) and the input-drift (PSI) over 10 weeks, with the reference and current windows marked. For each: decide whether to page on-call (the joint condition!), then classify the drift type from its signature, then pick the matching response. Watch the trap — the scenario where PSI is huge but you should not page, and the one where PSI is flat but quality has cratered.

Interactive: a drift-investigation console. Four scenarios as tabs, each a 10-week chart with the QUALITY line (online eval %, with a dashed 80% alert line) plus INPUT-DRIFT (PSI) bars along the bottom, and the reference window (weeks 1-3) + current window (weeks 8-10) shaded. Readouts show current PSI (with its <0.1/0.1-0.2/>0.2 band), quality now vs reference, and the quality delta (significant drop vs within-noise). The user works three steps: ① page on-call? (alert) — correct only on the JOINT condition (drift AND a real quality drop); ② classify the drift TYPE (data/input · concept · model · embedding) from the signature; ③ pick the matching RESPONSE (pin the model version · refresh RAG/knowledge · expand the eval set · no-page monitoring). The four scenarios teach the signatures: (1) 'sudden quality drop' — PSI flat ~0.05 but quality steps 88%→70%: a silent MODEL swap (the Stanford GPT-4 case), invisible to input-drift alarms → pin the version/rollback; (2) 'big input shift, no impact' — PSI 0.35 but quality holds ~87%: a FALSE ALARM, do NOT page (drift without eval impact); (3) 'slow decline' — PSI and quality both drift together gradually: CONCEPT drift (the world changed) → refresh knowledge/RAG; (4) 'a new segment fails' — PSI spikes 0.31 exactly as quality drops: DATA/input drift → expand the eval set + add examples. Wrong choices get a hint pointing at the signature. Teaches: alert on the joint condition, classify drift by its signature, and match the fix to the type.

The two trap scenarios are the whole lesson. Big input shift, quality fine → you click “No alert” and learn that drift alone is a false alarm. Flat input drift, quality cratered → you learn that a silent model swap is invisible to a drift monitor and only the eval trend catches it. Watch the symptom (quality); use drift to explain it.

Responses: Match the Fix to the Type

Once you've confirmed a real regression (drift and an eval drop) and named the type, the response follows directly:

Model drift (quality dropped, inputs stable → a provider update) → pin the model version (move off floating “latest” tags; L: Versioning & Reproducibility), re-evaluate your suite against the new version, and roll back to the pinned version if it regressed. Then re-tune the prompt for the new model deliberately.
Concept drift (the world moved) → refresh the knowledge: update the RAG corpus, re-label your gold set to the new ground truth, update the prompt's facts, and re-eval. The model's fine; your information is stale.
Data / input drift (a new segment) → expand the eval set with the new inputs so they're measured, add examples / RAG coverage for the segment, and (rarely) fine-tune. You're teaching the system the inputs it now sees.
Embedding / RAG drift → re-embed and re-index the corpus against the current embedding space.

And the meta-response that prevents the worst surprises: pin everything you can (model version, prompt, RAG snapshot) so behavior is reproducible, and add the regression to your eval suite (L175 (Catching Regressions Before They Ship)) so the same drift can't silently recur. Drift you've seen once should become a test.

🧪 Try It Yourself

Set up drift detection for one feature this week:

Plot the quality trend. Take your online-eval score (L180) and chart it over time, sliced by model and prompt_version. Eyeball it for a downward trend or a step.
Add two cheap proxies. Log refusal rate and response length per day — they're free and they move early. Alert if either shifts > ~20% week-over-week.
Compute one PSI. Pick a fixed reference window (a known-good month). Each week, compute PSI of response length (or input-embedding projection) vs. the reference. Flag > 0.2.
Wire the joint alert. Only page when PSI > 0.2 AND the quality score dropped significantly (L165). Route “drift, no drop” to a watch list, not the pager.
Pin your model version today, and add a canary eval that runs your gold set against the provider's latest weekly — so you find out about a silent model update before your users do.

Do this and the scariest production failure — silent decay — becomes a chart with an alert instead of a wave of complaints.

Mental-Model Corrections

“If something breaks, I'll get an error.” Drift is silent — no exception, just gradually worse output. You only see it if you measure the trend.
“My system degraded, so my code is buggy.” Often it's external: the provider silently updated the model (Stanford GPT-4, 97.6% → 2.4%). Same code, same prompt, different model. Pin the version.
“Alert whenever input drift is high.” No — page on the joint condition (drift and an eval drop). Input drift alone is usually benign and pages you for nothing; it also misses a silent model swap (drift flat, quality cratered).
“Drift is drift.” Four kinds — data, concept, model, embedding — and the fix differs. Pinning a version won't fix concept drift; re-indexing won't fix a provider update. Diagnose the type.
“I'll just watch the dashboard.” Watch the quality trend with a reference window and a statistic (PSI > 0.2), plus cheap proxies (refusal rate, length) that move early — eyeballing a noisy line misses gradual decay.
“Once I fix it, it's handled.” Add the regression to your eval suite (L175) and pin the artifacts — drift you've seen once should become a test so it can't silently return.

Key Takeaways

Drift is silent and often external — your system degrades with no error, and the provider can change the model under you (Stanford GPT-4: 97.6% → 2.4%, same task & model name). You catch it only by monitoring the trend.
Four kinds, four fixes: data/input (new segment → expand the eval set), concept (world changed → refresh knowledge/RAG), model (provider update → pin the version, rollback), embedding/RAG (re-index). Name the type before you treat.
Detect with three signals: the online-eval quality trend (direct), cheap proxy metrics (refusal rate, length, latency, topic mix — leading indicators), and distribution tests comparing a reference vs current window — PSI (< 0.1 stable · 0.1–0.2 moderate · > 0.2 significant), or KS/KL/chi-square.
The one rule: alert on the JOINT condition — input drift AND a measurable eval drop. Drift alone is a false alarm (burns on-call); a quality drop with no input drift is a silent model swap. Quality is the symptom; drift is the explanation.
Pin what you can (model, prompt, corpus) for reproducibility, and add every confirmed drift to your eval suite (L175) so it becomes a regression test.
Next: one of the richest early-warning signals is your users themselves — capturing & using user feedback (L182 (Capturing & Using User Feedback)).