Capturing & Using User Feedback

Introduction

Your users are running the largest evaluation of your AI product every single day — for free. Every thumbs-down, every regenerate, every time someone edits your answer or gives up and asks for a human, they're telling you exactly where the system fails. This lesson is about capturing those signals and — the subtle part — using them correctly.

The trap is to treat user feedback as ground truth. It isn't. A 👎 is a fantastic lens to understand your users, but a noisy learning signal: people thumbs-down when they're in a bad mood, when they were wrong, or when the answer was fine but slow. Train directly on raw thumbs and you'll teach your model the wrong lesson (you saw the extreme version — sycophancy from optimizing the 👍 proxy — in L: The Feedback Flywheel). The skill is to capture every signal, calibrate the noisy ones, and route each into the right part of your eval machinery — corrections become gold labels, complaints feed error analysis, rate-spikes feed drift detection. Done right, your users' feedback becomes your richest dataset.

An infographic titled 'Capturing and Using User Feedback' about turning production user signals into eval data without being misled by them. There are two kinds of signal. Explicit feedback, like thumbs up and down, star ratings, and report buttons, is direct but covers only about one to three percent of interactions and skews toward extremes, because the people who click are the very happy or very frustrated while the silent majority goes unheard. Implicit feedback, emitted by behavior, covers one hundred percent of interactions and is objective rather than self-reported: a user editing the answer is a correction and the highest-value signal because it is user-supplied ground truth, regenerating is dissatisfaction, copying is value, abandoning the conversation is failure, asking for a human is a limitation, and rephrasing the same question is a comprehension failure. The central rule is that a thumbs down is a signal, not a label: it is a great lens to understand users but a noisy learning signal, so you calibrate it with human review before treating it as ground truth, and you never fine-tune on raw thumbs, which is the Goodhart trap. You also read patterns, not individual signals, because one angry click means little but a spike in the abandonment rate on one topic from ten to thirty-two percent is a real problem. You capture each signal tied to the full trace with metadata, then route it to where it belongs: a correction becomes a gold eval label and a fine-tune pair, a thumbs down or abandon or escalate goes to the error-analysis queue and human review, a copy or return goes to the positive set, and a rate spike becomes a drift or monitoring alert. That routed and calibrated stream is your next dataset, which closes the loop into production-to-dataset. The takeaway banner reads: explicit feedback is sparse and biased while implicit signals cover everything; a thumbs down is a signal, not a label, so calibrate before you label; and route every signal into the machinery, corrections to gold, negatives to error analysis, spikes to drift.

Two Kinds of Signal: Explicit vs Implicit

Feedback comes in two flavors, and confusing their roles is the first mistake:

Explicit feedback — the user deliberately tells you: 👍/👎, star ratings, a “report a problem” button, a written comment. It's direct and labeled, but it has two fatal weaknesses:

Sparse. “Fewer than 1% of users click thumbs-up or thumbs-down”; even CSAT surveys hover around 20%. You hear from 1–3% of interactions.
Biased. “Responders skew toward extremes — people who are very happy or very frustrated click. The silent majority goes unheard.” It's a vocal-minority sample, not a representative one.

Implicit feedback — the user tells you through behavior, without being asked: they regenerate, copy the answer, edit it, abandon the chat, rephrase, escalate to a human. Its strengths mirror explicit's weaknesses:

Dense. ~100% coverage — every interaction emits implicit signals.
Objective. Based on what users actually did, not self-reported sentiment, so it's more representative of your real user base.

The rule that follows: lead with implicit signals (dense, objective) and use sparse explicit feedback to calibrate and spot specific issues, not as your primary measure. And never read one signal in isolation — “a single quick exit means little; but if 30% of users on one topic abandon while others see 10%, that pattern is a problem.” Patterns, not individual clicks.

The Implicit Signals — and What Each Means

Implicit signals are gold because you can read intent from behavior. Learn the vocabulary:

✎ Edit / correct the answer → the user handed you ground truth. This is the single highest-value signal — the corrected output is a ready-made gold label and a fine-tune (SFT) pair.
↻ Regenerate / retry → dissatisfaction with the response, with no stated reason. A strong negative.
⟲ Rephrase the same question → comprehension failure — “the AI didn't understand or answer effectively.” Direct evidence of a miss.
⧉ Copy the response → value — they found it good enough to use. A positive example.
🙋 Escalate to a human → limitation — the AI couldn't resolve it. A real failure.
🚪 Abandon the conversation → frustration / gave up without completing the goal. Negative.
↩ Return next day/week → trust / retention. A slow but strong positive.
⏱ Quick completion → ambiguous — fast resolution or immediate frustration. Read it with other signals.

Monitor each per intent / topic category, because global averages hide the truth: an internal copilot might have “high rephrasing on benefits questions but low on PTO — it handles PTO well and struggles with benefits.” The signal is in the breakdown, not the average.

A 👎 Is a Signal, Not a Label

This is the idea that separates teams who use feedback from teams who get misled by it. Raw user feedback is a noisy signal — never treat it as ground truth.

Why a 👎 lies often enough to matter:

Selection bias. Only the extremes respond; you're labeling from the vocal minority, not your user base.
The user is sometimes wrong. They thumbs-down a correct answer they didn't like, or one that contradicted a misconception. The label says “bad”; the answer was good.
Mood and irrelevant factors. Latency, an unrelated frustration, a fat-fingered click.
It's coarse. “Sparse, binary, occasionally adversarial, lacking the richness of curated preference pairs.”

The consequence: do not fine-tune (or auto-label your eval set) on raw thumbs. That's the path to Goodhart's trap — optimize the 👍 proxy and you get a model that's liked but not good (sycophancy; L: The Feedback Flywheel). Instead:

Treat feedback as a prioritized queue, not labels — a 👎 says “a human should look here,” not “this is the correct answer.”
Calibrate it. Sample the 👎s and review them (the spot-check discipline from L170 (Human-in-the-Loop)): what fraction are real failures vs. user-error vs. noise? Now you know how much to trust the signal.
Only the verified ones become labels. A reviewed-and-confirmed failure (or a user correction) is ground truth; a raw thumbs-down is a lead.

“A lens to understand your users, but noisy as a learning signal.” Understand first; label second.

See It: Triage the Feedback

Work a real feedback queue. Six production events arrive — explicit (👎) and implicit (edit, regenerate, copy, escalate) plus a rate spike — and you route each into the right home: a gold label, the error-analysis queue, a drift alert, the positive set, or human review. Watch for the trap: one event is a 👎 on an answer that was actually correct — route it straight to “gold label” and you'll train on a lie.

Interactive: a feedback triage / routing console. A coverage strip contrasts explicit feedback (~1-3% of interactions, skewed to extremes) vs implicit (100%, objective behavior). Then a queue of 6 production feedback events, each with its signal, the interaction, and the context: (1) ✎ the user EDITED '90 days' to '30 days' — a correction; (2) 👎 thumbs-down with no comment on a password-reset answer that logs show was actually CORRECT (the user succeeded) — the trap; (3) ↻ regenerated 3× on a vague contract summary; (4) ⧉ copied a SQL query; (5) 🙋 asked for a human on a double-charge; (6) 📈 abandonment on the 'billing' intent spiked 10%→32% this week. For each, the user picks a route from five: Gold label + SFT · Error-analysis queue · Drift / monitoring alert · Positive set · Human review (calibrate). Correct routings reveal the why; the trap (routing the vague 👎 to 'gold label') fires '✗ a 👎 is not ground truth — this answer was correct; labeling it teaches a wrong example → human review to calibrate first'. The right homes: correction→gold label + SFT; regenerate/escalate→error analysis; copy→positive set; the rate spike→drift alert; the trap 👎→review. A flywheel bar fills as events are routed (feedback ▸ review/calibrate ▸ dataset ▸ error analysis ▸ eval+improve) and a completion panel summarizes. Teaches: explicit is sparse & biased, implicit covers 100%, a 👎 is a signal not a label (calibrate), and route every signal into the eval/data machinery.

The trap event is the lesson in miniature. The instinct on a 👎 is “add it to the failures.” But this one's answer worked — the user just clicked angrily. Auto-label it and you've poisoned your gold set. Route it to review, confirm it's noise, and move on. Corrections are gold; thumbs are leads.

Capturing It Well

Good feedback data is engineered, not just collected. Five practices:

Make explicit feedback frictionless — a single 👍/👎 inline, an optional one-tap category (“wrong / unhelpful / unsafe”). Every extra field halves your already-tiny response rate.
Capture implicit signals automatically — instrument regenerate, copy, edit, abandon, escalate, dwell. These are free and cover 100% of traffic; most teams leave them on the floor.
Tie every signal to the full trace (L177 (Tracing LLM & Agent Calls)) — the input, retrieved docs, the exact output, the model and prompt version. A thumbs-down with no context is useless; with the trace, it's a debuggable case.
Run the pipeline: collect → label → clean → extract. Capture the raw signal + metadata, enrich with categories (issue type, sentiment) via auto-tagging, clean (remove spam, generic comments, and PII — feedback text is full of it), then extract the examples you need.
Mind governance — feedback contains personal data; honor redaction, retention, and the right-to-be-forgotten (GDPR / EU AI Act). Pseudonymize user IDs.

And ask at the right moment: request explicit feedback after a completed task, on high-value interactions — not as a pop-up that interrupts the work.

Using It: Route Into the Machinery

Captured and calibrated, feedback plugs into every part of the eval system you've built. The routing:

Corrections → gold labels + fine-tune data. A user's edited answer is ground truth: add it to your eval gold set (L173 (From Failures to Targeted Evals)) and your SFT pairs.
Negatives (👎, abandon, escalate, regenerate) → error analysis. They're a prioritized queue for “look at your data” (L171 (Look at Your Data — The #1 Rule)): read them, cluster into a taxonomy (L172), and each confirmed mode becomes a regression test (L175 (Catching Regressions Before They Ship)).
Positives (copy, return) → the “good” set — few-shot examples and positive eval cases.
Feedback rates → drift signals. A spike in 👎/abandon/refusal rate is a cheap leading indicator for drift detection (L181 (Detecting Drift & Degradation)) — patterns, not clicks.
Thumbs → a (weak, calibrated) online-eval signal (L180 (Online Evals & Guardrail Metrics)) — trend it, never trust a single one.

Here's the capture and the routing in code:

# Capture every signal — explicit AND implicit — tied to the FULL trace (L177), not just a score.
EXPLICIT = {"thumb_up", "thumb_down", "rating", "report"}

def log_feedback(trace, signal, value=None, correction=None):
    store({
        "trace_id":   trace.id,          # the whole interaction: input, retrieved docs, output (L177)
        "signal":     signal,            # thumb_down | edit | regenerate | copy | abandon | escalate ...
        "kind":       "explicit" if signal in EXPLICIT else "implicit",
        "value":      value,             # rating / issue category, if any
        "correction": correction,        # the user's edited answer = GROUND TRUTH, if they gave one
        "route":      trace.route, "ts": now(), "user": pseudonymize(trace.user_id),
    })

# Implicit signals are emitted by BEHAVIOR — capture them automatically -> ~100% coverage:
on("regenerate", lambda t: log_feedback(t, "regenerate"))           # dissatisfaction
on("copy",       lambda t: log_feedback(t, "copy"))                 # value
on("edit_send",  lambda t, new: log_feedback(t, "edit", correction=new))   # a CORRECTION = gold
on("abandon",    lambda t: log_feedback(t, "abandon"))              # gave up

# A 👎 is a SIGNAL, not a LABEL. Route each signal to where it belongs — and CALIBRATE the
# noisy ones with human review BEFORE they ever become training data.
def route(fb):
    if fb.signal == "edit" and fb.correction:            # user-supplied ground truth
        gold_set.add(input=fb.input, expected=fb.correction)        # an eval label + an SFT pair
    elif fb.signal in ("thumb_down", "regenerate", "abandon", "escalate"):
        review_queue.add(fb)              # a SIGNAL of trouble -> a human reads it to find the pattern
    elif fb.signal in ("copy", "return"):
        positive_set.add(fb)              # it worked -> few-shot / "good" pool

# Patterns, not individual signals: a SPIKE in a rate is the alert, not one angry click.
def watch(window):
    if rate(window, "abandon", intent="billing") > 1.5 * baseline:  # 10% -> 32% this week
        alert("drift on billing")         # -> drift (L181) + error analysis (L171)
# NEVER fine-tune on raw thumbs (Goodhart). Review-calibrate first -> THEN it becomes a dataset (L183).

That routed, calibrated stream — corrections as gold, negatives as a review queue, spikes as alerts — is your next dataset. Which is exactly the loop we close in the final lesson: turning production into a self-improving data engine (L183 (Closing the Loop: Production → Dataset)).

🧪 Try It Yourself

Turn on the feedback loop for one feature this week:

Add one implicit signal. Instrument regenerate (or copy, or edit) and log it tied to the trace (L177). It's free and covers 100% of users — far more than thumbs.
Add frictionless explicit feedback. A single inline 👍/👎 with an optional one-tap category, asked after the task. Expect a 1–3% response rate — that's normal.
Don't label — queue. Pipe every 👎 and every “regenerate” into a review queue, not your training set. Capture the full trace with each.
Calibrate. Sample 20 thumbs-downs and read them (L170). What fraction are real failures vs. user-error vs. noise? Now you know how much to trust the signal.
Route the verified ones. Confirmed failures → error-analysis taxonomy (L172) → regression tests (L175). User corrections → gold labels. A rising negative-rate → a drift alert (L181).

Do this and your users quietly build your eval set for you — and you avoid the trap of training on their noise.

Mental-Model Corrections

“A 👎 means the answer was bad.” A 👎 is a signal, not a label. Users thumbs-down correct answers (mood, they were wrong, latency). Calibrate before you trust it; never auto-label your eval set or fine-tune on raw thumbs (Goodhart).
“Thumbs up/down is my main quality metric.” It's sparse (1–3%) and biased to the extremes. Lead with implicit signals (100% coverage, objective); use explicit to calibrate and flag specific issues.
“Implicit signals are too vague to use.” Each one means something — edit = a correction (gold), regenerate = dissatisfaction, copy = value, abandon = failure. The gold is in reading them per topic, as patterns.
“One thumbs-down is a problem.” Patterns, not individuals. One angry click is noise; a rate spike (10% → 32% on a topic) is the signal.
“We collect feedback.” Collecting isn't using. Tie it to the trace, calibrate it, and route it — corrections to gold, negatives to error analysis, spikes to drift. Feedback you don't act on is just a chart.
“Feedback is for training the model.” First it's for understanding users (where they struggle). Only verified corrections are training data — and even then, review beats raw.

Key Takeaways

Two signal types: explicit (👍👎/ratings — direct but sparse ~1–3% and extreme-biased) and implicit (edit/regenerate/copy/abandon/escalate — 100% coverage, objective behavior). Lead with implicit; calibrate with explicit.
Read what each implicit signal means: edit = a correction (the highest-value, ground-truth signal), regenerate/rephrase = dissatisfaction/misunderstanding, copy/return = value/trust, abandon/escalate = failure. Monitor per topic, as patterns not individual clicks.
A 👎 is a signal, not a label. It's a great lens to understand users, a noisy learning signal. Selection bias + user-error + mood make it unreliable as ground truth — calibrate by review (L170) and never fine-tune on raw thumbs (Goodhart).
Capture it well: frictionless explicit, automatic implicit, tied to the full trace (L177); run collect → label → clean (PII) → extract; ask at the right moment.
Route into the machinery: corrections → gold labels + SFT (L173); negatives → error analysis (L171/L172) → regression tests (L175); rates → drift (L181); thumbs → a weak, calibrated online-eval signal (L180).
That calibrated, routed stream is your next dataset — which we turn into a self-improving engine next: closing the loop, production → dataset (L183 (Closing the Loop: Production → Dataset)).