Writing Judge Prompts & Rubrics

You have the idea (L166 (The Idea: Models Grading Models)) and the protocol (L167 (Pointwise vs Pairwise / Comparative Judging)). Now the part that actually decides whether your judge is trustworthy or theater: the rubric.

A judge is only as good as the rubric you hand it. Ask “is this answer good?” and the model has no choice but to fill in the blank with its own idea of good — uncalibrated, invisible, and different every run. The discipline of this lesson is turning a fuzzy quality (“be helpful,” “be faithful”) into a precise, operational rubric the judge applies the same way every time. The one-line warning that should haunt you: vague rubrics produce vague verdicts at scale. A bad rubric doesn't fail loudly — it quietly hands you thousands of confident, wrong numbers.

Hold one case in mind for the whole lesson. The source says: “Pro includes priority email support (4-hour SLA); phone is Enterprise-only.” The answer to grade says: “Yes — Pro includes priority phone support, 4-hour SLA.” It invented a channel. Now watch the rubric decide the verdict:

• Vague rubric — “Is this answer good?” → the judge shrugs, sees a confident on-topic reply, and returns “👍 4/5.” It rubber-stamped the hallucination, because “good” is a placeholder it filled with its priors.
• Precise rubric — “Faithfulness: every claim must be supported by the source; reason through each claim, then score” → the judge lists the claims, finds “phone support” isn't in the source, and returns FAIL.

Same judge, same answer, opposite verdict — decided entirely by the rubric. As one practitioner put it: “The response is high quality” is not a definition; it's a placeholder the judge fills with whatever its priors happen to be. The rubric isn't documentation around your eval — the rubric is your eval.

A reliable judge prompt has four parts — RCAF: a Role, the rubric as Context, a scoring Action, and a strict output Format — wrapped around the input/output to grade, run at temperature 0 so grading is deterministic.

Read the four roles in that prompt. Role sets the stance (“strict grader”). Context is the rubric itself — the criterion plus anchored levels. Action tells it how to decide (reason first, then score). Format pins the output to parseable JSON. And notice two non-obvious moves: the SOURCE is passed in so the judge can actually check each claim (you can't grade faithfulness without the evidence), and the comment's rule — score one criterion per call. A single prompt asked to rate faithfulness and relevance and fluency at once produces correlated, mushy scores; separate rubrics in separate calls, aggregated at the app layer, stay sharp.

The hardest, highest-leverage work is operationalizing the fuzzy word. “Helpful” is a feeling; “the response directly addresses every constraint in the user's query, including the negative ones” is a rubric. Write the criterion in your domain's language, not generic ML vocabulary — “every factual claim is explicitly supported by the retrieved context” tells the judge exactly what to look for; “high quality” tells it nothing.

The strongest criteria share four properties: specific (one clear thing), measurable (you could check it), criterion-independent (doesn't double-count what another criterion already scored), and exhaustive (together they cover the decision you actually care about). That last pair is why you decompose quality instead of asking for one global score:

• Faithfulness — does it stay inside the retrieved evidence (no inventions)?
• Relevance — does it answer what the user actually asked?
• Completeness — did it cover everything it needed to?

Score each criterion in its own call and aggregate at the application layer. The instinct to save tokens with one do-everything prompt is exactly what makes judge scores untrustworthy.

Even a perfect criterion needs a scale the judge can apply consistently — and the defaults people reach for are usually wrong.

• Use a categorical integer scale, not a float. Judges are far more consistent on “0–5” or “pass/fail” than on “0.0–1.0.” Research puts human alignment highest on a 0–5 scale, and binary pass/fail is the most robust of all — a yes/no verdict is hard to game with verbosity. Pick the coarsest scale that still captures the distinction you need.
• Anchor every level. Don't just say “score 1–5” — say what each number means: “5 = every claim supported · 3 = one unsupported claim · 1 = mostly unsupported.” Anchored descriptors are the difference between a stable ruler and a judge guessing what a “4” feels like today.
• Add one calibration example. A single worked example (“…24/7 phone line” → 2, invents a channel) stabilizes scoring like rater-training does for humans. Counterintuitively, one example beats many — models often get worse as you pile on few-shot examples, so choose one good one.

Here's the whole lesson as something you can do with your hands. The answer below hides the phone-support hallucination. Start with nothing and toggle rubric ingredients on — watch the judge prompt assemble line by line, the human-alignment meter climb, and the verdict flip from rubber-stamping the lie to catching it.

Toggle them in any order and the pattern is unmistakable. Drop the source and the judge can't check anything — it guesses “4/5.” Add a criterion but no anchors or reasoning and it gets a soft, drifting “3/5.” Add anchored levels + reason-first + the source and it does what you actually wanted: lists the claims, catches “phone support,” and returns FAIL. You didn't change the model or the answer — only the rubric — and that alone moved the verdict from wrong to right.

A few clauses in the rubric are really bias armor — leave them out and the judge quietly optimizes for the wrong thing:

• Reason first — but keep it structured. Asking the judge to “list each claim and mark supported / not, then score” (the G-Eval pattern) consistently improves agreement with humans over jumping straight to a number. The nuance: keep the reasoning a short, defined set of steps, not an open-ended essay. An unbounded rationale before a forced verdict can become post-hoc justification — the model commits early and rationalizes. Structured steps, then score.
• State length-neutrality explicitly. Judges score longer answers higher (verbosity bias). If your rubric doesn't say “a concise answer scores ≥ a verbose one at equal correctness,” it implicitly rewards length. One sentence closes the loophole.
• Pass the evidence in (reference-guided). For grounded criteria — faithfulness, correctness — give the judge the source or a gold answer in the prompt so it checks against fact, not vibes. A faithfulness rubric with no source is just the judge's imagination. (These biases — position, verbosity, self-preference — get their full calibration treatment in L169 (Judge Biases & How to Calibrate).)

Here's the humbling part, and the most important: you cannot fully write the rubric up front. Researchers studying how people build judges found a “criteria drift” phenomenon — “it is impossible to completely determine evaluation criteria prior to human judging of LLM outputs.” As you grade real data, your own understanding of “good” evolves, and you reinterpret the rubric. The rubric is discovered, not specified.

So the workflow isn't write rubric → done. It's a loop (the “who validates the validators?” problem): label a small sample by hand, run your judge on it, and read every disagreement — each one is a criterion you under-specified or a level you anchored wrong. Refine, re-run, and stop when judge↔human agreement is high.

Two cautions live in this loop. Rubric drift / Rubric-Induced Preference Drift: tiny natural-language edits to a rubric can silently shift the judge's preferences in a whole domain, so version your rubric like code and re-check agreement after every change. And the loop never fully ends — keep a path from judge disagreement → human adjudication (L170 (Human-in-the-Loop Spot Checks)). The rubric you ship is the one that provably tracks your humans (L158 (Building Your First Eval Set) is where those human labels come from), not the one that sounded right in a doc.

Take a vague rubric you (or a tutorial) already use and operationalize one criterion — 15 minutes, and the skill clicks:

1. Pick the fuzzy word. “Helpful,” “accurate,” “good tone” — whatever you're secretly asking a judge for.
2. Write the procedural definition. Turn it into something two people would grade the same way: not “is it accurate?” but “every factual claim is supported by the provided source.” In your domain's words.
3. Anchor a 0–5 (or pass/fail) scale. Say what a 5, a 3, and a 1 look like — one line each. Add one example answer with its score.
4. Grade 10 real outputs yourself, then run claude-opus-4-8 with your rubric on the same 10 (reason first, then score). Read the disagreements. Nine times out of ten the rubric was ambiguous, not the judge — fix that line and re-run.

When your judge agrees with you on those 10, you've done the actual job: you didn't write a prompt, you designed a measurement.

• “Just ask the judge if it's good.” “Good” is a placeholder the judge fills with its own priors. Write an operational criterion in your domain's words, or you're measuring the model's vibes, not your standard.
• “One prompt can score everything at once.” A do-everything prompt yields correlated, mushy scores. Score one criterion per call (faithfulness, then relevance, then completeness) and aggregate.
• “Use a 0.0–1.0 float for precision.” Judges are inconsistent on floats. Use a categorical scale — 0–5 (best human alignment) or binary (most robust, hard to game) — with anchored levels.
• “Score first, explain after.” Backwards for accuracy: reason first (list the evidence), then score (G-Eval) — but keep the reasoning structured and short, not an essay that becomes post-hoc justification.
• “My rubric is neutral.” Not unless you said so. Without an explicit length-neutrality clause it rewards verbosity; without the source passed in it can't check faithfulness at all.
• “Write the rubric once, then run it.” You discover the rubric (criteria drift). Align it to human labels, read the disagreements, and refine until it provably tracks your people.

• A judge is only as good as its rubric — vague rubrics produce vague verdicts at scale. “Is it good?” makes the judge grade by its own priors; an operational criterion makes it grade by yours.
• Structure the prompt as RCAF: Role · rubric as Context · scoring Action · strict (JSON) Format, at temperature 0, with the input/output to grade.
• Operationalize + separate the criteria: specific, measurable, criterion-independent, exhaustive; decompose into faithfulness / relevance / completeness and score one per call.
• Anchor the scale: categorical integers (0–5 best human alignment, or binary most robust), a descriptor for every level, plus one calibration example (one beats many).
• Build in the defenses: reason first but structured (G-Eval), an explicit length-neutrality clause, and pass the source in for grounded criteria. Full bias treatment in L169 (Judge Biases & How to Calibrate).
• You discover the rubric: criteria drift means you can't fully specify it up front — align to human labels, read the disagreements, refine, and version it like code. Those labels come from L158 (Building Your First Eval Set); escalation lives in L170 (Human-in-the-Loop Spot Checks).