Output Guardrails (Format, Safety, Grounding Checks)

In L231 (Input Guardrails: Validation, PII & Moderation) we guarded the front door — everything between the user pressing send and the prompt reaching the model. This lesson guards the exit door: the model has produced an answer, and the output guardrail is the layer that inspects it before the user — or a downstream system — ever sees it.

If input guardrails defend against what comes in, output guardrails defend against what the model puts out. And a language model can put out three very different kinds of bad: a response in the wrong shape (a broken JSON your code can't parse), a response that's unsafe to show (toxic, or leaking a secret), and — the subtle, dangerous one — a response that is confidently false: perfectly formatted, perfectly polite, and not supported by your sources at all.

So the output guardrail has three pillars:

• Format — is it the right shape? (schema / structured output)
• Safety — is it safe to show? (moderation + PII / secret leakage)
• Grounding — is every claim true to the source? (faithfulness)

And on a failure, the same control flow from L222 (Putting Guardrails in the Loop) applies: pass, repair, re-ask, or block. The one sentence to hold onto for the whole lesson: well-formed and inoffensive is not the same as true — and only one of these three pillars can tell the difference.

It's tempting to think a good model and a good input gate are enough. They aren't — the output is a separate attack surface and a separate quality surface:

• The model is non-deterministic. Even with a perfect prompt, it can return malformed JSON, drift off-tone, or invent a fact. You can't prompt your way to a guarantee; you verify the actual output.
• Some failures only exist on the way out. The model can emit PII it inferred, leak a secret from its context or a tool result, or regurgitate its own system prompt. None of that is visible at the input.
• Defense in depth. The input gate (L231) and the output gate are two independent layers — an attacker or a bad generation has to beat both. Re-running moderation on the output isn't redundant; it's the second net.
• Downstream systems trust your output. If the model's JSON drives a tool call, a database write, or a UI, a malformed or ungrounded output isn't an awkward chat message — it's a bug or an incident in the system that consumed it.

The mental model: the output guardrail is quality control at the end of the line. Nothing ships to the user or the next service until it has passed inspection — and the inspection is code that runs on the real bytes the model produced, not a hope encoded in the prompt.

For anything a machine consumes — a tool call, an API payload, a row to insert, a field in your UI — free-form text is a bug waiting to happen. You need the output in a guaranteed shape. There are three strategies, strongest to weakest:

1. Constrained decoding — the strong default. Instead of hoping the model emits valid JSON, you constrain the token sampler: at each step, tokens that would violate your JSON Schema (or grammar) are masked out, so an invalid output is literally impossible to generate. This is what OpenAI Structured Outputs (strict: true), Anthropic structured outputs (beta on Claude Sonnet 4.5 / Opus 4.1), and Gemini structured output do; under the hood, grammar backends like XGrammar and Outlines (the default in vLLM / SGLang / TensorRT-LLM) enforce it. Bonus: it can be faster, because the model skips generating boilerplate scaffolding tokens.
2. Validate-and-re-ask / repair — when you can't constrain decoding. Generate, then parse and validate against a schema (e.g. a Pydantic model). On failure, either re-ask — send the validation error back and regenerate — or auto-repair the output. This is the Guardrails AI loop, with on-fail actions: reask · fix · filter · refrain.
3. Prompt-and-pray — the weakest. "Respond only in JSON." Models deviate — wrapper prose, an extra field, a trailing comma. Never rely on this alone.

One important caveat, because it surprises people: over-constraining can hurt reasoning. Forcing the model into a rigid structure from the first token (the "Let Me Speak Freely?" finding) can lower answer quality, because the format fights the model's natural generation. The fix: let it reason in free text first — a reasoning or scratchpad field — then emit the structured answer. Structure the conclusion, not the thinking.

The second pillar asks: is this content safe to put in front of a human? It's the output-side twin of the moderation layer from L231 — and it does two jobs:

• Re-moderate the response. Run the same kind of classifier you ran on the input — OpenAI omni-moderation, Llama Guard 4 (which is explicitly built to classify both the prompt and the response), Azure AI Content Safety — now on what the model said. Toxicity, hate, violence, self-harm. This is defense in depth, not duplication: a benign input can still produce a harmful output (a jailbreak that beat the input gate, or the model simply going off the rails).
• Stop output-side leaks — the failure mode that only exists at the exit. The model can emit things it should never surface:
  • PII / another user's data it inferred or pulled from a tool result,
  • a secret — an API key, a connection string — that ended up in its context,
  • its own system prompt (a successful prompt-injection exfiltration).
    Scan the output for these and redact them (or block) before the bytes leave your perimeter. This is the last line where a leaked sk-live… key can still be caught.

Note the natural action split: toxic content is usually blocked (you can't safely 'fix' an abusive reply — fall back to a safe message), while a leak is usually redacted in place ([REDACTED_SECRET]) so the useful part of the answer still ships. Same pillar, different remediation — which is exactly the control-flow decision we formalize two sections down.

Here is the pillar that makes output guardrails hard, and the one most teams skip: an answer can be valid JSON, perfectly polite, and completely made up. Format and Safety both wave it through. Grounding (also called faithfulness) is the check that asks: is every claim in this answer actually supported by the source it's supposed to be based on?

The technique — the one popularized by RAGAS — is beautifully simple:

1. Decompose the answer into atomic claims (individual factual statements).
2. For each claim, run a natural-language inference (NLI) / LLM-judge check against the retrieved context: does the source entail this claim, contradict it, or say nothing?
3. Faithfulness score = (supported claims) / (total claims). Anything not entailed is flagged — and either stripped, corrected, or sent back for a re-ask.

You don't have to build this yourself — the platforms ship it:

• Azure AI Content Safety — Groundedness Detection: flags the exact ungrounded segments, with a reasoning mode that explains why, and a correction mode that rewrites the ungrounded text against your sources before the user sees it.
• Amazon Bedrock Guardrails — Contextual Grounding Check: returns grounding and relevance confidence scores (configurable thresholds, e.g. 0.7) to filter hallucinations; plus Automated Reasoning checks (GA Aug 2025) that use formal logic to give a verifiable proof a response complies with a policy — stated up to 99% verification accuracy.
• RAGAS / self-check patterns you run in-house for offline eval or as an online gate.

Two crucial framings. First, grounding is relative to the source — a claim isn't 'true' in the abstract, it's supported by the context you check against (edit the source in the lab and watch the same answer flip from ungrounded to grounded). Second, this is the gate, not the cure: detecting an ungrounded claim at the exit is the output guardrail's job; reducing how often the model produces them — retrieval, prompting, decoding, abstention — is L233 (Hallucination Mitigation), next. And it's distinct from the RAG lesson Citing Sources & Grounding Answers (the retrieval-side of grounding); here it's the verification gate on the way out.

Run the exit gate on a real (editable) model output. You're given a support agent's JSON reply and the knowledge-base source it's supposed to be grounded in. Toggle the three pillars, pick the on-violation policy, and Run the output gate — then watch the per-claim grounding verdicts and the faithfulness score:

What the presets prove:

• “Ungrounded claim” — valid JSON, friendly tone, so Format ✓ and Safety ✓ — but it promises “90 days” when the source says 30. Only Grounding catches the contradiction. Well-formed and inoffensive ≠ true.
• “Fluent but unsupported” — adds a confident “award-winning… fastest in the industry” brag with zero support in the source. The faithfulness score drops, and repair strips the invented claim and ships only the grounded sentence.
• “Leaks data” — a friendly reply that smuggles an API key and another customer's email. Safety redacts them on the way out.
• Now edit the source to say 90 days and re-run the ungrounded preset — the same answer is suddenly grounded. Truth, for a guardrail, is relative to the source you check against.

When a pillar fires, what do you do? This is the control flow from L222 (Putting Guardrails in the Loop), now with concrete output-side choices. The action should match the violation and its reversibility:

Two engineering rules wrap this:

• Bound the re-ask loop. A re-ask costs a full extra generation (latency + money), and a model that failed once can fail again — cap retries (e.g. 1–2) and fall back when you hit the cap, exactly like the retry/fallback discipline coming in L234 (Fallbacks, Retries & Circuit Breakers).
• The gate is outside the model. As in L222 and L231, the check is deterministic infrastructure wrapping the call — not a line in the system prompt asking the model to please be grounded. The thing you're verifying cannot also be the verifier.

Output guardrails collide head-on with one of the best UX wins from earlier in the course: streaming. In L225 (Streaming UX: TTFT, Token-by-Token & Markdown Buffering) we made responses feel instant by rendering tokens as they arrive. But an output guardrail wants to inspect the whole response — and you can't fully validate a response you haven't finished generating. You can't moderate, schema-check, or ground a stream mid-flight. Three ways teams resolve it:

• Validate, then stream (safest). Generate fully, run the gate, then stream the approved output. You pay the full generation time as TTFT — bad for perceived latency, but nothing unsafe is ever shown.
• Stream optimistically, retract on violation. Show tokens immediately, run the gate in parallel, and pull back (the remend idea from L225) if it fails. Great latency, but the user may briefly glimpse the bad content — only acceptable for low-severity checks.
• Layer by latency. Run cheap, incremental checks during the stream (a partial-JSON validator, a fast moderation pass on each chunk) and buffer the expensive ones (full grounding) to a final gate before the last chunk commits.

The decision is a severity call (a callback to L227 — Defensive UX: Designing for Uncertainty & Failure): the higher the cost of the user seeing a bad token, the more you lean toward validate-then-stream; the lower the stakes, the more you can stream optimistically. There's no free lunch — safety and time-to-first-token are in direct tension, and you choose the trade per surface.

Use the Output Guardrail Lab and what you've learned:

1. Load “Ungrounded claim.” Format and Safety both pass — so why is the output still wrong, and which pillar catches it?
2. With that same preset, edit the source to say 90 days and re-run. What happens, and what does it tell you about what 'grounded' means?
3. Load “Malformed JSON” with the policy on repair, then switch to re-ask. Why is re-ask the right action for broken JSON, and what would constrained decoding have done?
4. Load “Leaks data.” What does the Safety pillar do, and why is this failure invisible to an input guardrail?
5. Your answers stream token-by-token for great UX. What breaks when you add a grounding gate, and what's your trade-off?

→ (1) The JSON is valid and the tone is friendly, so Format and Safety have nothing to flag — but the reply promises “90 days” when the source says 30. Only Grounding — which decomposes the answer into claims and checks each against the source — catches the contradiction. Well-formed and inoffensive ≠ true. (2) The same answer becomes grounded, because grounding is relative to the source: a claim is 'true' only insofar as the context you check against supports it. Change the source, change the verdict. (3) You can't reliably hand-repair arbitrary broken JSON, so you re-ask (regenerate). Better still, constrained decoding would have made the break impossible — invalid tokens are masked during generation, so it never produces malformed JSON in the first place. (4) Safety redacts the leaked API secret and the other user's email before the answer ships. An input guardrail can't see this — the leak is something the model emitted, which only exists on the way out. (5) You can't fully validate a response you haven't finished generating, so a grounding gate fights streaming: either validate-then-stream (safe, but the whole generation becomes your TTFT) or stream-then-retract (fast, but the user may glimpse the bad content). The trade is safety vs. time-to-first-token, chosen by severity.

• “If the input was clean and the model is good, the output is fine.” No — the model is non-deterministic and can be malformed, unsafe, or fabricated regardless. Verify the output.
• “Output moderation just repeats the input check.” It's defense in depth — a benign input can still produce a harmful output (a jailbreak that slipped through, or the model going off-rails).
• “Valid JSON means a correct answer.” Format ≠ truth. A response can be schema-perfect and completely fabricated. That's what grounding is for.
• “Prompt the model to only return JSON.” Prompt-and-pray deviates. Use constrained decoding (valid by construction) or validate-and-re-ask.
• “Constrain every token for max structure.” Over-constraining can degrade reasoning — let it reason in free text first, then structure the conclusion.
• “Grounding is the same as the RAG citation lesson / hallucination mitigation.” Here it's the verification gate on the output; reducing hallucinations is L233, and retrieval-side grounding is the RAG lesson. Different jobs.
• “Stream the answer and validate it too.” You can't fully validate what isn't finished — safety and TTFT trade off; pick per surface by severity.

• The output guardrail is the exit door: inspect the model's answer before the user/system sees it — across Format × Safety × Grounding — because well-formed and inoffensive is not the same as true.
• Format = valid by construction: prefer constrained decoding (OpenAI / Anthropic / Gemini structured outputs, XGrammar) so invalid tokens can't be sampled; else validate-and-re-ask against a schema. Don't over-constrain — let it reason first, structure the conclusion.
• Safety = moderate + stop leaks: re-run moderation on the response (defense in depth with L231), and redact PII / secrets / system-prompt leaks the model might emit — the failure that only exists on the way out.
• Grounding = faithfulness: decompose the answer into atomic claims and NLI-check each vs the source (RAGAS, Azure Groundedness, Bedrock Contextual Grounding + Automated Reasoning). It's relative to the source, it's the gate (not the cure — that's L233 Hallucination Mitigation).
• On a violation: pass · repair · re-ask · block (L222) — match the action to the violation, bound the re-ask loop, and keep the gate outside the model.
• Mind the streaming tension (L225): you can't validate an unfinished response — safety trades against time-to-first-token, chosen per surface by severity (L227).