Putting Guardrails in the Loop

Introduction

We've wired the pipeline (L221). Now we make it safe to put in front of real users — by wrapping the model in guardrails. From L218 you know the shape: an input gate before the model and an output gate after it. This lesson is about getting that placement and its control flow right — because guardrails aren't a box you bolt on; they're a loop the request flows through, and every decision in that loop is an engineering trade-off.

The one-line mental model: guardrails wrap the model in a control loop — the input gate decides should this even reach the model? (pass / mask / block), and the output gate decides is this safe to return? (pass / retry / fall back). The art is what each gate does on a violation, and what it costs.

In this lesson:

Runtime controls, not evals — what guardrails actually are, and their two failure modes
The two checkpoints — input gate and output gate, and where they sit
The guardrail loop — the policy actions on a violation (block, mask, retry, fallback)
Fail-open vs fail-closed — the critical default when a guard itself fails
The latency tax & the streaming conflict — the real costs, and how to keep them down

Scope — important: this is the placement / control-flow view. The deep detection methods — how you actually detect PII, jailbreaks, hallucinations; content moderation; the full fallback/retry/circuit-breaker treatment; governance, compliance (EU AI Act, NIST), audit logs, and a hands-on build — are a whole later section: "Guardrails, Safety, Reliability & Governance." Here we place the gates in the loop and reason about the flow; we defer the how of each check to that section. (And the gates live in the gateway (L219) / orchestration layer (L221).)

Infographic titled 'Putting Guardrails in the Loop' — the safety layer of a production LLM application. The big idea: guardrails wrap the model in a control LOOP — an INPUT gate before the model and an OUTPUT gate after it. The flow, left to right: a request hits the INPUT GATE, which runs fast checks (PII, prompt injection / jailbreak, topic limits) and takes a policy ACTION — PASS it through, MASK / redact sensitive spans and continue, or BLOCK it outright so the model is never called. If it passes, the MODEL generates, and the response hits the OUTPUT GATE, which checks quality, safety, grounding and format and again takes an action — PASS and return, RETRY (regenerate, up to a cap), or FALL BACK to a safe templated or retrieval-only response. That retry-then-fallback path is the loop: output fails → regenerate → still fails → fallback. THE KEY POINTS. Guardrails are RUNTIME CONTROLS, not evals: they enforce policy on LIVE traffic in milliseconds, on every request, and they fail two opposite ways — too LOOSE (harmful content passes) or too STRICT (legitimate requests get blocked, false positives). FAIL-OPEN vs FAIL-CLOSED is the critical default: when a guard itself errors or times out, do you let the request through unchecked (fail-open — availability over safety) or block it (fail-closed — safety over availability)? High-stakes apps fail closed. THE LATENCY TAX: every gate adds milliseconds (input rail budget under about 100ms, output under about 150ms), and a single output RETRY adds a whole extra generation — 1.5 to 3 seconds — so retries need a hard CAP plus a fallback. THE STREAMING CONFLICT: blocking mode vets the full response before showing anything (safe, slower first token); streaming mode shows tokens as they decode (fast) but you cannot fully vet what you have already shown — a real safety-versus-speed trade. Build for SPEED: run cheap deterministic SCANNERS before expensive LLM CLASSIFIERS, run independent guards in PARALLEL, and keep the heavy checks off the critical path. The deep detection methods, content moderation, governance and the hands-on build live in the later Guardrails, Safety, Reliability and Governance section; this lesson is about PLACEMENT and CONTROL FLOW. Roadmap strip: reference architecture (L218), gateway and router (L219), adding caches (L220), orchestration (L221), guardrails (L222), observability (L223). Takeaway banner: guardrails wrap the model in a control loop — an input gate that blocks or masks and an output gate that retries or falls back — as runtime controls that cost latency, so cap retries, decide fail-open versus fail-closed deliberately, and run cheap checks first and in parallel.

Guardrails Are Runtime Controls, Not Evals

First, a distinction that prevents a lot of confusion. An eval runs offline, on a test set, to measure quality (Container-wide topic; observability is L223). A guardrail runs online, on live traffic, in milliseconds, and acts — it lets a request through, changes it, or stops it. Same spirit ("is this good?"), totally different job: an eval reports; a guardrail enforces.

Because a guardrail acts on every live request, it can be wrong in two opposite ways — and they look nothing alike:

Too loose (false negative) → harmful content passes. A jailbreak slips through, PII leaks, a toxic or hallucinated answer reaches the user. The failure is a safety incident.
Too strict (false positive) → legitimate requests get blocked. A valid medical question is refused, a normal output is held back. The failure is a broken product and frustrated users.

You can't maximize both — tightening a guard to catch more bad traffic also blocks more good traffic. So a guardrail isn't "on/off"; it's a policy with a tunable strictness, and you choose the operating point by how costly each mistake is for your app. A medical or financial app tunes toward safety (accept more false positives); a casual creative tool tunes toward availability (accept more false negatives). Everything else in this lesson follows from that tension.

The Two Checkpoints — Input Gate & Output Gate

Guardrails sit at two main points in the loop, and they protect different parties:

Input gate (before the model) — protects the system and the org. It inspects the incoming request and stops bad things from reaching the model:

Sensitive-data leaks — detect and mask PII / secrets before they're sent to a third-party model (the Samsung-leak class of problem from L218).
Prompt injection / jailbreaks — catch attempts to override instructions or trigger unsafe actions.
Scope / topic limits — reject out-of-bounds requests with a stock response (don't even pay for a model call).

Output gate (after the model) — protects the user and the org. It inspects the generated response before it's returned:

Safety / toxicity / brand risk — is this response harmful or off-brand?
Grounding / hallucination — is it supported by the retrieved context, or made up?
Format / structure — is the JSON valid, the schema satisfied? (Often enforced, not just checked.)

In a fuller system there are more rails than two — a retrieval rail (don't retrieve poisoned docs), a tool-call rail (don't run a dangerous action — critical for the write actions from L218). But the input gate / output gate pair is the backbone, and the rest are variations on the same idea: a checkpoint, a policy, an action. What that action is — next.

The Guardrail Loop — What Happens on a Violation

A guardrail that can only block is a blunt instrument. Real gates pick from a menu of policy actions, matched to how severe the violation is:

PASS — clean; continue.
MASK / REDACT — the content is usable once the bad part is removed: strip the PII (reversibly) and continue. (Input gate's gentlest action.)
BLOCK — reject the request or cut off the stream; return a safe refusal. (For jailbreaks, hard policy violations.)
RETRY / REGENERATE — the output failed a check, so call the model again (maybe with a stricter prompt) for a better answer. The defining move of the output loop.
REDIRECT / FALLBACK — give up on the model for this request and return a safe alternative: a retrieval-only answer, a cheaper model, a templated response, or a human handoff.
ALERT — log/notify without blocking (a soft guardrail for monitoring).

The output gate strings these into a loop: check → fail → retry → check → fail → fallback. In code:

async def guarded(query: str) -> str:
    # ── INPUT GATE — run cheap guards in PARALLEL; ALL must pass ──────────────
    verdicts = await asyncio.gather(pii_scan(query), jailbreak_scan(query), topic_scan(query))
    if any(v.block for v in verdicts):
        return safe_refusal()                  # BLOCK — the model is never called
    query = mask_pii(query, verdicts)          # MASK — redact, then continue

    # ── THE OUTPUT LOOP — generate, check, RETRY up to a cap, else FALL BACK ──
    for attempt in range(MAX_RETRIES):         # ← the CAP is non-negotiable (see Latency Tax)
        out = await model(query)
        if output_gate(out).ok:                # safety · grounding · format all pass?
            return out                         # PASS — return it
        # else: unsafe/hallucinated/malformed → loop and regenerate
    return fallback_response(query)            # FALLBACK — a safe, templated/retrieval answer

Two things to notice in that code — and they're the whole rest of the lesson. (1) The input guards run asyncio.gather — in parallel (L221), so the gate costs its slowest check, not their sum. (2) The retry loop has a hard cap and a fallback — because, as the next sections show, an uncapped retry loop is a latency disaster, and a guard that errors needs a defined default.

Fail-Open vs Fail-Closed — The Default That Matters Most

Here's a question most teams don't ask until it bites them: what happens when the guardrail itself fails? Your PII scanner times out, your safety classifier 500s, the moderation API is down. The request is mid-flight and the guard can't give a verdict. You must have a default, and there are only two:

Fail-closed → block the request (or don't return the output). You chose safety over availability: better to refuse a good request than risk letting a bad one through unchecked.
Fail-open → let it through unchecked. You chose availability over safety: better to serve the request than to go down because a guard is flaky.

Neither is universally right — it's a per-app, per-rail decision driven by stakes:

High-stakes (health, finance, anything with write actions or legal exposure) → fail-closed. A wrong answer or a leak costs far more than a blocked request.
Low-stakes / availability-critical (a casual assistant where downtime is the bigger harm) → fail-open may be acceptable — but alert loudly so you know you're running unguarded.

The trap is having no decision — a guard that throws an unhandled exception fails however your code happens to catch it, which is usually the worst of both worlds (a 500 and no safety). Decide the default for each gate on purpose, and make fail-open states loud and monitored. The lab lets you flip this and watch a guard-timeout become either a block or an unchecked pass.

The Latency Tax & The Streaming Conflict

Guardrails are runtime controls, which means they sit on the critical path — every one you add makes every request slower. Treat latency as a budget:

Input rail: keep it under ~100ms (it gates every request). Use cheap deterministic scanners first (regex/rules for secrets, PII, known-jailbreak patterns — single-digit ms) and reserve an LLM-based classifier for the hard 1-3% of traffic.
Output rail: budget ~150ms for the check — but the retry is the killer. Each regeneration is a whole extra model call (1.5-3s). A real report: ~8% of responses failed the first check, and with retries that meant 5+ second responses on 1 in 12 messages. This is why the loop needs a hard cap (2-3 retries) and a fast fallback — never an open-ended "keep trying."

The streaming conflict (the sharpest trade in the lesson). From L212, streaming tokens as they generate is the biggest perceived-latency win — but it fights the output gate:

Blocking mode — generate the full response, run the output gate, then show it. Safe (nothing unvetted reaches the user) but you lose streaming (slow first token).
Streaming mode — show tokens as they decode. Fast, but you can't fully vet what you've already shown — by the time a guard flags token 200, the user has seen tokens 1-199. The mitigations are partial: moderate in chunks with a small look-back buffer, and be ready to cut the stream mid-flight on a violation.

The throughline: safety costs latency, and you buy it back with engineering — run cheap checks first, run independent guards in parallel (L221), keep heavy classifiers off the critical path where you can, cap retries, and choose streaming-vs-blocking by how much an unvetted token actually costs you.

See It — The Guardrail Loop

Send different requests through the loop — clean, PII, jailbreak, unsafe output, persistent-bad — and flip fail-open vs fail-closed. Watch each gate's action and the latency it costs:

Send different requests through the guardrail loop and watch the gates act. A **clean** request sails through (but still pays the guardrail latency tax). **PII in the prompt** → the input gate **masks** it and continues. A **jailbreak** → the input gate **blocks** it and the model is never called. An **unsafe output** → the output gate **retries** (a regeneration fixes it — at ~2× latency). **Persistent bad output** → retries hit the cap and the loop **falls back** to a safe response. And flip **fail-open vs fail-closed** to see what happens when a guard itself errors: pass it through unchecked (availability) or block it (safety). Watch the **outcome** and the **latency** — every check, and especially every retry, costs. (Illustrative latencies.)

Six scenarios, six behaviors — and that's the point: a violation isn't one thing. A jailbreak is blocked before the model runs (cheap, safe); PII is masked and continues; an unsafe output is retried (safe, but ~2× latency); a persistently-bad output hits the cap and falls back (safe, but two wasted generations); and a guard that errors becomes a block or an unchecked pass depending on your fail-open/closed default. Notice the latency tile: even the clean path pays a guardrail tax, and every retry is a whole extra generation. Safety is never free — the skill is making it cheap.

🧪 Try It Yourself

Reason through these, then confirm with the Guardrail Lab:

Predict: in the lab, which scenario never calls the model at all, and why is that the cheapest and safest outcome?
Why does the input gate mask PII but block a jailbreak — why not treat them the same?
The "persistent bad output" scenario costs the most latency. What two design choices keep that from becoming an infinite, runaway cost?
Your safety classifier API goes down for 10 minutes. With fail-open, what's the risk? With fail-closed, what's the cost? Which would you pick for a mental-health chatbot?
You want to stream responses for speed, but the output gate needs the full answer to judge safety. What's the conflict, and what's a partial mitigation?

→ (1) Jailbreak — the input gate blocks it, so the model is never called: no token cost, and nothing unsafe is ever generated. Catching bad input early is the cheapest safety you can buy. (2) Severity + usability: PII can be removed and the request is still valid → mask and continue; a jailbreak's intent is the violation → there's nothing to salvage, so block. Match the action to the violation. (3) A hard retry cap (2-3) and a fallback — so after N failed regenerations you serve a safe templated/retrieval answer instead of looping forever. (4) Fail-open risks serving unvetted, possibly harmful responses for 10 minutes; fail-closed blocks/degrades legitimate users for 10 minutes. For a mental-health app the harm of a bad answer is severe → fail-closed (and alert). (5) Streaming shows tokens before the gate can vet the whole output, so unsafe content can reach the user mid-stream. Mitigation: moderate in chunks with a look-back buffer and cut the stream on a violation — partial, not perfect.

Mental-Model Corrections

"A guardrail is an eval." No — an eval measures offline; a guardrail enforces online, on live traffic, in ms, and acts (pass/mask/block/retry/fallback).
"Guardrails just block bad stuff." Blocking is one action. The menu is pass · mask/redact · block · retry · fallback · alert — matched to the violation's severity.
"Tighter is always safer." Tightening catches more bad traffic and blocks more good traffic. It's a false-positive vs false-negative trade; pick the operating point by your stakes.
"One guardrail, in one place." It's a loop with two gates — input (before the model) and output (after) — protecting different parties, plus retrieval/tool rails.
"Retry until it's safe." Cap the retries and add a fallback — each retry is a whole extra generation (1.5-3s); uncapped retries are a latency/cost disaster.
"Guards never fail." They do (timeouts, API outages). Decide fail-open vs fail-closed on purpose per gate; high-stakes → fail-closed; make fail-open states loud.
"Just stream everything." Streaming fights the output gate — you can't vet tokens you've already shown. Choose streaming-vs-blocking by how costly an unvetted token is; mitigate with chunked checks + stream cut-off.
"Guardrails are free safety." They're runtime controls on the critical path — they cost latency. Make it cheap: scanners before classifiers, run in parallel, heavy checks off the hot path.

Key Takeaways

Guardrails wrap the model in a control loop — an input gate (before: PII-mask / jailbreak-block / topic) and an output gate (after: safety / grounding / format → retry / fallback). They're runtime controls, not evals: live, in ms, on every request — and they act.
Two failure modes, opposite costs: too loose (harm passes) vs too strict (legit blocked). You can't max both — tune the operating point to your stakes.
The policy-action menu: pass · mask/redact · block · retry/regenerate · redirect/fallback · alert — match the action to the violation. The output gate is a retry-then-fallback loop.
Fail-open vs fail-closed is the default that matters most when a guard itself errors: pass unchecked (availability) or block (safety). High-stakes → fail-closed, and make fail-open states loud.
Mind the latency tax: input rail <~100ms, output <~150ms, and each retry is a whole extra generation (1.5-3s) → cap retries + fallback. Keep it cheap with scanners-before-classifiers and parallel guards (L221).
The streaming conflict: blocking = safe but slow first token; streaming = fast but you can't fully vet what you've shown. Mitigate with chunked checks + a buffer + stream cut-off.
Deferred to the Guardrails, Safety, Reliability & Governance section: how to detect PII / jailbreaks / hallucinations, content moderation, deep fallbacks, governance & compliance, audit, and a hands-on build.
Next — L223: The Monitoring & Observability Layer. The last piece — you can't operate (or tune these guardrails) without seeing what your system is actually doing: metrics, logs, and traces.