Red-Teaming Your AI Application

This is the capstone of the security section — and the answer to a question the last four lessons have been building toward: how do I know my defenses actually work? You've learned the threats (L239 — The OWASP LLM Top 10) and the defenses (input/output guardrails, hallucination mitigation, fallbacks, the guardrail layer, and the injection/exfil/agent hardening of L240–L242). Red-teaming is how you verify them — and find the gaps you missed.

Red-teaming is adversarial testing: you put on the attacker's hat and systematically try to break your own AI system — jailbreak it, inject it, exfiltrate from it, abuse its tools — before a real adversary or a curious user does. The name comes from military and security practice (the "red team" attacks; the "blue team" defends). The core truth it's built on: you don't actually know a guardrail works until you've tried to defeat it.

This lesson covers the practice: what red-teaming is (and how it differs from benign evals), the loop (scope → attack → measure → fix → re-test), manual vs automated red-teaming, the tools (PyRIT, Garak, Promptfoo), the metric (Attack Success Rate), and why it must be continuous. It's the lesson that turns everything you've learned from a checklist into a tested system.

Scope: this is adversarial, security/safety-focused testing — distinct from capability evals (measuring how good the model is at its job). Here we measure how hard it is to break.

Three things it's often confused with:

• vs. evals. Capability evals ask "how well does it do the task?" with benign inputs. Red-teaming asks "how badly can I make it fail?" with adversarial inputs. You need both, but they're different muscles — a model can ace its evals and be trivially jailbroken.
• vs. benchmarks. Benchmarks (HarmBench, AdvBench, JailbreakBench, AgentDojo, MLCommons AILuminate) are standardized attack sets — great for comparison and a starting library, but your app has its own system prompt, tools, data, and threat model, so generic benchmarks miss app-specific holes.
• vs. pentesting. Classic pentesting targets infrastructure (servers, networks, auth). AI red-teaming targets the model and the AI-specific behaviors — though a complete assessment does both (an LLM app is still a web app).

What it actually produces: a prioritized list of vulnerabilities (which attacks succeeded, how severe, in which category) and the fixes for them — documented in a form that satisfies a regulator (the EU AI Act mandates adversarial testing for high-risk and systemic-risk systems as part of risk management).

The mindset is the whole game: think like an attacker. Assume your system will be probed by clever, motivated, anonymous people, and go find what they'll find — first.

The methodology has stabilized into a five-phase loop — and the word loop is the point; it's never "done":

1. Scope & threat-model. What are you protecting, and from what? Use the OWASP LLM Top 10 (L239) against each stage of your app (input, retrieval, model, tools, output) to enumerate what could go wrong. Define what "failure" means for your app before you write a single attack.
2. Attack. Run a battery of attacks across the categories — jailbreaks, direct & indirect injection, PII extraction, system-prompt leakage, data exfiltration, tool abuse, harmful content, induced hallucination, resource exhaustion. Start from a benchmark library, then craft app-specific ones.
3. Measure. Score what succeeded. The headline metric is Attack Success Rate (ASR) — the fraction of attacks that landed — broken down by category (so you know where you're weak), often judged by an LLM-as-judge (e.g. Llama Guard) plus human review.
4. Fix. Close each gap with the right control from this course — an input/output guardrail (L231/L232), least privilege & egress locks (L241), agent hardening (L242), the whole guardrail layer (L238).
5. Re-test. Re-run the campaign to confirm the fix worked and didn't regress anything — then turn the attack into a permanent test (next section).

Watch ASR fall across rounds in the lab: an undefended app is ~100%; each control you add closes a category and drops ASR. You drive it down, round after round — but, crucially, not to zero (more on that below).

You need both kinds of red-teaming, because they catch different things:

• Manual red-teaming — humans (ideally including domain experts and people unlike your team) bring creativity: novel jailbreaks, multi-turn social engineering, and the weird app-specific edge cases an automated scanner never imagines. Irreplaceable for discovering new attack classes.
• Automated red-teaming — gives you scale, coverage, and regression: thousands of attacks on every release, in CI. The 2025–26 shift is autonomous red-teaming: an attacker LLM is given an objective, then selects, composes, and evolves attacks against the target while an LLM judge scores success (crescendo, tree-of-attacks). It now solves many black-box challenges faster than humans.

The tooling is layered and complementary:

Practical layering: a broad scan on every build (Garak/Promptfoo), a compliance scan against the OWASP categories per PR, and deep human + PyRIT campaigns on a slower cadence. The cheap automated layers catch regressions; the expensive human layer finds the novel stuff.

Be the red team. Set your app's defense posture, run the campaign, read the ASR, then fix the gaps and re-run — and watch the score fall across rounds:

What you just did is the job: attack → measure ASR → fix → re-test, repeated. Notice that each surviving attack points at exactly one missing control (an exfil that lands → you need the egress lock; a tool-abuse that lands → least privilege). And notice the round history — that downward ASR trend is the red-team flywheel, which the next section makes permanent.

The single biggest mistake is treating red-teaming as a one-time, pre-launch milestone. It isn't, for a reason unique to AI systems: your system changes underneath you. A model version bump, a tweaked system prompt, a new tool or MCP server, an updated RAG corpus — any of these can silently reopen a vulnerability you'd closed. A red-team report from last quarter is stale.

So the teams shipping trustworthy AI treat adversarial testing as a continuous engineering discipline, powered by the red-team → regression-test flywheel:

1. Red-teaming finds a vulnerability.
2. You fix it (add the guardrail / control).
3. You convert the attack payload into a permanent automated test — a regression case.
4. That test runs in CI on every release, so the vuln can never silently come back.

Every finding becomes a permanent test asset, and your suite grows stronger over time. Layer on production monitoring (the L223 observability + L237 audit signals — a spike in blocked attacks or a new jailbreak pattern in the wild is live red-team intel) and a bug-bounty / responsible-disclosure program, and you have continuous coverage.

And the humbling truth the lab hints at: you never reach 0% ASR in reality. New attacks are invented constantly, and a probabilistic model can always be coaxed. The goal isn't a solved system — it's a managed, monitored, continuously-tested one whose ASR you drive down and keep down. Security is a process, not a state.

Use the Red-Team Lab and what you've learned:

1. Run the campaign with no defenses. What's the ASR, and what does that tell you about an undefended app?
2. Turn on only the input guardrail + injection classifier and re-run. Which attacks still land, and why do they need different controls?
3. Drive ASR to 0% in the lab. Why does the lab let you hit zero — and why won't reality?
4. Why is red-teaming continuous rather than a one-time pre-launch check? Give a concrete trigger that reopens a closed vulnerability.
5. You find a novel jailbreak in a manual session. What are the next two steps to make sure it never works again?

→ (1) 100% ASR — every attack lands. An undefended AI app fails every category; the model alone is not a security boundary. (2) Jailbreak and harmful-content attacks get blocked, but exfil, tool abuse, and DoS still land — they live at the output/egress, tool, and rate-limit layers, so input controls can't touch them. Different attack → different control. (3) The lab models clean attack→control mappings, so full coverage = 0%. Reality won't, because new attacks are invented constantly and a probabilistic model can always be coaxed — you manage ASR, you don't eliminate it. (4) Because the system changes underneath you: a model-version bump, a system-prompt edit, or a new tool/MCP server can silently reopen a closed hole — so a one-time report goes stale. (5) Fix it (add the control that blocks it), then convert the attack into a permanent regression test in CI so it's caught automatically forever — the red-team flywheel.

• “My evals pass, so it's secure.” Evals measure capability on benign inputs; red-teaming measures resistance to adversarial ones. A model can ace evals and be trivially jailbroken.
• “I ran a benchmark, so I'm covered.” Benchmarks are a generic starting library; they miss your app's system prompt, tools, data, and threat model. Add app-specific attacks.
• “Red-teaming is a one-time pre-launch task.” It's continuous — model updates, prompt changes, and new tools reopen vulnerabilities. Run it in CI, every release.
• “Automated tools are enough.” They give scale & regression but miss novel attacks; manual human creativity (and domain experts) finds the new classes. Use both.
• “The goal is 0% attack success.” You can't reach zero against a probabilistic model and an evolving threat landscape. The goal is managed, monitored, driven-down ASR.
• “A finding is fixed once I patch it.” A patch without a regression test can silently regress. Every finding becomes a permanent test.
• “Red-teaming is the security team's job.” Engineers run automated red-teaming in CI; it's a shared, continuous discipline, like testing itself.

• Red-teaming = adversarially attacking your own app to find vulnerabilities before adversaries do — you don't know a guardrail works until you've tried to break it. It's distinct from benign capability evals.
• The loop: scope (threat-model with the OWASP Top 10, L239) → attack (a battery across the categories) → measure (Attack Success Rate, by category) → fix (the right control from the course) → re-test. It's a loop — never "done."
• Manual + automated: humans find novel attacks; automation gives scale & regression (an attacker-LLM evolves attacks, a judge scores ASR). Tools layer up — Garak (broad scan), Promptfoo (CI regression), PyRIT (deep multi-turn).
• It's continuous: model/prompt/tool changes reopen gaps, so make it a discipline — the flywheel turns every finding into a permanent regression test in CI, backed by production monitoring (L223/L237) and bug bounties. The EU AI Act requires adversarial testing.
• You never reach 0% ASR — security is a managed process, not a state. This closes the section: you learned the threats (L239) and the defenses (L240–L242, plus the guardrails of L231–L238); red-teaming is how you prove they hold — and keep proving it.