Contextual Retrieval & Metadata Enrichment (Anthropic, Late Chunking)

Introduction

Through this whole section we've been improving how we cut documents — but every method so far shares one stubborn weakness, and it's the deepest chunking problem of all: the moment you split a chunk out of its document, the chunk forgets where it came from. A perfect, clean chunk that says "It expires after 30 days" is useless if nothing in it says what expires.

This lesson is about giving each chunk its context back — and there are three techniques, falling on two distinct axes:

Change the text you embed — metadata enrichment (cheap, do-first) and Contextual Retrieval (Anthropic's LLM-written context blurbs).
Change when you pool — late chunking (Jina), which embeds the whole document first, then splits.

These are some of the highest-leverage upgrades in all of RAG (recall L92's ROI ladder put them near the top). You'll learn exactly how each works, the real numbers, and — crucially — when you don't need any of them at all.

The Missing-Context Problem

Here's the failure that no chunk-size or chunk-boundary trick can fix. Consider a chunk pulled from a refund policy:

"It expires after 30 days."

To a human reading the whole document, this is obviously about enterprise refunds. But as an isolated chunk, its embedding has no idea. There's no "refund", no "enterprise", no "policy" — so when a user asks "how long is the enterprise refund window?", this chunk's vector sits far from the query and never gets retrieved. The answer was right there and the system missed it.

This happens constantly in real documents, because writing is full of context that lives outside the sentence:

Pronouns & anaphora — "it", "they", "the city", "this approach" point to something defined paragraphs earlier.
Implicit subjects — a financial table row says "+3.2%" with the company and quarter established in a heading far above.
Section-dependent meaning — "the limit is 10" means different things under "Rate Limits" vs "Refund Caps".

Naive RAG embeds each chunk in isolation, so all of that context is lost. The three techniques in this lesson each restore it — by a different route.

Fix 1 — Metadata Enrichment (Cheapest, Do This First)

Before anything clever, do the highest-ROI thing in RAG: prepend the chunk's structural metadata to its text before embedding. The section/header path, document title, and date carry exactly the context the bare chunk is missing.

# Highest-ROI, near-free: prepend structural metadata to the chunk text BEFORE embedding.
enriched = f"[{doc_title} › {section_path}] (updated {date})\n{chunk_text}"
embedding = embed(enriched)        # the chunk now carries its title, section path, and recency
# A query like "enterprise refund window" can now match a chunk whose body never says either word.

That's it — a few lines, no model calls. Now "It expires after 30 days" embeds as "[Refund Policy › Enterprise plans] It expires after 30 days", and the "enterprise refund" query lands right on it. This is why capturing metadata at ingestion time (L86) mattered so much — you're cashing in that investment here.

Practitioners consistently rank prepending the header path one of the best effort-to-payoff changes you can make. Always do this first; only then consider the heavier techniques below.

An infographic titled 'Give Each Chunk Its Context Back'. A red problem strip: a chunk, split from its document, forgets its context — the chunk 'It expires after 30 days.' never says what expires or whose policy, so the embedding of a context-stripped chunk matches poorly and gets missed. Two axes follow. Axis one, Change the TEXT you embed, has two methods. Metadata enrichment: prepend the header path, title, and date to the chunk text before embedding, for example bracket Refund Policy then Enterprise; it is the highest-ROI, near-free fix and you should do it first. Contextual Retrieval (Anthropic): an LLM writes a chunk-specific context blurb such as 'This passage from the Q3 report...' and prepends it before both embedding and BM25; it cuts retrieval failures by 35 percent (contextual embeddings), 49 percent (plus contextual BM25), and 67 percent (plus reranking), at about $1.02 per million tokens with prompt caching. Axis two, Change WHEN you pool, has one method. Late Chunking (Jina): do not touch the text — embed the whole document's tokens first with a long-context model so every token attends to the full document, then split and mean-pool per chunk, so each vector is conditioned on full-document context and anaphora like 'it' or 'the city' survive; about a 3.5 percent relative gain on BeIR, around 30 lines with no retraining, needs a long-context mean-pooling model, and is capped by the context window with benefit growing with document length. A note: corpus under about 200k tokens (about 500 pages)? Anthropic says skip retrieval entirely and put the whole corpus in the prompt, since prompt caching makes it cheap. A banner: a split chunk forgets its document — give it back its context; start with metadata (free), add late chunking or contextual retrieval, then rerank; they stack.

Fix 2 — Contextual Retrieval (Anthropic)

Metadata gives you structural context. Contextual Retrieval gives you semantic context — and it's Anthropic's headline technique for this problem. The idea: for each chunk, ask a cheap LLM to write a short, chunk-specific blurb situating it in the document, then prepend that blurb before embedding and before BM25 indexing.

# Anthropic Contextual Retrieval: an LLM writes a 1–2 sentence, chunk-specific
# context blurb, which you PREPEND before embedding AND before BM25 indexing.

def contextualize(doc, chunk):
    msg = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=100,
        messages=[{"role": "user", "content": [
            # cache the BIG document once → you don't re-pay for it on every chunk
            {"type": "text", "text": f"<document>{doc}</document>",
             "cache_control": {"type": "ephemeral"}},
            {"type": "text", "text": f"<chunk>{chunk}</chunk>\n"
                "Give a short context situating this chunk in the document, to aid search. "
                "Answer with ONLY the context."},
        ]}],
    )
    context = msg.content[0].text
    return f"{context}\n\n{chunk}"      # ← embed THIS and index THIS for BM25 (not the bare chunk)

# Prompt caching is what makes this affordable: ~$1.02 per MILLION document tokens.

So the bare chunk "It expires after 30 days" becomes "This passage from the Refund Policy explains that Enterprise-plan refunds expire 30 days after purchase. It expires after 30 days." — a vector (and a keyword index entry) that's now richly matchable.

The numbers are striking (Anthropic measured reduction in retrieval failures, top-20):

Contextual Embeddings alone: −35%
+ Contextual BM25 (hybrid): −49%
+ Reranking: −67%

Two things make it practical. First, it improves both the dense (embedding) and sparse (BM25) sides — a preview of hybrid search (next section). Second, the obvious objection — "an LLM call per chunk is expensive!" — is solved by prompt caching: you load the full document into the cache once and reference it for every chunk, bringing the one-time cost to ~$1.02 per million document tokens. Cheap enough to run over a whole corpus.

See It: Context Injection

This is the whole idea in one widget. The query and the underlying chunk never change — only the context you give the chunk before embedding. Toggle from naive → metadata → LLM context and watch the match score climb past the retrieval threshold, flipping a miss into a hit:

Same chunk, same query — toggle the enrichment level (naive → + metadata → + LLM context) and watch what gets embedded change and the match score climb past the retrieval threshold.

Notice the progression: the bare chunk misses (its vector lacks the query's concepts); metadata pushes it to the edge; the LLM context blurb pushes it comfortably over. Same chunk, same query — context is the entire difference.

Fix 3 — Late Chunking (a Different Axis Entirely)

Metadata and Contextual Retrieval both change the text you embed. Late Chunking (Jina AI) does something cleverer — it changes when you pool, leaving the text untouched.

Recall how embeddings work: a transformer produces a contextual vector for every token (each token 'sees' its neighbors via attention), then those token vectors are pooled (usually mean-pooled) into one chunk vector. Naive chunking splits the document first, so each chunk is encoded blind to the rest of the document. Late chunking flips the order:

# Late Chunking (Jina): don't change the text — change WHEN you pool.
# 1) Encode the WHOLE document's tokens at once (long-context model) → every token
#    embedding is contextualized by the entire document.
token_embeddings = model.encode_tokens(full_document)     # e.g. up to 8192 tokens

# 2) Decide chunk spans (sentences, fixed, recursive — any boundary you like).
spans = chunk_boundaries(full_document)

# 3) Mean-pool each chunk's tokens AFTER encoding → each chunk vector "knows" its neighbors.
chunk_vectors = [token_embeddings[s:e].mean(axis=0) for (s, e) in spans]

# Naive chunking pools BEFORE the model sees the rest of the doc; late chunking pools AFTER.
# Result: an anaphor like "It" / "the city" still points at the right entity in vector space.

Encode the whole document's tokens first (so every token's vector is shaped by the entire document via attention), then split and mean-pool per chunk. Each chunk vector is now conditioned on full-document context — so an anaphor like "It" or "the city" still carries the meaning of what it referred to, even though that antecedent is in a different chunk.

The trade-offs are honest and specific:

Cheap & simple: ~30 lines, no extra LLM calls, no retraining (unlike Contextual Retrieval).
Requires a long-context, mean-pooling embedding model (e.g. jina-embeddings-v2/v3) — it doesn't work with API embedding endpoints that only take short text or use CLS pooling.
Capped by the context window (~8192 tokens): the whole document must fit in one encode, so the benefit grows with document length up to that cap.
Modest but consistent gains: Jina reports ~3.5% relative (≈1.8% absolute) improvement on BeIR (e.g. NFCorpus nDCG@10 23.46 → 29.98, SciFact 64.20 → 66.10), and it almost always beats its naive counterpart.

The Two Axes — and How They Stack

Keep the mental map clean. This lesson's techniques restore context by different mechanisms, and they're distinct from the previous lesson:

Technique	What it changes	Cost	Needs
Metadata enrichment	the text (prepend structure)	~free	metadata from ingestion
Contextual Retrieval	the text (prepend LLM blurb)	LLM call/chunk (cache it)	an LLM + prompt caching
Late Chunking	when you pool (embed→split)	~free	long-context mean-pooling model
(L93) Small-to-big	what you return	~free	a docstore

Because they operate on different parts of the pipeline, they stack. A strong 2026 stack might do: metadata enrichment (always) + late chunking or Contextual Retrieval (give vectors their context) + hybrid search + reranking (next section) — and Anthropic's own −67% number is the stack (contextual embeddings + contextual BM25 + reranking). None of these is the whole answer; each is a layer.

The Honest Take: Do You Even Need This?

Powerful as these are, the expert move is knowing when to reach for them — and when not to.

First, the question that can delete the whole problem: if your entire knowledge base is under ~200k tokens (~500 pages), Anthropic's own advice is to skip retrieval altogether — put the whole corpus in the prompt and let prompt caching make it cheap and fast. No chunking, no embeddings, no missing-context problem. Don't build RAG you don't need.

If you do need RAG, order by ROI:

Metadata enrichment — always. Near-free, high payoff.
Late chunking — if your embedding model supports it (long-context, mean-pooling), it's a cheap, no-retraining win — especially for longer documents.
Contextual Retrieval — the most powerful for the missing-context problem, but it costs an LLM call per chunk (mitigated, not eliminated, by caching) and adds index-time complexity. Worth it when retrieval quality is critical and chunks are genuinely context-dependent.
Then reranking (next section) — it compounds with all of the above.

And as always: measure on your retrieval eval. These techniques have real, published gains, but the lift on your corpus depends on how context-dependent your chunks actually are. A clean, self-contained FAQ needs none of this; a dense technical manual full of cross-references needs all of it.

🧪 Try It Yourself

Drive the widget, then decide like an architect:

In the interactive, step naive → metadata → contextual. At which step does the chunk first cross the retrieval threshold? Why does the score rise even though the original sentence never changes?
Your whole internal wiki is ~120k tokens. A teammate is scoping a Contextual-Retrieval pipeline. What do you suggest instead?
You want better context-awareness but can't afford an LLM call per chunk and don't want to re-embed via a new API. Your embedding model is long-context and mean-pooling. Which technique fits?
A chunk reads "The figure rose to 12% in the third quarter." and your "ACME 2024 revenue growth" query misses it. Give a near-free first fix and a stronger second fix.
Contextual Retrieval improves both your embedding search and your keyword search. Why both — and what does that foreshadow?

→ (1) Usually at + metadata it nears/crosses, and + contextual clears it comfortably; the score rises because the embedded text gains the query's concepts ("refund", "enterprise") even though the bare sentence is unchanged. (2) Skip RAG — 120k < ~200k tokens, so put the whole wiki in the prompt with caching. (3) Late chunking — no extra LLM calls, no retraining, and your model supports it. (4) Near-free: metadata enrichment (prepend "[ACME 2024 Annual Report › Revenue]"); stronger: Contextual Retrieval (LLM blurb naming ACME, 2024, revenue). (5) Because it prepends context to the text used by both the dense embedding and the BM25 index — foreshadowing hybrid search (dense + sparse), the next section.

Mental-Model Corrections

"A clean chunk is a complete chunk." No — a chunk split from its document loses its context (pronouns, implicit subjects, section meaning). Good boundaries don't fix this; restoring context does.
"Contextual Retrieval and late chunking are the same idea." Different axes: Contextual Retrieval (and metadata) change the text you embed; late chunking changes when you pool (embed the whole doc, then split). Same goal, different mechanism.
"An LLM call per chunk is too expensive." Prompt caching loads the document once and reuses it across chunks → ~$1.02/M tokens. Affordable at corpus scale.
"Late chunking works with any embedding model." It needs a long-context, mean-pooling model and is capped by the context window — it won't work on short-text / CLS-pooling API endpoints.
"More context techniques = always better." If your corpus is < ~200k tokens, skip RAG entirely. And on self-contained text, these add cost for little gain — measure.
"This replaces good chunking / small-to-big." It stacks with them — they change what you return; this changes what each vector knows. Layers, not substitutes.

Key Takeaways

The deepest chunking problem: a chunk split from its document loses its context ("It expires after 30 days" — what expires?), so its embedding misses. The fix is to give the chunk its context back.
Two axes. Change the text you embed: metadata enrichment (prepend header path/title/date — highest-ROI, do first) and Contextual Retrieval (Anthropic — an LLM blurb prepended before embedding and BM25: −35% → −49% → −67% failures; ~$1.02/M tokens thanks to prompt caching). Change when you pool: late chunking (Jina — embed the whole doc, then split & mean-pool → vectors conditioned on full-doc context; ~3.5% rel; needs a long-context mean-pooling model; window-capped).
They operate on different pipeline stages, so they stack (and stack with small-to-big and reranking).
Don't over-build: under ~200k tokens, skip RAG and put the whole corpus in the prompt. Otherwise order by ROI — metadata → late chunking/contextual → reranking — and measure on your eval.
That completes the chunking & indexing section. Next: Advanced Retrieval — starting with hybrid search (dense + sparse / BM25), which Contextual Retrieval just hinted at.