Multi-Vector & Summary Indexing

Introduction

We close the chunking & indexing section with the most general form of an idea you've now seen twice. In small-to-big (L93) you learned that the chunk you match needn't be the chunk you return. This lesson takes that to its logical conclusion:

The thing you index needn't be the document at all. Index any representation that matches queries well — a summary, a list of questions, a smaller chunk — and return the original.

This is multi-vector retrieval: give one document several embeddings, each a different "door in," all pointing back to the same original. It's a small architectural change with an outsized payoff, because different queries want to match different things.

In this lesson you'll learn:

The decoupling principle and the two-store architecture (MultiVectorRetriever)
The three workhorse representations: summaries, hypothetical questions, and smaller chunks
RAPTOR — recursive summary trees for multi-level reasoning
The costs, the ColBERT disambiguation, and when it's worth it

The Principle: Index One Thing, Return Another

Naive RAG embeds the chunk and returns the same chunk — index and answer are welded together. But that's an assumption, not a law. Break the weld and a question opens up: what is the best text to match this document against queries? Often it's not the raw document:

The raw text may be noisy — boilerplate, tangents, filler — diluting the vector (recall L90).
A user's query is usually a short question, which may look very different from the declarative prose of the document that answers it (a vocabulary/style gap).

So we index a better-matching representation and keep the original aside to return. The mechanism is the now-familiar two-store architecture:

from langchain.retrievers.multi_vector import MultiVectorRetriever
from langchain.storage import InMemoryByteStore
import uuid

# Two stores, linked by a shared id (this is the WHOLE pattern):
#   vectorstore → the embedded REPRESENTATIONS (summaries / questions / chunks)
#   docstore    → the ORIGINAL documents, fetched by id and returned to the LLM
store = InMemoryByteStore()
ID_KEY = "doc_id"
retriever = MultiVectorRetriever(vectorstore=vectorstore, byte_store=store, id_key=ID_KEY)

doc_ids = [str(uuid.uuid4()) for _ in docs]
retriever.docstore.mset(list(zip(doc_ids, docs)))         # stash the ORIGINALS

# Build a SUMMARY per doc and index THAT (tagged with its doc's id):
summaries = summarize_chain.batch(docs)                   # LLM: 1–2 sentence summary each
summary_vectors = [Document(page_content=s, metadata={ID_KEY: doc_ids[i]})
                   for i, s in enumerate(summaries)]
retriever.vectorstore.add_documents(summary_vectors)      # search hits the SUMMARY → returns the ORIGINAL

The vectorstore holds the embedded representations; the docstore holds the originals; a shared doc_id links them. Search hits a representation → the retriever follows the id → returns the original document. (You may notice this is exactly the small-to-big architecture — and indeed ParentDocumentRetriever is a special case of MultiVectorRetriever where the representation happens to be "smaller chunks.")

An infographic titled 'Multi-Vector & Summary Indexing — Index One Thing, Match Another'. The principle: the thing you index need not be the thing you return; give one document several embeddings — each a different door in — but always return the original (small-to-big was one case, where the door is a smaller chunk). Four ways to represent one document for matching. Summaries: embed a dense LLM summary of the doc, stripping filler and sharpening the signal; best for long or noisy docs and high-level questions. Hypothetical questions: embed questions the doc answers, turning search into question-to-question matching; this is index-time, not query-time HyDE; best for FAQ and question-heavy workloads. Smaller chunks: embed small children and return the parent — exactly small-to-big from the previous lesson, a special case of this pattern; best for precise matching plus full context. RAPTOR tree: recursively cluster and summarize into a multi-level tree, embed every level, and answer at the right altitude; best for multi-step reasoning over big corpora. Architecture: the MultiVectorRetriever keeps representations in the vectorstore and originals in the docstore, linked by doc_id; match any representation and return its original, deduped. Costs: index-time LLM calls for summaries and questions, more vectors to store, regenerate on edits (staleness); token-level ColBERT is a different kind of multi-vector covered in the reranking lesson; measure the lift. Banner: one document, many doors in — index the representation your queries actually match, and always return the original.

Representation 1 — Summaries

The simplest powerful representation: embed an LLM-generated summary of each document, but return the full document.

Why it helps: a good summary is dense and de-noised — it states what the document is about without the filler that muddies the raw text's embedding. So a high-level query like "what's our refund policy?" lands cleanly on the summary's vector, then you hand the LLM the complete policy to answer from.

Best for: long or noisy documents (reports, transcripts, web pages) and high-level / thematic queries where the raw text's specifics would scatter the vector. The cost: one LLM summarization call per document at index time (a one-time, cacheable expense).

Representation 2 — Hypothetical Questions

This one is clever. The deepest mismatch in retrieval is shape: a user types a question ("how long until I get my money back?"), but your documents are written as statements ("refunds post within 5 business days"). Question and answer can be semantically related yet sit awkwardly apart in vector space.

Fix the shape mismatch at the source: for each document, have an LLM generate the questions that document answers, embed those, and tag them with the doc's id. Now retrieval becomes question-to-question matching — the user's question is compared against stored questions, which is a far more natural alignment.

# Hypothetical questions: for each doc, ask an LLM what questions it ANSWERS,
# embed those, and tag them with the doc's id. Queries (which ARE questions) then
# match stored questions directly — "question-to-question" retrieval.
questions = qgen_chain.batch(docs)                         # e.g. ["How long is the refund window?", ...]
q_vectors = [Document(page_content=q, metadata={ID_KEY: doc_ids[i]})
             for i, qs in enumerate(questions) for q in qs]
retriever.vectorstore.add_documents(q_vectors)

# You can index ALL representations at once — summary + questions + smaller chunks —
# every one a different "door in" that resolves back to the same original document.
# NOTE: this is INDEX-time. HyDE (a later lesson) generates a hypothetical answer at QUERY time.

Best for: FAQ-style and question-heavy workloads (support, docs search). Practitioners report it can meaningfully sharpen precision by closing the query↔document style gap — though, as always, measure the lift on your data rather than trusting a quoted number.

Don't confuse this with HyDE (a later lesson). Hypothetical questions are an INDEX-time representation (generated once per document). HyDE generates a hypothetical answer at QUERY time for the incoming query. Same spirit (bridge the shape gap), opposite ends of the pipeline.

See It: One Doc, Many Doors In

Here's the payoff made concrete. One source document is indexed under three representations — the raw chunk, a summary, and hypothetical questions. Pick a query and watch which representation matches best — it changes with the query's intent — while the system always returns the same original document:

One document indexed three ways. Pick a query and watch which representation matches best — it changes with the query's intent — while the system always returns the same original document.

Notice the pattern: a detailed query lands on the raw chunk, an overview query on the summary, and a natural-language question on the hypothetical questions. No single representation wins them all — which is the entire argument for indexing several. You widen the set of queries that can find this document, at the cost of a few extra vectors.

RAPTOR — Recursive Summary Trees

RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval) scales the summary idea into a hierarchy. Bottom-up, it clusters similar chunks, summarizes each cluster into a higher-level node, then clusters and summarizes those summaries — recursively — until you have a tree: leaves are raw chunks, branches are mid-level summaries, the trunk is the whole-corpus gist.

Every level is embedded, so retrieval can match at the right altitude for the question:

A specific question → a leaf (detail).
A broad, synthesizing question ("what are the main themes across these reports?") → a high-level summary node — without the model having to read and reconcile hundreds of leaf chunks.

This is what makes RAPTOR strong for multi-step reasoning over large corpora: the abstraction is precomputed into the tree. The trade-off is index-time cost: embedding + clustering + recursive summarization is expensive to build and slow/costly to update when content changes — but query time stays simple. Reach for it on large, relatively static corpora where questions span abstraction levels.

Costs, Caveats & a Name Clash

Multi-vector indexing is powerful but not free — engineer for these:

Index-time LLM cost. Summaries and hypothetical questions mean an LLM call per document at ingestion. One-time and cacheable, but real — budget for it on large corpora.
More vectors to store & search. Several representations per doc multiplies your index size (and a touch of query cost).
Dedup to one original. Multiple representations of the same doc may all match — the retriever must return the original once (handled by the doc_id link).
Staleness. When a document changes, you must regenerate its summaries/questions — representations can silently drift out of date (the update discipline from L82, again).
A name clash to know: "multi-vector" also describes token-level late-interaction models like ColBERT, which store one vector per token and match "late." That's a different technique (it's about matching, not representations) and it belongs to the reranking lesson. Here, multi-vector means multiple document-level representations.

And the honest bar: these add index-time complexity for gains that are workload-dependent. A clean FAQ may need nothing; a noisy, long, question-heavy corpus may love summaries + hypothetical questions. Prototype, then measure recall/precision on your eval before committing.

🧪 Try It Yourself

Drive the widget, then choose representations for real systems:

In the interactive, which representation wins for the detailed query? the overview query? the natural-language question? What does that tell you about indexing only one of them?
You're building search over 2,000 long, rambling meeting transcripts; users ask high-level questions ("what did we decide about pricing?"). Which representation, and why?
A customer-support bot: users type questions, your KB articles are written as statements. Which representation closes the gap — and what's the index-time cost?
Your teammate says "let's use multi-vector with ColBERT." What's the terminology mix-up?
You add summaries and your docs get edited weekly. What new maintenance task did you just sign up for?

→ (1) Detail → raw chunk, overview → summary, question → hypothetical questions; indexing only one would miss the other query types — index several. (2) Summaries — they de-noise rambling transcripts and match high-level questions; return the full transcript section for the answer. (3) Hypothetical questions (question-to-question matching closes the style gap); cost = one LLM question-generation call per article at index time. (4) Multi-vector here means multiple document-level representations (summary/questions/chunks); ColBERT is token-level late interaction — a different technique (the reranking lesson). (5) Regenerating each edited doc's summaries/questions, or they go stale and misrepresent the current content.

Mental-Model Corrections

"You must index the document text itself." No — index the best-matching representation (summary, questions, chunk) and return the original. Index ≠ return.
"Summaries lose information, so they hurt." You embed the summary to match, but return the full document to answer — you lose nothing at generation time.
"Hypothetical questions = HyDE." No: hypothetical questions are index-time (per document); HyDE generates a hypothetical answer at query-time. Different ends of the pipeline.
"Multi-vector means ColBERT." Two different things: here it's multiple document-level representations; ColBERT is per-token late interaction (reranking lesson).
"More representations are always better." Each adds index-time LLM cost, storage, and staleness. Gains are workload-dependent — measure.
"This is unrelated to small-to-big." Small-to-big is a multi-vector case (the representation = a smaller chunk). Same architecture, broader idea.

Key Takeaways

The decoupling principle, generalized: the thing you index needn't be what you return. Give one document several embeddings ("doors in") and resolve them all to the original via MultiVectorRetriever (vectorstore = representations, docstore = originals, linked by doc_id).
Summaries — embed a dense, de-noised summary; best for long/noisy docs and high-level queries.
Hypothetical questions — embed the questions a doc answers → question-to-question matching; best for FAQ/question-heavy workloads. (Index-time — not query-time HyDE.)
Smaller chunks = small-to-big (a special case). RAPTOR = a recursive cluster-and-summarize tree that lets you retrieve at the right abstraction level for multi-step reasoning over big corpora.
Costs: index-time LLM calls, more vectors, dedup, and staleness on edits; and "multi-vector" also names ColBERT token-level late interaction (a different technique → reranking). Measure the lift.
That completes Chunking & Indexing. You can now make chunks coherent (L90–92), decouple what you return (L93), restore each chunk's context (L94), and index multiple representations (L95). Next section: Advanced Retrieval — hybrid search, fusion, reranking, query transformation, and routing.