Ingestion & Hybrid Retrieval
Introduction
In L1 (Project Brief & Architecture) you drew the system and scaffolded the repo — a runnable service with a /health check and empty ingest/ and retrieve/ modules waiting to be filled. This lesson fills them. By the end you'll have a working offline indexer (make ingest) that turns your documents into a searchable index, and an online retrieve(query) that returns the handful of chunks an answer should be grounded in. This is the retrieval engine of the whole assistant — and retrieval quality caps everything downstream: no generator, however strong, can answer well from the wrong chunks.
We're building the two halves of retrieval the architecture promised:
- ① The offline index — load → chunk → (optionally) add context → embed → index. Runs once per document change, as a batch job. Its output is a Qdrant collection holding, for every chunk, a dense vector (for meaning) and a sparse BM25 vector (for exact terms), plus the raw text and metadata for citations.
- ② The online retrieval — embed the query → hybrid search (dense + BM25) → fuse with RRF → rerank. Runs on every question, in well under a second. Its output is the top ~6 chunks, best-first, that L3 will hand to Claude.
Why hybrid, and why this is the lesson that earns its keep: the naive demo does dense-only retrieval, and dense-only has a famous blind spot — it matches meaning, not tokens, so it misses the exact strings users actually type: error codes, IDs, function names, acronyms. Add a keyword (BM25) leg and you catch those; fuse the two ranked lists with Reciprocal Rank Fusion and you get the best of both without tuning score weights. Then a reranker re-scores the survivors with a model that reads query and chunk together, lifting the genuinely-best chunk to the top. That chain — hybrid → RRF → rerank — is the single biggest retrieval-quality upgrade over naive RAG, and it's what this lesson ships.
Everything here is runnable code on the course's stack (Voyage-3 embeddings, Qdrant hybrid, Voyage rerank-2.5, Claude Haiku for context). Build it against your corpus as you read.
① The Offline Index — Four Stages
The indexer is a batch job: it runs when your documents change, not on the request path, so it can afford to be thorough (chunk carefully, call an LLM for context, embed in big batches). Four stages, each a small module under src/rag/ingest/:
- Load — pull raw documents from the source and extract clean text. PDFs (
pypdf), HTML, Markdown, code — whatever your corpus is. Carry metadata (source path/URL, title, section) from the very start; you need it for citations in L3. - Chunk — split each document into retrieval-sized units. The chunk is the atom of retrieval — get this wrong and nothing downstream can fix it. We use recursive, token-based chunking and keep the metadata on every chunk.
- Embed — turn each chunk's text into a dense vector with Voyage-3, in batches.
- Index — write each chunk to Qdrant as a point carrying both a dense vector and a sparse BM25 vector, plus the text and metadata, in one collection.
The contract of the offline side is a single typed Chunk, threaded through all four stages. Define it once:
# src/rag/ingest/types.py
from dataclasses import dataclass, field
@dataclass
class Chunk:
id: str # stable id: f"{source}::{ordinal}"
text: str # the chunk body (what gets embedded + BM25-indexed)
source: str # file path or URL -> citation
title: str = "" # doc / section title -> citation
ordinal: int = 0 # position of the chunk within its document
context: str = "" # optional 1-line context prefix (Contextual Retrieval)
meta: dict = field(default_factory=dict)
@property
def indexed_text(self) -> str:
# what we actually embed + BM25-index: context prefix (if any) + body.
return f"{self.context}\n\n{self.text}" if self.context else self.text
That indexed_text property is the quiet hero of this lesson: both the embedding and the BM25 index are built from it, so when we turn Contextual Retrieval on, both retrievers benefit from the same prepended context. Hold that thought — we get there in two stages.
Chunking That Actually Works
Chunking is where most RAG systems are quietly lost or won. Two real failure modes: too big and the chunk dilutes the relevant sentence (and blows your context budget); too small and it loses the surrounding meaning. You want chunks that are one coherent idea, with enough context to stand alone.
What the evidence says (2025–2026): despite the hype around fancy strategies, recursive (structure-aware) splitting at ~512 tokens with ~10–20% overlap is the production default that keeps winning benchmarks. A February-2026 comparison of seven strategies across 50 papers put recursive 512-token splitting first at ~69% accuracy, while pure semantic chunking came in lower (~54%) — it tended to produce tiny fragments (averaging ~43 tokens) that lost context. Start simple; recursive 512 / 64 works for the large majority of corpora. Reach for semantic or late chunking only when you can measure that it helps (you'll have the eval harness in L4).
Recursive splitting tries to break on the biggest natural boundary first (paragraphs → sentences → words), so chunks rarely cut mid-sentence. Use the token-based variant so 512 means 512 model tokens, not characters:
# src/rag/ingest/chunker.py
from langchain_text_splitters import RecursiveCharacterTextSplitter
from rag.ingest.types import Chunk
# 512-token chunks with 64-token (~12%) overlap, counted with a real tokenizer.
_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
chunk_size=512,
chunk_overlap=64,
separators=["\n\n", "\n", ". ", " ", ""], # paragraph -> line -> sentence -> word
)
def chunk_document(*, text: str, source: str, title: str = "") -> list[Chunk]:
"""Split one document into overlapping, metadata-carrying chunks."""
parts = _splitter.split_text(text)
return [
Chunk(
id=f"{source}::{i}",
text=part,
source=source,
title=title,
ordinal=i,
)
for i, part in enumerate(parts)
]
The overlap matters: 64 tokens of shared text between neighbours means a fact that straddles a boundary still appears whole in at least one chunk. The metadata matters more: source, title, and ordinal ride along so that when L3 cites a chunk, it can say "from auth-guide.md, §Rotating keys" — citations are only as good as the metadata you preserved here.
Contextual Retrieval — The High-Leverage Upgrade
Here's the deepest problem with chunking, and the fix that moves the numbers most. When you split a document, each chunk loses the context of the whole. A chunk that reads "It must be rotated every 90 days" is useless in isolation — what must be rotated? The embedding and the keyword index both see an ambiguous fragment, and retrieval suffers.
Anthropic's Contextual Retrieval fixes this at ingest time: before embedding and BM25-indexing each chunk, use a cheap model to write a 50–100 token context that situates the chunk inside its document, and prepend it. "It must be rotated every 90 days" becomes "This chunk is from the Auth Guide's key-rotation policy; it specifies how often API access keys must be rotated. It must be rotated every 90 days." Now both retrievers can find it.
The payoff is large and measured (Anthropic's numbers): contextual embeddings alone cut retrieval failures by 35% (5.7% → 3.7%); adding contextual BM25 takes it to 49% (→ 2.9%); and with reranking on top, 67% (→ 1.9%). That's why we wire it in now — it touches both the dense and sparse indexes through that one indexed_text property.
The trick to making it cheap: put the whole document in the prompt's system block with cache_control, so Anthropic's prompt caching charges you for the long document once and then reuses it across all of that document's chunks. Use Haiku — this is a high-volume, easy task.
# src/rag/ingest/contextualize.py
from anthropic import Anthropic
from rag.config import settings
from rag.ingest.types import Chunk
_client = Anthropic(api_key=settings().anthropic_api_key)
_PROMPT = (
"Here is a chunk from the document above. In 50-100 tokens, write a short context that situates "
"this chunk within the whole document to improve search retrieval. Answer ONLY with the context, "
"no preamble.\n\n<chunk>\n{chunk}\n</chunk>"
)
def add_context(doc_text: str, chunks: list[Chunk]) -> list[Chunk]:
"""Prepend a 1-line, document-aware context to each chunk (Anthropic Contextual Retrieval).
The full document sits in a CACHED system block, so prompt caching charges for it once
per document and reuses it for every chunk -> cheap even on large corpora.
"""
for c in chunks:
resp = _client.messages.create(
model=settings().cheap_model, # claude-haiku-4-5
max_tokens=128,
system=[{
"type": "text",
"text": f"<document>\n{doc_text}\n</document>",
"cache_control": {"type": "ephemeral"}, # <- prompt caching
}],
messages=[{"role": "user", "content": _PROMPT.format(chunk=c.text)}],
)
c.context = resp.content[0].text.strip()
return chunks
Is it worth the cost? For most corpora, yes — it's the highest-leverage single change after hybrid search, and prompt caching makes it ~1–2¢ per document, paid once at index time (never on the query path). Make it a config flag (contextual: bool) so you can A/B it against your eval set in L4. (In the lab below, toggle Contextual Retrieval and watch the identifier query that dense got wrong flip to right once the context is prepended.)
Embedding with Voyage-3
Now turn each chunk's indexed_text into a dense vector. We use Voyage-3 (specifically voyage-3-large) — it's a top-scoring general-purpose embedding model with a 32K-token context (plenty for any chunk), Matryoshka dimensions (2048 / 1024 / 512 / 256 — pick the trade-off you want), and strong quality per dollar. We'll use 1024-d (a good quality/storage balance) and the model's input_type distinction — "document" when indexing, "query" when searching — which measurably improves retrieval.
Batch your embeds. Embedding one chunk per call is slow and wasteful; send them in batches:
# src/rag/ingest/embedder.py
import voyageai
from rag.config import settings
from rag.ingest.types import Chunk
_vo = voyageai.Client(api_key=settings().voyage_api_key)
def embed_chunks(chunks: list[Chunk], batch_size: int = 128) -> list[list[float]]:
"""Dense-embed each chunk's indexed_text with Voyage-3 (input_type='document')."""
vectors: list[list[float]] = []
for i in range(0, len(chunks), batch_size):
batch = [c.indexed_text for c in chunks[i : i + batch_size]]
resp = _vo.embed(
batch,
model=settings().embed_model, # voyage-3-large
input_type="document", # asymmetric: docs vs queries
output_dimension=1024, # Matryoshka: 2048/1024/512/256
)
vectors.extend(resp.embeddings)
return vectors
def embed_query(query: str) -> list[float]:
"""Embed a search query (note input_type='query')."""
return _vo.embed(
[query], model=settings().embed_model, input_type="query", output_dimension=1024
).embeddings[0]
Why a managed embedder? Quality and zero ops. If you must self-host (cost, data residency), swap in open BGE behind the same two functions — the rest of the pipeline doesn't change. That's the payoff of the L1 architecture: the boxes are swappable. (The dense vector is only half the index, though — the other half is the sparse BM25 vector, which is what catches the exact terms embeddings miss. We build both into one Qdrant point next.)
Indexing — Dense + Sparse in One Qdrant Collection
This is the structural decision that makes hybrid search clean: store both vectors per chunk in one Qdrant collection using named vectors — a dense vector (Voyage) and a bm25 sparse vector. Qdrant computes BM25 natively when you declare the sparse vector with the IDF modifier; we generate the sparse vectors with fastembed (Qdrant/bm25), which gives token indices + values with no model to host.
Create the collection with one dense and one sparse vector space:
# src/rag/ingest/indexer.py
from fastembed import SparseTextEmbedding
from qdrant_client import QdrantClient, models
from rag.config import settings
from rag.ingest.embedder import embed_chunks
from rag.ingest.types import Chunk
_client = QdrantClient(url=settings().qdrant_url)
_bm25 = SparseTextEmbedding("Qdrant/bm25") # local, no API
def ensure_collection() -> None:
if _client.collection_exists(settings().collection):
return
_client.create_collection(
collection_name=settings().collection,
# dense vector space (Voyage, cosine)
vectors_config={
"dense": models.VectorParams(size=1024, distance=models.Distance.COSINE)
},
# sparse BM25 space — IDF computed by Qdrant
sparse_vectors_config={
"bm25": models.SparseVectorParams(modifier=models.Modifier.IDF)
},
)
Upsert each chunk as a point carrying both vectors plus the text and citation metadata:
def index_chunks(chunks: list[Chunk]) -> int:
ensure_collection()
dense = embed_chunks(chunks) # Voyage dense vectors
sparse = list(_bm25.embed([c.indexed_text for c in chunks])) # BM25 sparse vectors
points = [
models.PointStruct(
id=i,
vector={
"dense": dense[i],
"bm25": models.SparseVector(
indices=sparse[i].indices.tolist(),
values=sparse[i].values.tolist(),
),
},
payload={
"text": c.text,
"context": c.context,
"source": c.source,
"title": c.title,
"ordinal": c.ordinal,
},
)
for i, c in enumerate(chunks)
]
_client.upsert(settings().collection, points=points)
return len(points)
And the runner that ties load → chunk → (context) → index together — this is your make ingest:
# src/rag/ingest/run.py -> `python -m rag.ingest.run ./data`
import sys
from pathlib import Path
from rag.config import settings
from rag.ingest.chunker import chunk_document
from rag.ingest.contextualize import add_context
from rag.ingest.indexer import index_chunks
def main(data_dir: str) -> None:
total = 0
for path in Path(data_dir).rglob("*.md"): # extend: .pdf, .html, code
doc = path.read_text(encoding="utf-8")
chunks = chunk_document(text=doc, source=str(path), title=path.stem)
if settings().contextual: # config flag (A/B in L4)
chunks = add_context(doc, chunks)
total += index_chunks(chunks)
print(f" indexed {path.name}: {len(chunks)} chunks")
print(f"done -> {total} chunks in '{settings().collection}'")
if __name__ == "__main__":
main(sys.argv[1] if len(sys.argv) > 1 else "./data")
Run docker compose up -d qdrant then python -m rag.ingest.run ./data, and your corpus is now a hybrid index. (Add contextual: bool = False to Settings in config.py.) Indexing is idempotent on id, so re-running updates in place.
② Hybrid Retrieval — Dense + BM25, Fused with RRF
Now the online side. A query runs two searches in parallel — dense (semantic) and BM25 (keyword) — and we fuse their ranked lists. The naive way to fuse is a weighted sum of scores, and it's a trap: dense cosine scores are bounded (~0–1); BM25 scores are unbounded and shift per query, so any fixed weight gets dominated by whichever retriever happens to have larger raw numbers on that query.
Reciprocal Rank Fusion (RRF) sidesteps this by using ranks, not scores. Each document's fused score is the sum, over the lists it appears in, of 1 / (k + rank) — where rank is its position in that list and k (≈60) damps the influence of low ranks. A document near the top of both lists wins by consensus; a document that's #1 in one list still scores well even if the other list misses it entirely. No score normalization, no per-query tuning.
Qdrant's Query API does the whole thing server-side: two prefetch legs (one per vector space) feed a FusionQuery(fusion=RRF). One round trip:
# src/rag/retrieve/hybrid.py
from fastembed import SparseTextEmbedding
from qdrant_client import QdrantClient, models
from rag.config import settings
from rag.ingest.embedder import embed_query
_client = QdrantClient(url=settings().qdrant_url)
_bm25 = SparseTextEmbedding("Qdrant/bm25")
def hybrid_search(query: str, limit: int | None = None) -> list[dict]:
"""Dense + BM25 retrieval fused with RRF, server-side in one round trip."""
limit = limit or settings().top_k_retrieve # ~40 candidates
dense_q = embed_query(query) # Voyage (input_type='query')
sparse_q = next(_bm25.query_embed(query)) # BM25 sparse query vector
res = _client.query_points(
settings().collection,
prefetch=[
models.Prefetch(query=dense_q, using="dense", limit=limit),
models.Prefetch(
query=models.SparseVector(
indices=sparse_q.indices.tolist(),
values=sparse_q.values.tolist(),
),
using="bm25",
limit=limit,
),
],
query=models.FusionQuery(fusion=models.Fusion.RRF), # <- the fusion
limit=limit,
with_payload=True,
)
return [
{**p.payload, "id": p.id, "score": p.score} for p in res.points
]
That returns ~40 candidates ranked by RRF — already far better than dense-only, because the BM25 leg rescues every exact-term query the embeddings blur. But ~40 chunks is too many (and too noisy) to hand a generator. The genuinely-best chunk might be RRF-rank 7. That's the reranker's job.
Reranking — From 40 Candidates to the Best 6
Hybrid search optimizes for recall — get the right chunk into the candidate set. Reranking optimizes for precision — get the right chunk to the top. It's the highest-precision single step in retrieval, and it's cheap.
Why it works: the retrievers score the query and each chunk independently (embed separately, then compare) — fast, but it can't model how query and chunk interact. A cross-encoder reranker feeds the (query, chunk) pair together through a model that reads them jointly, producing a far more accurate relevance score. You can't run it over the whole corpus (too slow), but over 40 candidates it's perfect. We use Voyage rerank-2.5; keep the top 6 as the final context:
# src/rag/retrieve/rerank.py
import voyageai
from rag.config import settings
_vo = voyageai.Client(api_key=settings().voyage_api_key)
def rerank(query: str, candidates: list[dict]) -> list[dict]:
"""Cross-encoder rerank; keep the best top_k_rerank (~6) with a fused relevance score."""
if not candidates:
return []
docs = [c["text"] for c in candidates]
reranked = _vo.rerank(
query, docs, model=settings().rerank_model, top_k=settings().top_k_rerank
)
out = []
for r in reranked.results: # r.index -> into candidates; r.relevance_score
c = dict(candidates[r.index])
c["rerank_score"] = r.relevance_score
out.append(c)
return out
Compose the public entry point — this is the retrieve(query) the rest of the system calls:
# src/rag/retrieve/__init__.py
from rag.retrieve.hybrid import hybrid_search
from rag.retrieve.rerank import rerank
def retrieve(query: str) -> list[dict]:
"""Full retrieval: hybrid (dense + BM25 + RRF) -> rerank -> top-6 chunks for grounding."""
candidates = hybrid_search(query) # ~40, recall-optimized
return rerank(query, candidates) # top-6, precision-optimized
# try it
>>> from rag.retrieve import retrieve
>>> hits = retrieve("how do I rotate my API credentials?")
>>> for h in hits:
... print(round(h["rerank_score"], 3), h["title"], "--", h["text"][:60])
0.982 Auth Guide -- Rotate your access keys or password from Settings ...
0.741 Accounts -- Update your login credentials after verifying your ...
That retrieve() — hybrid → RRF → rerank — is the retrieval engine of the whole assistant. L3 takes its top-6 chunks, builds a grounded prompt, and has Claude answer with citations (the source / title you carried since the chunker) and abstain when the top rerank_score is below min_score. Retrieval done right is what makes that grounding possible.
✅ Definition of Done (this step)
Before moving to L3, your ingest/ and retrieve/ modules should be real and runnable:
- Qdrant up (
docker compose up -d qdrant) and a collection with nameddense+bm25vectors. -
python -m rag.ingest.run ./dataindexes your corpus — load → chunk (recursive 512/64) → embed (Voyage-3) → upsert, with metadata for citations. - Contextual Retrieval wired behind a flag (
settings().contextual) — Haiku + a cached document block — so you can A/B it in L4. -
hybrid_search()returns RRF-fused candidates via Qdrant's Query API (dense + BM25prefetch→FusionQuery(RRF)). -
retrieve()composes hybrid → rerank (Voyage rerank-2.5) → top-6. - A quick smell test: an exact-term query (an error code, an ID) returns the right chunk — proof the BM25 leg is doing its job that dense-only couldn't.
If an identifier query that used to miss now lands, you've felt the entire point of this lesson: hybrid + RRF + rerank. Next, L3 turns these chunks into grounded, cited answers.
See It — The Ingest → Hybrid Retrieval Lab
This lab is the pipeline you just built, made playable. The top panel is ingest: set chunk size and overlap and toggle Contextual Retrieval, and watch the offline flow update (docs → chunk → [prepend context] → embed Voyage-3 → Qdrant dense + sparse). The bottom panel is retrieval: type any query — or pick a preset — and watch three live-ranked columns recompute over the same eight chunks: Dense (vectors), BM25 (sparse), and RRF (fused).
Run the three presets and watch the lesson's core claims land:
- paraphrase ("change my login secret") — shares almost no exact words with the right chunks, so BM25's recall drops while dense and RRF recover them by meaning.
- identifier ("E-4012 keeps failing") — a rare code carries the meaning, so dense ranks the wrong chunk #1 (a ✗ on hit@1) while BM25 nails it. Now flip Contextual Retrieval ON and watch the prepended "Auth · Troubleshooting" context lift dense back onto the right chunk — the 35% win, live.
- consensus ("lower the request timeout") — both retrievers like the same chunk, and RRF turns that agreement into a rock-solid #1.
Drag the RRF k slider to feel how a larger k flattens the rank advantage (more democratic) while a smaller k rewards each list's #1.
Notice three things. One: dense and BM25 fail on opposite queries — paraphrases break BM25, rare identifiers break embeddings — which is exactly why you run both. Two: RRF needs no shared score scale because it fuses by rank; that's why it beats weighted-score fusion. Three: Contextual Retrieval changes what the indexes can "see" — the same query flips from miss to hit once the chunk carries its context.
🧪 Try It Yourself
Reason these out, then check against the lab and the code.
1. A user searches your API docs for the exact error string E_4012. Your dense-only demo returns three semantically-related chunks but not the one that defines E_4012. Which part of this lesson's pipeline fixes it, and why does dense-only miss it?
2. Why does RRF fuse dense and BM25 better than a weighted sum of their scores like 0.5*dense + 0.5*bm25? Name the specific property of the two score distributions that breaks the weighted sum.
3. You set top_k_retrieve = 40 and top_k_rerank = 6. What is each number for, and what goes wrong if you skip the reranker and just send the top-6 by RRF to the generator?
4. Contextual Retrieval calls an LLM per chunk at index time. Why is that not a latency problem for your users, and what one technique keeps its cost low on large documents?
5. Your eval (L4) shows recall is fine but answers still cite slightly-wrong chunks. Is the fix more likely in chunking, hybrid search, or reranking — and why?
Answers.
1. The BM25 (sparse) leg of hybrid search. Dense embeddings map text to meaning, and a rare token like E_4012 carries almost no semantic signal — it gets blurred toward generic "error" neighbours. BM25 matches the exact token (and IDF makes a rare token high-weight), so it puts the defining chunk #1; RRF then keeps it. (In the lab, the identifier preset shows dense missing and BM25 nailing it.)
2. Dense cosine scores are bounded (~0–1); BM25 scores are unbounded and rescale per query. A fixed weight over raw scores is therefore dominated by whichever retriever has larger raw magnitudes on that query — so the weight you tuned on one query is wrong on the next. RRF uses ranks, which are always 1, 2, 3… on both sides, so the two lists are directly comparable with no normalization.
3. top_k_retrieve = 40 is the recall budget — cast a wide enough net that the right chunk is somewhere in the candidates. top_k_rerank = 6 is the precision budget — the cross-encoder re-scores those 40 reading query+chunk together and lifts the genuinely-best to the top, and 6 is what you can afford to put in the prompt. Skip the reranker and you feed the generator chunks ordered by RRF, where the truly-best is often #5–#10 — the model gets distracted by near-misses and grounds on the wrong one.
4. It runs in the offline batch job, never on the request path, so users never wait for it — queries only hit retrieve(). Cost stays low via prompt caching: put the whole document in a cached system block, so you pay for the long document once and reuse it across all of its chunks (~1–2¢/doc).
5. Most likely reranking (precision), with chunking the runner-up. Good recall means the right chunk is retrieved — the problem is ordering: the generator grounds on a near-miss ranked above it, which is exactly what the cross-encoder reranker fixes. If even the reranked top chunk is almost right, suspect chunk boundaries (the fact is split across two chunks) — revisit size/overlap or add Contextual Retrieval. Hybrid search governs whether it's retrieved at all, which your recall number says is fine.
Mental-Model Corrections
- "Vector search is retrieval." → Dense vectors are half of it. They miss exact terms (codes, IDs, names); the BM25 leg catches those. Hybrid + RRF is the production default.
- "Fuse by averaging the scores." → Dense and BM25 scores live on different, shifting scales — a weighted sum is dominated by whichever is larger. RRF fuses by rank, no normalization.
- "Smaller chunks are better — more precise." → Too-small chunks lose context and retrieve noisily (semantic chunking's failure mode). Recursive ~512/64 is the benchmark-winning default; fix context loss with Contextual Retrieval, not by shrinking.
- "Reranking is optional polish." → It's the highest-precision step and cheap — it turns a recall-optimized candidate set into the right top-6. Hybrid gets the chunk in; rerank gets it to the top.
- "Contextual Retrieval is too expensive / too slow." → It's an offline, one-time cost with prompt caching (~1–2¢/doc), never on the query path — and it cuts retrieval failures ~35–49%. Wire it behind a flag and let the eval decide.
- "Embeddings are embeddings." → Use the model's
input_type(documentvsquery) and a sane dimension; and remember the embedder is swappable (Voyage ↔ Cohere ↔ BGE) behind the same two functions. - "Store vectors; fetch text from a separate DB." → Keep the text + citation metadata in the payload alongside the vectors. Retrieval should return everything L3 needs to cite — in one round trip.
Key Takeaways
- Retrieval quality caps the whole assistant — no generator can answer well from the wrong chunks. This lesson built the engine: offline index + online
retrieve(). - The offline index is four stages: load → chunk (recursive 512/64, keep metadata) → embed (Voyage-3,
input_type='document', 1024-d) → index into Qdrant as one point with a dense and a sparse BM25 vector. - Contextual Retrieval is the high-leverage upgrade: prepend a Haiku-written 50–100 token context to each chunk before embedding and BM25 — cutting retrieval failures ~35–49%, cheap via prompt caching, behind a config flag.
- Online retrieval = hybrid → RRF → rerank. Run dense + BM25 in parallel, fuse with Reciprocal Rank Fusion (by rank, so no score-scale problem) for ~40 candidates, then rerank (Voyage rerank-2.5) down to the best 6.
- Dense and BM25 fail on opposite queries — paraphrases break BM25, identifiers break embeddings — so you run both; RRF wins by consensus and rescues each list's solo hits.
- Hybrid optimizes recall; rerank optimizes precision. Get the right chunk in the candidate set, then get it to the top — the single biggest retrieval upgrade over naive RAG.
- Next — L3 (Generation, Citations & Guardrails): take
retrieve()'s top-6, build a grounded prompt, and have Claude answer with citations and abstain when the top rerank score is belowmin_score— turning retrieved chunks into trustworthy answers.