Why Chunking Makes or Breaks RAG

Welcome to the section that moves the needle most. Across the RAG research, one preprocessing choice consistently dominates retrieval quality — more than the embedding model, more than the vector DB, more than the prompt. It's chunking: how you split documents into the passages you embed and retrieve.

Why does a splitting decision matter so much? Because of one stubborn fact: a chunk is the atomic unit of retrieval — you can only ever retrieve whole chunks. If a fact is split across two chunks, neither one fully answers the question, and no embedding model or LLM downstream can reassemble what chunking tore apart. Chunking sets the ceiling on everything that follows.

This first lesson is the why — the mental model that makes every chunking technique in this section make sense. In it you'll learn:

• Why the chunk is the atomic unit of retrieval
• The Goldilocks tradeoff — and the deep reason big chunks blur (embedding dilution)
• What overlap is for, and why one fixed size is always a compromise
• The guiding principle: chunk to ideas, not character counts

Internalize this one sentence and the whole section follows: retrieval returns whole chunks — nothing smaller, nothing reassembled. Your vector search finds the k most similar chunks and hands them, verbatim, to the model. It cannot return half a chunk, and it cannot stitch two chunks into one coherent passage.

That has a sharp consequence. Suppose the answer to "What's the refund window for enterprise plans?" requires a sentence from paragraph 2 ("Enterprise refunds...") and a sentence from paragraph 5 ("...are processed within 10 business days"). If your chunker split those into different chunks, then no single chunk contains the answer — retrieval can surface one or the other, but never the complete fact. This is exactly Failure #7 (Incomplete) from the last lesson, and its root cause is chunking, three boxes upstream.

So the real design question isn't "how big should chunks be?" — it's "what is a self-contained answer to a question a user might ask, and how do I make that a chunk?" Size is downstream of that. A good chunk is a complete, standalone idea.

If a chunk must be a self-contained idea, why not just make chunks huge so they always contain enough? Because size cuts both ways — a genuine tradeoff with a sweet spot:

🔬 Too small (< ~100 tokens). Fragments are precise — they match a query on exactly one fact — but they lose surrounding context. "It expires after 30 days" is useless if the chunk doesn't say what expires. Studies show shrinking chunks (e.g., 1024 → 64 tokens) can raise fact-recall@1 by 10–15 points on entity-heavy data, but tanks narrative questions that need broader context.

🐘 Too big (> ~1000 tokens). Large chunks carry lots of context but dilute the signal (next section explains the mechanism), waste your context budget, and — past ~2,500 tokens — actually degrade generation quality. The model drowns in mostly-irrelevant text.

🎯 Just right (~200–500 tokens). For most applications, a single complete idea plus a little overlap lands here — a vector that's still sharp with enough context to be useful. Use it as a starting point, then tune on your data — it's a default, not a law.

Here's the mechanism beginners miss — and it's the most important idea in this lesson. An embedding compresses an entire chunk into one point in vector space, and that point behaves like an average of the chunk's meaning.

Picture a chunk that crams together three topics: billing, refunds, and shipping. Each topic 'pulls' the chunk's vector toward its own region — and the result lands somewhere in the muddy middle, close to none of them. So when a user asks a sharp question about refunds, this chunk is a weak match, because its vector got watered down by billing and shipping. The signal is diluted.

One idea per chunk → one clean vector → a strong match. Split those three topics into three chunks and each gets a crisp vector that lands squarely in its region. This is why "just split every 1000 characters" underperforms "split on topic boundaries": the goal isn't a size, it's semantic coherence — each chunk saying one thing clearly.

Researchers even quantify this as intra-chunk cohesion (the mean cosine similarity of a chunk's sentences to the chunk's own vector). High cohesion = a focused chunk = a sharp, retrievable vector. Low cohesion = a blurry vector that matches everything weakly and nothing well.

Drag the chunk size below and watch the two forces fight. Precision (red — matching exact facts) is highest when chunks are tiny and falls as they grow and dilute. Context completeness (teal — having enough to actually answer) is poor for fragments, rises with size, then sags when chunks get so big they bloat the context and degrade generation. The usable region is where they cross — roughly the 200–500 token band most apps adopt.

Notice there's no single peak that maximizes both — that's the whole point. Wherever you fix the size, you're trading one for the other. The cleverer techniques later in this section (semantic chunking, parent-child / small-to-big, multi-scale indexing) exist precisely to escape this one-dimensional tradeoff — e.g., match on a small, sharp chunk but feed the model the big, context-rich parent.

Two practical truths complete the picture:

Overlap (10–20%) is insurance against boundary cuts. When you slice a document, an idea can land across a boundary — the setup in chunk A, the payoff in chunk B. Overlap repeats a little text between neighbors so that straddling idea survives intact in at least one chunk. The research consensus is 10–20% of chunk size (NVIDIA found ~15% optimal at 1024 tokens); below 10% you start losing boundary context, and too much overlap just bloats your index with duplicates. It's a small setting with an outsized effect on recall.

One fixed size can never be optimal — because the query decides what size it wants. A lookup question ("What's the API rate limit?") wants a tiny, precise chunk; a synthesis question ("How does onboarding work end-to-end?") wants a big, context-rich passage. Indexing at a single size leaves 20–40% recall gaps versus the ideal across benchmarks — and indexing the same corpus at multiple scales and merging results measurably beats it. Keep this tension in mind: it's the thing the advanced chunking strategies (parent-child, multi-vector) are built to resolve.

Use the widget, then reason about real cases:

1. In the interactive, drag to 64 tokens, then 1500 tokens. Which curve wins at each end? Find roughly where they cross — what token range is it?
2. You're chunking an FAQ where each Q&A is a self-contained answer to a likely question. What's the natural chunk boundary — and why is character-count splitting a bad fit here?
3. A chunk contains: "...returns. Refunds post in 5 business days. Shipping is free over $50. Our hours are 9–6 PT..." Why will this chunk be a weak match for "how long do refunds take?" — and what's the fix?
4. You set chunk size to 800 tokens with 0% overlap and users complain that answers spanning section breaks are wrong. What single setting do you change first?

→ (1) At 64 tok precision wins (context is starved); at 1500 tok context completeness wins (precision is diluted); they cross in the ~200–500 band. (2) Chunk per Q&A pair — each is one complete idea; fixed character counts would split a question from its answer or merge two unrelated Q&As (dilution). (3) It mixes four topics, so its vector is a muddy average — refunds is only ¼ of the signal. Fix: split by topic so the refund sentence gets its own sharp chunk. (4) Add 10–20% overlap so cross-boundary ideas survive in at least one chunk.

• "Chunking is a trivial preprocessing detail." It's the single biggest, most model-agnostic lever on RAG quality — and it sets a ceiling the rest of the pipeline can't exceed.
• "Bigger chunks are safer — more context." Bigger chunks dilute the embedding (one vector ≈ an average) and degrade generation past ~2,500 tokens. More text ≠ better retrieval.
• "Smaller chunks are always more precise, so go small." Tiny fragments lose context and can't answer narrative questions. It's a tradeoff, not a direction.
• "Pick the optimal chunk size." There isn't one — the right size is query-dependent. One fixed size is always a compromise (hence multi-scale / parent-child later).
• "Overlap is wasted duplication." Overlap (10–20%) rescues ideas that straddle boundaries; skipping it silently drops cross-boundary answers.
• "Split every N characters." Split on ideas/topic boundaries — aim for semantic coherence (one idea per chunk), not a fixed count.

• A chunk is the atomic unit of retrieval — you only ever retrieve whole chunks, so chunking sets the ceiling on RAG quality. A fact split across chunks can never be retrieved whole (Failure #7).
• Goldilocks: too small = sharp but context-poor; too big = rich but diluted (and >2,500 tok degrades generation); ~200–500 tokens is a common sweet spot — a starting point, not a law.
• Embedding dilution is the deep mechanism: 1 chunk → 1 vector ≈ an average. Multi-topic chunks land in a muddy middle and match nothing well. One idea per chunk keeps the vector sharp.
• Overlap 10–20% rescues ideas that straddle boundaries; one fixed size is always a compromise (queries want different sizes — a single size leaves 20–40% recall gaps).
• The principle: chunk to ideas, not character counts — optimize for semantic coherence.
• Next: the concrete methods — fixed, recursive, and sentence chunking — and exactly how each respects (or butchers) those idea boundaries.