Ingestion: Loading & Parsing Documents

Introduction

The last lesson gave you the full RAG map. Now we build the first box — Ingestion, the Load & Clean stage. Its one job: turn your messy source files (PDFs, web pages, Word docs, scans, spreadsheets) into clean, structured, traceable text + metadata, ready to chunk.

This box is the least glamorous and most under-invested part of RAG — and that's exactly why it quietly sinks so many projects. Here's the iron law: parsing quality caps RAG quality. The best embedding model and the fanciest reranker cannot fix text that was garbled at ingestion. Garbage in → garbage retrieved. An afternoon improving your parser often beats weeks of tuning retrieval.

In this lesson you'll learn:

Why a PDF is a layout format, not a text format (the root of all the pain)
The 2026 split: heuristic parsers vs vision/VLM parsers — and how to choose
Why you parse to Markdown, and how to handle tables, OCR, and the web
The two things you must do every time: extract metadata and clean the text

The Core Problem: A PDF Is a Picture, Not a Paragraph

Beginners assume a PDF contains text. It mostly doesn't — not in the way you'd hope. A PDF is a layout format: it stores "draw glyph 'A' at coordinate (72, 640)" thousands of times. There's usually no notion of "this is a heading," "this is a paragraph," "this row belongs to that table." That structure is visual — obvious to your eye, invisible to a naive extractor.

So when you dump the raw text, you get classic failures:

Scrambled reading order in multi-column layouts (it reads straight across both columns).
Shattered tables — cells flattened into a meaningless stream of numbers.
Headers, footers, and page numbers injected into the middle of sentences.
Hyphenated line breaks (inter-\nnational) splitting words in two.
Scanned PDFs that are just images — naive extraction returns nothing.

Every one of these becomes a bad chunk, which becomes a bad retrieval, which becomes a wrong answer. The entire job of ingestion is to recover the structure a human sees and emit clean text. HTML, DOCX, and slides have their own versions of this mess (nav bars, comments, speaker notes). Same principle everywhere.

An infographic titled 'Ingestion — From Messy Files to Clean, Structured Text'. A top strip of MESSY SOURCES shows PDF, HTML, DOCX/PPTX, Scanned, Tables, and Email/code. An arrow leads to step one, CHOOSE A PARSER — match it to the document, because a PDF is a layout format, not text. Two options sit side by side. Rule-based / heuristic (fast, cheap, deterministic): PyMuPDF, pdfplumber, MarkItDown, Unstructured, Tika — best for clean, digital, simple-layout docs and bulk scale. Vision / VLM (reads the page like a human, costs money, slower): LlamaParse, Docling, Marker, Azure Document Intelligence, Reducto — best for complex layout, tables, multi-column and scanned documents via OCR. An arrow leads to step two, THEN ALWAYS produce clean, structured, traceable text, with three cards: Parse to Markdown (keep headings, lists and tables intact so chunking and retrieval use the structure); Extract metadata (source, page number, section, date — powers citations and filters, and you must capture it now because you can't recover it later); and Clean and normalize (strip headers and footers, de-hyphenate line breaks, fix encoding, drop boilerplate). The output is clean structured text plus metadata, ready to chunk — the next box in the pipeline. A banner reads: parsing quality caps RAG quality — garbage in, garbage retrieved; an afternoon on your parser beats weeks tuning retrieval.

The 2026 Split: Heuristic vs Vision Parsers

Document parsing has split into two paradigms — and picking the right one for each document is the core skill:

🛠️ Rule-based / heuristic — fast, cheap, deterministic, runs locally. Tools: PyMuPDF (fast), pdfplumber (tables), MarkItDown, Apache Tika, Unstructured (30+ formats). They use format-specific heuristics. ✅ Great for clean, digital, simple-layout documents and bulk scale — millions of pages cheaply.

👁️ Vision / VLM — a vision-language model "reads" the page like a human. Tools: LlamaParse (cleanest Markdown from visually complex docs), Docling (IBM, open-source, strong on tables/layout), Marker (great on academic papers), Azure Document Intelligence, Reducto. They cost money and run slower, but they handle multi-column, complex tables, figures, and scanned pages (OCR) that heuristics mangle.

The decision is not "which tool is best" — it's "which tool fits this document." Clean digital text → a cheap heuristic extractor. Complex layout / tables / scans → a vision parser. Paying for a VLM on a clean text file is waste; using naive extraction on a scanned multi-column report is malpractice. Many production pipelines route documents: cheap path by default, VLM path for the hard ones.

# 2026 default: parse to Markdown so structure (headings, tables) survives.

# --- A) Clean, digital PDF → cheap & fast heuristic extraction ---
import pymupdf  # PyMuPDF
doc = pymupdf.open("handbook.pdf")
pages = []
for i, page in enumerate(doc):
    pages.append({
        "text": page.get_text("text"),          # plain text in reading order
        "page": i + 1,                            # ← metadata you can NEVER recover later
        "source": "handbook.pdf",
    })

# --- B) Complex layout / tables / scanned → vision/VLM parser → Markdown ---
from llama_cloud_services import LlamaParse
md_docs = LlamaParse(result_type="markdown").load_data("annual_report.pdf")
markdown = md_docs[0].text   # headings as #, tables as | … | … |, lists preserved

Parse to Markdown (Not Plain Text)

Notice both code paths aim for Markdown, not a flat blob of text. This is the 2026 default output format, and the reason is downstream:

Markdown preserves the structure your later boxes depend on. Headings (#), lists, bold, and especially tables (| … | … |) survive as signal:

Chunking can split on heading boundaries instead of blindly every N characters → semantically whole chunks.
Retrieval benefits because a heading like "## Refund Policy" travels with its paragraph.
Tables stay intact instead of dissolving into number soup.

Plain text throws all of that away the moment you extract it — and you can't get it back. (The richer alternative is structured JSON with layout coordinates and element types, which some parsers emit; Markdown is the pragmatic, model-friendly sweet spot.) Rule of thumb: parse to the most structured format your parser can produce, and preserve it as far down the pipeline as you can.

The Hard Parts: Tables, Scans & the Web

Three cases deserve special care because they're where pipelines silently break:

📊 Tables — the hardest thing in parsing. Naive extraction splits cells and scrambles rows, so "Q3 revenue: 4.2" might end up next to the wrong label. Use a layout-aware / vision parser (LlamaParse, Docling, Azure DI) that detects table boundaries and outputs a Markdown or JSON table. Then — critically — keep each table as a single chunk (never split mid-row), and consider adding a one-line natural-language summary of the table to help retrieval find it.

🖼️ Scanned documents & images → OCR. If the PDF is images of text, heuristic extractors return empty strings. You need OCR — built into Docling, Marker, Unstructured, Textract, Azure DI. Always spot-check: low-quality scans produce subtle character errors (l→1, O→0) that poison retrieval.

🌐 The web (HTML). A raw web page is mostly nav bars, ads, cookie banners, and footers. Extract the main content with a boilerplate-stripping tool (Trafilatura, readability, Firecrawl) — don't embed the navigation menu. Same idea for DOCX (drop comments/track-changes) and PPTX (pull speaker notes deliberately, not by accident).

The Two Things You Must Always Do

Whatever parser you pick, every ingestion pipeline must do two more things before the text moves on:

1. 🏷️ Extract metadata — and do it now. For each piece of text, capture: source/filename, URL, title, page number, section/heading, author, date, doc type. This is not optional housekeeping — it is what later powers:

Citations ("according to handbook.pdf, p. 12") — the next lesson.
Metadata filtering (WHERE tenant_id=… AND lang='en') — Section 2.
Recency / authority ranking and debugging.

⚠️ You can capture the page number at parse time. You cannot recover it once the text is chunked and embedded. Metadata you don't grab now is gone forever.

2. 🧹 Clean & normalize. Fix what the format mangled, before chunking:

import re

def clean(text: str) -> str:
    # 1. De-hyphenate words split across a line break:  "inter-\nnational" → "international"
    text = re.sub(r"(\w+)-\n(\w+)", r"\1\2", text)
    # 2. Join soft-wrapped lines, but keep real paragraph breaks (blank lines)
    text = re.sub(r"(?<!\n)\n(?!\n)", " ", text)
    # 3. Collapse runs of whitespace
    text = re.sub(r"[ \t]+", " ", text)
    return text.strip()

# Also: strip repeated headers/footers & page numbers (lines that appear on most pages),
# drop nav/cookie boilerplate from HTML, and fix mojibake (encoding) BEFORE chunking.

De-hyphenate split words, join soft-wrapped lines (while keeping real paragraph breaks), strip repeated headers/footers/page numbers, remove boilerplate, and fix encoding (mojibake like â€™ → '). Cleaning is unglamorous and enormously high-leverage.

Do It Once, Do It Incrementally — and Don't Over-Engineer

Two practical guardrails:

Make ingestion idempotent and incremental. Your corpus changes — docs get added, edited, deleted. Hash each document's content and use deterministic IDs, so re-running ingestion re-processes only what changed (and updates rather than duplicates). This is the same lifecycle discipline from the index-maintenance lesson, applied at the front door. Re-parsing your entire corpus on every run wastes time and money.

Don't over-engineer. A folder of clean .md or .txt files needs a loader, not a vision model. Reach for VLM parsing where it pays off — complex layouts, tables, scans — not by default. The art is matching effort to the document.

A pragmatic 2026 starting stack: Unstructured or PyMuPDF for the easy 80%, a vision parser (LlamaParse / Docling) routed in for the hard 20%, parse-to-Markdown, metadata captured per element, and a cleaning pass. Measure parse quality on a sample of your real documents before you trust it at scale.

🧪 Try It Yourself

Be the ingestion engineer — for each document, name the parser path and the one thing most likely to break:

50,000 clean, digitally-generated invoices, all the same simple template. → parser? cost concern?
A 1990s scanned engineering manual (images of text) with multi-column pages and dense tables. → parser? two risks?
A folder of internal .md runbooks already in Markdown. → parser?
Live product docs on a marketing website with nav bars, cookie banners, and a chat widget. → what must you strip, and what metadata must you keep?

→ 1: heuristic (PyMuPDF/Unstructured) — clean digital + huge volume means a VLM would be needlessly expensive; keep invoice_id, date. 2: vision/VLM with OCR (Docling/Marker/Azure DI) — risks: OCR character errors (O/0) and shattered tables, so spot-check and keep tables as single chunks. 3: a plain Markdown loader — don't over-engineer; the structure's already there. 4: strip nav/cookie/chat boilerplate (Trafilatura/Firecrawl) and keep URL, page title, and last-updated date as metadata for citations + recency. The skill is matching the parser to the document — never one tool for everything.

Interactive: an ingestion parser-matching game. Five documents appear one at a time with their attributes (50,000 clean digital invoices on one template; a 1990s scanned multi-column manual with dense tables; a folder of Markdown runbooks; live product docs on a marketing site with nav bars and cookie banners; a financial report of complex nested tables), and the user picks the parser path — Loader, Heuristic (PyMuPDF/Unstructured), or Vision/VLM (LlamaParse/Docling). A correct pick explains why it fits; a wrong pick shows exactly what breaks — empty or garbled text when a heuristic parser hits a scan, ~100× wasted cost when a vision model is used on bulk clean invoices, scrambled table rows, or boilerplate poisoning retrieval from a raw web loader — then reveals the right path. A running score tracks accuracy across all five. The lesson's core skill made tangible: there is no universal parser — match it to each document, parse to Markdown, and capture metadata now, because parsing quality caps RAG quality. Deterministic.

Mental-Model Corrections

"A PDF contains text." It contains a layout — glyphs at coordinates. Structure (headings, tables, reading order) must be recovered, and naive extraction often fails to.
"Any extractor is fine." No — match the parser to the document. Heuristic for clean/simple/bulk; vision/VLM for complex layout, tables, and scans.
"Parse to plain text." Parse to Markdown (or JSON) so structure survives for chunking and retrieval — especially tables.
"Metadata is optional / I'll add it later." Capture source, page, section, date at parse time — it powers citations and filters, and page numbers can't be recovered after chunking.
"VLM parsers are always best." They're slower and cost money — overkill (and wasteful) on clean digital docs. Route the hard 20% to them, not everything.
"Ingestion is a one-time script." Make it idempotent and incremental (content hashes, deterministic IDs) — your corpus keeps changing.

Key Takeaways

Ingestion = the Load & Clean box. Its output is clean, structured, traceable text + metadata, ready to chunk — and its quality caps the whole RAG system. Garbage in → garbage retrieved.
A PDF is a layout format, not text. Expect scrambled reading order, broken tables, headers in the body, and empty scans.
2026 split: heuristic parsers (PyMuPDF, pdfplumber, MarkItDown, Unstructured — fast/cheap/bulk) vs vision/VLM parsers (LlamaParse, Docling, Marker, Azure DI — complex layout, tables, OCR). Match the parser to the document.
Parse to Markdown, handle tables (keep them whole) and scans (OCR + spot-check) deliberately, and strip web/Office boilerplate.
Always extract metadata at parse time (source/page/section/date) and clean/normalize before chunking. Make ingestion incremental.
Next: the text is clean — now we retrieve, augment, and generate, building the query side of RAG from scratch.