RAG over Tables, PDFs & Images (Multimodal)

Introduction

Every technique in this entire container has quietly assumed one thing: that your knowledge lives in text. But open any real document — a financial report, a research paper, a product manual, a slide deck — and the answer is often not in the prose. It's in a revenue table, a trend chart, an architecture diagram, or a scanned page.

Text-only RAG is structurally blind to all of it. As you parse a PDF, the charts and figures get dropped, the tables get mangled into number-soup, and the layout that gave numbers meaning is gone — information is lost before retrieval even begins. And no chunking strategy can recover what was never indexed. As the research bluntly puts it: text-only RAG has an upper bound on figure-heavy corpora that no amount of tuning can fix.

Multimodal RAG retrieves and reasons over visuals as first-class content. In this lesson:

Why text-only RAG can't see tables, charts, and images
The three approaches: caption→text, multimodal embeddings, and page-image (ColPali)
Why generation needs a vision LLM — and how to handle tables
The honest cost, and when multimodal is worth it

An infographic titled 'Multimodal RAG — When the Answer Is in a Chart, Not the Text'. A red strip: real documents aren't pure text; charts, tables, diagrams, and scanned pages often carry the actual answer, but text-only RAG strips them out before retrieval even begins, and no chunking strategy fixes a figure-heavy corpus. THREE WAYS TO MAKE VISUALS RETRIEVABLE, as three cards. Caption to text: a VLM describes each image or table as text (caption or markdown), embed that text, then normal text RAG — the L95 summary trick applied to visuals; cheap and reuses text infra, but lossy because the caption may miss the detail. Multimodal embeddings: embed text plus images in ONE space (CLIP, voyage-multimodal, Cohere, SigLIP-2), so a text query retrieves the right image directly — true cross-modal search; simple with modest storage, but single-vector loses on the hardest figures and risks misalignment. Page-image (ColPali): treat each PDF page as an image and have a VLM make multi-vector embeddings (late interaction), skipping OCR, parsing, and chunking while capturing layout and figures; best on figure-heavy docs, but storage-heavy at about 1.3 gigabytes versus 50 megabytes per 10,000 pages. A strip: generation needs a vision LLM (GPT-class, Claude, Gemini, Qwen-VL) to actually read the retrieved chart, table, or page; and tables are hardest — keep a table whole and summarize it, and for exact numbers or aggregates route to text-to-SQL. A caveat: multimodal is pricier (images cost more to store, encode, and generate over) and cross-modal embeddings can misalign; it's worth it for visually-rich docs like slides, financial reports, blueprints, and dense PDFs, but overkill for pure text. Banner: if the answer lives in a chart, table, or scanned page, text-only RAG can't see it — embed the pixels (or describe them) and let a vision LLM read.

The Blind Spot: Answers Live in Pixels

Picture one page of an annual report: a paragraph of policy text, a table of quarterly revenue, and a bar chart of growth. Now three questions:

"How long do refunds take?" → the paragraph. Text RAG nails this.
"What was Q3 revenue?" → the table. Naive parsing scrambles the cells; the number may be lost or misaligned.
"Which quarter grew fastest?" → the chart. There is no text to retrieve — the answer is visual, requiring you to look at the bars. Text RAG returns nothing useful.

That third question is the heart of it: the answer exists, plainly, on the page — and a text index has zero representation of it. Multimodal RAG closes this gap two ways: either make the visual retrievable (so the right chart/table/page comes back), and/or let the generator see the visual (so it can read the bars). You'll usually need both. The interactive at the end lets you run exactly these three queries against each approach.

Approach 1 — Caption → Text

The simplest, cheapest bridge — and you already know the pattern from L95 (multi-vector indexing): index a text proxy, return the original. Here, a vision LLM (VLM) describes each visual as text — a caption for an image, a markdown rendering + summary for a table — and you embed that description. Retrieval is plain text RAG; at generation you pass the description and/or fetch the original image.

# APPROACH 1 — Caption → text (the L95 multi-vector trick, applied to visuals).
# A vision LLM DESCRIBES each non-text element; you embed the DESCRIPTION, return the ORIGINAL.
def describe(element):                       # element = an image, chart, or table crop
    return vlm("Describe this in detail for search. If it's a table, output markdown. "
               "If a chart, state the trend and key values.", image=element)

for el in extract_elements(pdf):             # text stays text; images/tables get a text proxy
    proxy = el.text if el.kind == "text" else describe(el)
    index.add(embed(proxy), payload={"proxy": proxy, "original": el})   # search proxy, keep original
# ✅ reuses your text retriever  ⚠️ lossy — the caption may omit the exact number you need

Why it's attractive: it reuses your entire existing text stack — same embedder, same vector DB, same reranker. You're just turning pictures into words at index time.

The catch is lossiness. A caption is a compression of the image: if the VLM's description says "a bar chart of quarterly revenue" but omits that "Q3 spiked 58%," then the exact answer is gone — you can retrieve the chart's description but not its detail. Caption→text is a great first step for moderately-visual corpora, but it has a ceiling.

Approach 2 — Multimodal Embeddings

Instead of converting images to text, embed images and text into the same vector space so you can search across modalities directly. This is the CLIP idea, now matured: models like voyage-multimodal-3.5 (2026), Cohere Embed (multimodal), SigLIP-2, and Jina CLIP embed a picture and a sentence into one space where a text query can retrieve a relevant image (and vice-versa).

# APPROACH 2 — Multimodal embeddings: text AND images in ONE vector space.
from your_provider import embed_multimodal     # e.g. voyage-multimodal-3.5, Cohere, CLIP, SigLIP-2

index.add(embed_multimodal(image=chart_png))   # an image embedded directly…
index.add(embed_multimodal(text=paragraph))    # …alongside text, in the SAME space

hits = index.search(embed_multimodal(text="which quarter grew fastest?"))  # text query → image hit!
# ✅ true cross-modal retrieval  ⚠️ single-vector models trail multi-vector on the hardest figures

# APPROACH 3 — Page-image / ColPali: embed each PAGE as an image (multi-vector, late interaction).
# Skips OCR/parsing/chunking entirely; the page picture preserves tables, charts, and layout.
page_vectors = colpali.embed(render_pdf_pages(pdf))   # ~1030 vectors/page → storage-heavy

Now "which quarter grew fastest?" (text) can land directly on the chart image — true cross-modal retrieval, with no captioning step and no information thrown away at index time.

The honest tradeoffs: single-vector multimodal embeddings are simple and modest in storage, but they lose to multi-vector late interaction on the hardest, most figure-dense queries — and when the two modalities don't truly share a semantic space, you get cross-modal misalignment (a text query returning an unrelated image). Multimodal embedding quality has historically trailed best-in-class text embeddings, though the 2026 models have closed much of the gap.

Approach 3 — Page-Image Retrieval (ColPali)

The most radical — and most distinctly 2026 — approach: stop parsing the document at all. Treat each PDF page as a screenshot (an image), and embed the whole page with a vision model. The leading method is ColPali (Contextualized Late Interaction over PaliGemma) and its successors ColQwen / ColQwen2.5: they produce multi-vector embeddings over the page's image patches and score with MaxSim late interaction — exactly the ColBERT mechanism from L98, applied to pixels.

The payoff is striking: because the model looks at the page like a human, it captures text, tables, charts, and spatial layout in one shot — and you skip the entire brittle pipeline of OCR, table parsing, and chunking (L86). You render pages → embed → retrieve the most relevant page images → feed them to a vision LLM. For complex, figure-heavy documents where parsing always breaks, this is often the highest-quality option.

The price is storage and speed: a ColPali page is ~1,030 vectors × 128 dims — for 10,000 pages, ≈1.3 GB of vectors vs ~50 MB for plain text embeddings — and MaxSim scoring is slower than single-vector ANN (expect 200–500ms at 100k pages unoptimized). It's the accuracy-for-storage trade, and the right call when documents are too visual to parse reliably.

Generation & the Special Case of Tables

Retrieval is only half the system. The generator must be able to see — a vision LLM (GPT-class, Claude, Gemini, Qwen-VL, Llama Vision) that reads the retrieved chart/table/page image alongside any text and reasons over it. You retrieved the bar chart; now the model has to look at the bars to answer "which quarter grew fastest?"

# GENERATION — a VISION LLM must actually READ the retrieved visual to answer.
def answer(query, hits):
    images = [h.original for h in hits if h.kind in ("image", "table", "page")]
    texts  = [h.proxy    for h in hits if h.kind == "text"]
    return vlm(f"Answer using the provided text and images.\nText:\n{texts}\n\nQ: {query}",
               images=images)        # GPT-class / Claude / Gemini / Qwen-VL reads the chart itself

# Tables needing EXACT numbers or aggregates? Don't embed — ROUTE to text-to-SQL (L101):
#   "total revenue in H2" → extract table → text→SQL → SELECT SUM(...) → exact answer.

Tables deserve special handling — they're the hardest element in all of document AI. Three plays, often combined:

Keep a table whole (never split mid-row, from L86) and attach a text summary for retrieval.
Send the table as an image to a vision LLM for layout-faithful reading.
For exact numbers and aggregations, don't embed at all — route to text-to-SQL (L101 routing): extract the table into a structured form, translate the question to SQL, and compute the answer. "Total revenue in H2" needs a SUM, not a fuzzy similarity match. Embeddings approximate meaning; they don't do arithmetic.

See It: What Each Approach Can Retrieve

Here's the whole lesson in one widget. A single page holds a text paragraph, a table, and a chart. Pick a query — each one's answer lives in a different kind of content — and watch which retrieval approach can actually find it:

One page, three kinds of content (text, table, chart). Pick a query and see which retrieval approach can actually find the answer — text-only RAG is blind to the chart and mangles the table; multimodal approaches see them.

The pattern is the point: text-only RAG fails the chart entirely and mangles the table. Caption→text rescues both if the caption was good enough. Multimodal embeddings and page-image/ColPali truly see the visuals — at higher cost and storage. There's no single winner — there's a fit between your document type and the approach.

The Honest Take: When Is Multimodal Worth It?

Multimodal RAG is genuinely necessary for some corpora and expensive overkill for others. The discipline is the same as every advanced technique in this course: match the tool to a real failure.

The costs are real:

Storage & compute: images are larger to store, slower to encode, and consume more tokens at generation (a vision LLM reading a page costs more than reading a paragraph). Page-image retrieval (ColPali) can be ~25× the vector storage of text.
Cross-modal misalignment: multimodal embeddings can return irrelevant cross-modal results when the shared space is imperfect; the technology is improving but not flawless.
Harder evaluation: judging whether the model correctly read a chart is trickier than checking a text answer.

Reach for multimodal when documents are visually rich and the answers live in the visuals — financial reports, slide decks, blueprints, catalogs, scientific papers, dense PDFs full of tables and figures. For these, text-only RAG has a hard ceiling no chunking can break. But for pure-text, high-volume corpora (a wiki, a code base, chat logs), text-only is cheaper and just as good — don't pay the multimodal tax you don't need. And a pragmatic 2026 starting point: try caption→text first (cheap, reuses your stack); escalate to multimodal embeddings or ColPali only when your eval shows the captions are losing the answer.

🧪 Try It Yourself

Drive the widget, then choose approaches for real documents:

In the interactive, run the chart question against each approach. Which one(s) fail, and why is text-only fundamentally unable to answer it?
For the table question, why is plain text-only RAG unreliable — and what's the most precise way to answer "what was total H2 revenue?" specifically?
You're building RAG over 5,000 dense financial PDFs packed with tables and charts; parsing keeps breaking. Which approach, and what's the cost you accept?
Your corpus is a company wiki — almost entirely clean text, with the rare embedded screenshot. What do you use, and why not full multimodal?
Caption→text is retrieving the right charts but the model still gets exact values wrong. What's happening, and what do you change?

→ (1) Text-only fails — the chart is an image, so there's literally no text to index or retrieve; the answer requires looking at the bars (visual reasoning). Multimodal embeddings and page-image succeed; caption→text only if the caption captured the trend. (2) Naive parsing scrambles table cells; for an exact aggregate the precise play is text-to-SQL over the extracted table (compute a SUM), not embedding similarity. (3) Page-image / ColPali — it skips the parsing that keeps breaking and reads layout+figures directly; you accept higher storage (~25×) and slower MaxSim retrieval. (4) Text-only RAG (maybe caption→text for the rare screenshot) — full multimodal would be expensive overkill on an almost-pure-text corpus. (5) The caption is lossy — it described the chart but dropped the exact numbers; switch that content to multimodal embeddings or page-image so a vision LLM reads the actual values, or extract the data and use text-to-SQL.

Mental-Model Corrections

"RAG is about text." Real documents are visual — tables, charts, diagrams, scans often hold the answer, and text-only RAG drops them before retrieval.
"Just OCR/parse the PDF and it's text again." Parsing mangles tables and can't represent a chart's meaning; figure-heavy corpora have a ceiling no chunking fixes. Page-image (ColPali) skips parsing by treating the page as an image.
"Caption→text and multimodal embeddings are the same." Caption→text converts visuals to words (lossy, reuses text infra); multimodal embeddings put images and text in one space (true cross-modal, no captioning).
"Multimodal embeddings always beat text." They unlock visuals, but single-vector models lose on the hardest figures and can misalign across modalities. Page-image multi-vector is stronger but storage-heavy.
"Retrieving the chart is enough." You also need a vision LLM at generation to read it — and for exact numbers, text-to-SQL beats embeddings.
"Always go multimodal to be safe." It's pricier (storage, encoding, generation). Use it for visually-rich docs; for pure text, text-only is cheaper and as good.

Key Takeaways

Text-only RAG is blind to visuals — tables, charts, diagrams, and scans carry answers it strips out before retrieval. Figure-heavy corpora have a ceiling no chunking fixes.
Three approaches: caption→text (VLM describes visuals → embed text; cheap, reuses your stack, but lossy); multimodal embeddings (text + images in one space — CLIP/voyage/Cohere/SigLIP; true cross-modal, but single-vector trails on hard figures + misalignment risk); page-image / ColPali (page as image, multi-vector late interaction, skips OCR/parsing/chunking, captures layout — best on figure-heavy docs, but storage-heavy ~25×).
Generation needs a vision LLM (GPT-class/Claude/Gemini/Qwen-VL) to read the retrieved chart/table/page.
Tables are hardest: keep whole + summarize; for exact numbers/aggregates, route to text-to-SQL (embeddings can't do arithmetic).
Honest: multimodal is pricier (storage, encode, generation tokens) and can misalign. Worth it for visually-rich docs (financial reports, slides, blueprints, dense PDFs); overkill for pure text. Start with caption→text, escalate when your eval shows captions losing the answer.
Next: Agentic RAG — Corrective and Self-RAG, where the system reasons in a loop: retrieve, judge the results, and decide whether to retrieve again.