Multimodal RAG & Pipelines

This is the finale of the Multimodal AI Engineering section — and a finale's job is to assemble. Across the last five lessons you taught a model to see (L250 Vision & VQA), parse documents (L251 Document AI), hear (L252 Audio: STT & TTS), converse (L253 Voice Agents), and create (L254 Image & Video Generation). Each is a capability. None of them, alone, lets a user ask "what does our Q3 deck say the fastest-growing region was?" and get a trustworthy answer grounded in your PDFs, screenshots, and recordings.

Multimodal RAG is the architecture that makes that work. It takes the RAG pattern you learned earlier in the course — retrieve relevant context, then generate an answer grounded in it — and breaks it out of text-only. The corpus is no longer just paragraphs; it's charts, tables, diagrams, scanned pages, photos, and audio. And the answer comes from a vision-language model (VLM) that can actually read the retrieved image (L250), not just a string.

The single most important idea in this lesson is uncomfortable for text-RAG veterans: on real-world documents, the answer is often in the pixels, not the text. A revenue figure lives in a chart's bar heights; a spec lives in a table's layout; an instruction lives in a diagram. OCR-then-chunk throws that away. Multimodal RAG keeps it.

In this lesson:

• Why text RAG breaks on charts, tables, and layout — the motivating failure.
• The multimodal RAG loop — the end-to-end pipeline, and how it reuses the whole section.
• Three retrieval architectures — caption-and-index, unified embeddings, and page-as-image (ColPali).
• Embedding strategy — the modality gap and the 2026 model landscape.
• Late interaction (ColPali) — the SOTA for visually-rich documents.
• Grounding a VLM — feeding retrieved evidence to Claude, and citing it.
• Audio, video, ingestion & evaluation — the rest of a production pipeline.

Scope: this is the capstone — it builds on the RAG section (chunking, embeddings, vector stores, retrieval, reranking, grounding & citations) and on L250–L254. It is the last Multimodal lesson, so it defers nothing within multimodal; the next section, The Frontier, turns to reasoning models and computer use.

Classic RAG over documents does this: run OCR (or a text extractor) on each page, chunk the text, embed the chunks, and retrieve. It works beautifully on a wall of prose. It fails quietly on the documents people actually care about — financial reports, slide decks, datasheets, research papers, forms — and it fails for one reason: OCR-then-chunk linearises a 2-D visual artifact into a 1-D string, and a lot of meaning lives in the geometry it discards.

• Charts: "Which region grew the fastest?" is answered by the relative heights of bars. OCR might scrape the axis labels ("APAC", "EMEA") and maybe a few numbers, but the relationship — which bar is tallest — is gone. There is often no text string that contains the answer at all.
• Tables: OCR flattens rows and columns into a run-on sequence; the link between "Q3" and "$4.2M" (which row, which column) gets scrambled, so the model grabs the wrong cell.
• Layout & figures: a callout box, an arrow in a diagram, a stamp, a signature, a highlighted row — all carry meaning that has no textual form.

The lab at the end makes this visceral: ask "which region grew fastest?" with Text-only RAG and the chart page doesn't even get retrieved — there's no text to match — so the model answers "I can't find that" or, worse, hallucinates a number. That single demo is the whole motivation for this lesson.

The fix is to stop destroying the visual signal: retrieve and reason over the document as it looks, not just the text you could scrape off it. Everything below is a way to do that.

Multimodal RAG is the same shape as text RAG, with every stage upgraded to handle pixels and audio as first-class citizens. End to end:

1. Ingest — collect the corpus: PDF pages (rendered as images), standalone images, and audio/video. Document parsing (L251) and STT (L252) feed this stage.
2. Embed — turn each item into a vector (or vectors). Which embedding strategy you pick — caption, unified, or page-image — is the central design decision (next section).
3. Index — store the vectors in a multimodal vector store, ideally under a single namespace per document so a page's caption, alt-text, and raw pixels share an id.
4. Retrieve — embed the query and pull the top-k most relevant items.
5. Rerank (optional but valuable) — a cross-encoder/reranker reorders the candidates for precision (the same reranking idea from the RAG section, now over mixed media).
6. Generate (the multimodal twist) — feed the retrieved evidence — including the actual images — to a VLM (Claude, L250), which reads the chart/table/page and writes a grounded answer.
7. Cite — surface where the answer came from (page, region, bounding box), because a VLM that reads a chart can also misread it.

Notice how this lesson is the section: ingestion leans on L251/L252, the generator is the VLM from L250, and if the system needs to produce an image rather than find one, that's L254. Multimodal RAG is the glue that turns five capabilities into one product.

Everything hinges on step 2: how do you turn a chart-laden page into something retrievable? In 2026 three architectures dominate, in increasing order of visual fidelity (and cost):

1. Caption-and-index (the simplest). Run a VLM to caption each image/page ("a bar chart of regional revenue growth"), then embed the caption text with an ordinary text embedder and do normal text RAG. Pros: trivial to add to an existing text stack, cheap, any text vector DB works. Cons: caption drift — the caption is a lossy summary that rarely contains the specific fact a future query needs. Your caption said "a chart of regional growth"; the query asks "which region grew fastest?"; the caption never recorded the answer. Good for coarse retrieval, bad for precise facts.

2. Unified (native multimodal) embeddings. Use a single model that embeds both text and images into one shared vector space (the CLIP idea, productionised: Cohere Embed v4, Voyage Multimodal 3.5, Jina CLIP v2, Gemini Embedding). A text query retrieves images directly — no captioning step, no second hop. Pros: true cross-modal retrieval, one index, strong general quality. Cons: a page is squeezed into one vector, so very fine detail (which of four bars is tallest) can get blurred; you still feed the retrieved image to a VLM to read the exact value.

3. Page-as-image with late interaction (ColPali) — SOTA for visual docs. Treat each page as an image and keep patch-level vectors instead of pooling to one; score with late interaction (MaxSim). No OCR at all. Pros: preserves layout, tables, charts, fonts — the best retrieval on visually-rich documents. Cons: multi-vector storage is heavier; it's a specialised retriever. (Deep-dived two sections down.)

A pragmatic 2026 default: unified embeddings for a general image+text corpus; page-image (ColPali) when the documents are visually rich (finance, scientific, forms); caption only when you must bolt onto a pure-text stack. The lab lets you watch all three succeed and fail on the same query.

Whatever architecture you choose, the embedding model decides retrieval quality. Two ideas matter.

The modality gap. In a shared space, text and image embeddings don't perfectly overlap — they tend to cluster in separate regions, with a gap between them. A bigger gap means a text query lands far from the matching image and cross-modal retrieval suffers. Good multimodal embedders are trained (contrastively) to shrink that gap so "a red sports car" lands near photos of red sports cars. Storing modalities in separate vector spaces and trying to compare across them is the classic mistake — it produces retrieval drift. Keep one joint space.

The 2026 model landscape (verify — this moves fast):

• Gemini Embedding (Google) — the strongest all-modality embedder; the default if you search across text, images, and video.
• Cohere Embed v4 — top of the MTEB-Multimodal leaderboard; excellent for text+image enterprise search.
• Voyage Multimodal 3.5 & Jina Embeddings v4 — strong, and they keep ~99% quality at 256 dims (Matryoshka), which slashes storage.
• SigLIP-2 (ViT-SO400M) — the strongest open image-text model; Jina CLIP v2 adds multilingual and an 8192-token context (whole-document embedding).
• The original CLIP / OpenCLIP still anchor the open ecosystem; ImageBind binds six modalities (incl. audio) into one space.

Choosing: for cross-modal search across many modalities, reach for Gemini Embedding or Cohere Embed v4; for open/self-hosted, SigLIP-2 or Jina; and remember dimensionality is a real cost lever — Matryoshka models let you truncate to 256-d with little loss. (This is the multimodal extension of the embedding-selection lesson from the RAG section.)

ColPali (and its stronger successor ColQwen) is the technique that made "just embed the page as an image" state-of-the-art — and it's worth understanding because it inverts how you think about document retrieval.

The pipeline has no OCR. Each PDF page is rendered to an image and fed through a vision-language backbone (ColPali uses a SigLIP vision encoder; ColQwen uses Qwen2-VL/Qwen3-VL). The page is cut into a grid of patches — ColPali produces 1,024 patch embeddings per page (a 32×32 grid), each a compact 128-dim vector. The query text is likewise embedded into a handful of token vectors.

Late interaction (MaxSim) is the scoring trick — inherited from ColBERT. Instead of pooling the page into a single vector and taking one dot product, you keep all the patch vectors and, for each query token, find its maximum similarity to any patch on the page, then sum those maxima. Intuitively: "does some region of this page strongly match each part of my query?" That's why it nails fine-grained, localised content — a single cell, a tiny axis label, the tallest bar — that a single pooled vector would average away. It's the SOTA on ViDoRe, the visual-document-retrieval benchmark.

The trade-off is storage and compute: ~1,000 vectors per page instead of one. You manage it with vector DBs that support multi-vector / late-interaction indexes (and pooling tricks). For visually-rich, layout-heavy corpora, that cost buys retrieval quality nothing else matches. In practice you reach for a wrapper rather than implementing MaxSim yourself:

Retrieval found the right page; now the "G" in RAG has to read it. This is where the section comes full circle: the generator is the vision-language model from L250. You pass the retrieved images (and any text) directly into the model's context and ask it to answer using only that evidence — and to cite where it read the answer. Because Claude is a text + vision model, it's a natural fit as the grounded generator (it also makes a fine captioner for the caption-and-index path):

Three rules make grounded generation trustworthy:

• Pass the pixels, not a caption. The whole point is that the model reads the actual chart. A caption in the prompt re-introduces caption drift.
• Constrain to the evidence and forbid guessing. "Using only the page above… if it isn't shown, say you can't find it." This is the grounding/citation discipline from the RAG section, applied to vision — and it's the main defence against the failure mode below.
• Always cite — VLMs hallucinate chart numbers. A model that can read a bar chart can also misread one. Make it point to the page and region (a bounding box is ideal) so a human — or an eval — can verify. The lab draws exactly this: the citation box on the tallest bar.

Audio & video join the same loop with one extra choice per modality:

• Audio: the pragmatic path is transcribe-then-text — run STT (L252) and retrieve over the transcript (cheap, and it composes with your text stack). Use native audio embeddings only when non-lexical signal matters (speaker, tone, a sound effect) that the transcript loses.
• Video: index keyframes (treat them as images) and/or the transcript, or use a native video embedder (e.g. TwelveLabs-style models) for true motion/temporal search. A query then retrieves the right clip or frame, which you feed to a VLM.

Ingestion & indexing at scale is where pipelines live or die:

• Chunk by document structure, not blindly: a page, a figure, a table, a slide, a 30-second audio window — keep semantically whole units.
• One namespace per document: a page's image, its caption, its extracted text, and its transcript should share a document id so retrieval can return the page with all its evidence, not orphaned fragments.
• Mind the storage math: single-vector (unified) indexes are cheap; multi-vector (ColPali) indexes are ~1000× the vectors per page — budget for it, and use Matryoshka truncation / pooling where you can.
• Freshness: when a document changes, its embeddings go stale; wire re-embedding into your ingestion so the index doesn't drift from the source.

A realistic production system is hybrid: BM25/keyword for exact strings (codes, SKUs), unified embeddings for general cross-modal recall, and ColPali for the visually-rich subset — fused and reranked. Same toolbox as the RAG section, now spanning modalities.

You cannot trust a multimodal RAG system you haven't evaluated per modality — a text-RAG benchmark says nothing about whether you can read a chart. Build dedicated visual test sets and measure the pipeline in three places:

• Retrieval: did the right page/figure/clip make the top-k? Report recall and precision per modality (text vs table vs chart vs image) — an aggregate number hides that you're blind to charts. Benchmarks: ViDoRe (v1/v2/v3) for visual document retrieval, UNIDOC-BENCH and VisRAG for end-to-end document RAG.
• Generation: given the retrieved evidence, did the VLM produce the correct, grounded answer? A chart-QA eval set (questions whose answers are only in figures) is essential.
• Grounding: is the citation right? Evaluate bounding-box accuracy — does the model point to the region it actually used?

Watch for the named failure modes:

• Caption drift — captions miss the queried fact (the caption-and-index tax).
• Dominant-modality bias — the system over-retrieves text and ignores the figure that holds the answer.
• Modality leakage / retrieval drift — comparing across separate vector spaces returns near-irrelevant cross-modal hits.
• Chart-number hallucination — the VLM invents a value; only citations + a chart-QA eval catch it.
• Stale embeddings — the source changed; the index didn't.

And the flywheel from L246 (CI/CD for prompts & evals) applies unchanged: every production miss becomes a new eval case — now including the chart, table, or page that fooled you.

This lab is the whole lesson in one screen. The corpus is a 4-page report: a text paragraph (refund policy), a revenue table, a regional-growth bar chart, and an architecture figure. Pick a query — each one's answer lives in a different block — and a retrieval architecture, then watch the pipeline run end to end and the VLM's answer change.

Start on the chart query ("which region grew the fastest?") with Text-only RAG: the chart page doesn't even get retrieved, and the answer is missing — there's no text to match. Now click through the architectures: Caption→text retrieves the chart but drifts (the caption never recorded the winner), Unified embeddings retrieves it but is shaky on the exact bar, and Page-image (ColPali) retrieves it at the top and the VLM reads the bars and cites the box → APAC, +38%. Then try the text and table queries to see that for easy content, every approach works — the differences explode on the chart.

Notice three things. One: for the chart query, only page-image reliably gets the evidence and the answer — text-only is blind and captioning drifts. Two: retrieval and generation are separate wins — you can retrieve the right page and still answer wrong if the evidence was lossy. Three: the citation (the green box on the tallest bar) is what lets you trust the number — without it, you can't tell a read from a hallucination.

Predict first, then check in the lab (or reason it out).

1. Your text-RAG system over scanned financial PDFs answers prose questions well but is "randomly wrong" on anything involving a number from a chart or table. What's happening, and what's the minimal-change fix vs. the best fix?

2. A teammate proposes: "Let's caption every image with a VLM and keep our existing text vector DB." When is that a good call, and what specific failure will it produce on a query like "what's the exact churn rate in the Q3 cohort chart?"

3. You switch to ColPali and retrieval gets great, but your vector storage bill explodes. Why — and name two levers to bring it down.

4. Your VLM confidently answers *"revenue was $5.1M"* citing a chart — but it was actually$4.2M. Retrieval was correct. What broke, and what two guardrails would have caught it?

5. You need search across a corpus of text docs, product photos, and support-call recordings. Sketch the retrieval design (which embedding strategy per modality, one index or many).

Answers.

1. OCR-then-chunk discards the visual signal — the answer (a bar height, a table cell) has no text form, so retrieval misses it or the cell is scrambled. Minimal fix: add a caption-and-index pass so charts are at least retrievable (accepting caption drift). Best fix: page-as-image (ColPali) + a VLM that reads the actual page and cites it.

2. Captioning is a good call when you must bolt onto a pure-text stack and only need coarse retrieval ("find the slide about churn"). It fails on precise facts: the caption "a chart of cohort churn" never recorded the exact rate, so the query can't match it and the VLM has nothing to ground on — caption drift.

3. ColPali keeps patch-level multi-vectors (~1,000 vectors per page instead of one), so the index is orders of magnitude larger. Levers: Matryoshka/dimension truncation (e.g. 128→fewer dims), token/patch pooling to fewer vectors per page, and using ColPali only for the visually-rich subset while cheaper unified embeddings cover the rest.

4. Generation broke, not retrieval: the VLM hallucinated/misread the chart number. Guardrails: (a) require a citation / bounding box so the claim is checkable, and (b) a chart-QA eval set that would have flagged the model's number-reading error before ship — plus prompting it to quote the value and refuse if unsure.

5. Use one joint (unified) vector space so a text query can hit all three: embed text docs and photos with a unified multimodal embedder (Cohere Embed v4 / Gemini Embedding); for call recordings, transcribe with STT (L252) and embed the transcript into the same index (add native audio embeddings only if tone/speaker matters). One namespace per asset; retrieve → rerank → feed the hit (image or transcript snippet) to a VLM → grounded, cited answer.

• "Multimodal RAG is just RAG with images bolted on." → Every stage changes: you embed pixels, retrieve across a modality gap, and generate with a VLM that reads the evidence. The architecture choice (caption / unified / page-image) is the whole game.
• "OCR the PDF, then it's a normal text-RAG problem." → OCR destroys charts, tables, and layout — the answer is often in pixels with no text form. Retrieve over the page as an image.
• "Captioning images is good enough." → Captions are lossy summaries → caption drift. Fine for coarse retrieval; useless for the specific fact a query needs.
• "Store text and image vectors in separate indexes and compare them." → That causes retrieval drift across the modality gap. Use one joint space (a unified multimodal embedder).
• "ColPali just OCRs the page better." → ColPali does no OCR. It keeps patch-level vectors and scores with late-interaction MaxSim, matching query tokens to page regions.
• "If the VLM read the chart, the number is right." → VLMs hallucinate chart numbers. Always cite the region and keep a chart-QA eval; retrieval being correct doesn't make generation correct.
• "Aggregate retrieval recall tells me it works." → Measure per modality — a great text recall can hide that you're blind to charts. And evaluate generation and grounding, not just retrieval.
• "Claude can be the whole system." → Claude is the VLM generator (and a captioner) — the G in RAG. It still needs a retriever + index in front of it to ground answers in your corpus.

• Multimodal RAG = retrieve across text, tables, charts, images, audio, then ground a VLM's answer in what was found. It's the finale that turns L250–L254 into one product: ingest with L251/L252, generate with the L250 VLM, optionally create with L254.
• Text RAG goes blind on real documents: OCR-then-chunk discards the geometry where the answer often lives (a bar height, a table cell, a diagram). Retrieve over the page as it looks.
• Three architectures: caption-and-index (cheap, caption drift), unified embeddings (one shared text+image space — Cohere Embed v4, Gemini Embedding, Voyage — true cross-modal search), and page-image + late interaction (ColPali/ColQwen) (patch vectors + MaxSim, no OCR, ViDoRe SOTA on visually-rich docs).
• Mind the modality gap: keep one joint vector space; separate spaces cause retrieval drift. Dimensionality (Matryoshka) is a real cost lever.
• Generation grounds in pixels: feed the retrieved image to a VLM (Claude), constrain it to the evidence, forbid guessing, and always cite (a bounding box) — VLMs hallucinate chart numbers.
• Audio/video join via transcribe-then-text (L252) or native embedders; ingest with structure-aware chunking, one namespace per document, and re-embed on change.
• Evaluate per modality (ViDoRe, UNIDOC-BENCH, chart-QA) across retrieval, generation, and grounding — and feed every miss back as an eval case (L246).
• Section complete. You can now make a model see, read, hear, speak, create, and retrieve-and-ground across all of them. Next — L256 begins The Frontier (Reasoning Models & Test-Time Compute): we leave perception and turn to how models think.