Document AI: OCR → Agentic Extraction

Last lesson ended at a wall: a single image of a dense document loses its fine print, because the provider caps a page's pixels and the small text falls below the model's effective resolution. This lesson is how you climb that wall — and it happens to be the highest-value vision application in production. Invoices, receipts, contracts, claims forms, lab reports, KYC documents, shipping manifests: turning a messy real-world document into clean structured data (JSON your code can act on) is a multi-billion-dollar problem, and modern AI has just rewritten how it's solved.

The naive version — ‘send the PDF to a model, ask for JSON’ — works in a demo and fails in production, because the failure mode is the worst kind: the model returns a confident, well-formatted, wrong number, and it flows straight into someone's ledger. The job of this lesson is to turn that demo into a system you can trust: extract to a schema, ground every value to the source, score per-field confidence, verify, and escalate what's ambiguous to a human — at a cost you control.

In this lesson:

• The spectrum of approaches — classic OCR → layout models → VLM → agentic — and when to use each
• Native PDF understanding — how a frontier model reads a document as hybrid image + text with citations
• Why naive extraction is dangerous — confident hallucination and the missing-field trap
• Grounding, per-field confidence, and the verify-then-escalate loop — the production pattern
• Human-in-the-loop, evals, and routing by difficulty to control cost

Scope: this builds directly on L250 (vision foundations — an image is tokens, and the dense-doc limit). It defers: audio (speech-to-text / text-to-speech) to L252; voice agents to L253; image / video generation to L254; and multimodal RAG (retrieving over a corpus of documents) to L255 — here we turn one document into structured data. It also builds on structured outputs (L232), evals (L246), and hallucination mitigation (L233).

‘Document AI’ isn't one technique; it's a ladder that trades cost for robustness. Knowing where to stand on it is half the job.

1. Classic OCR (Tesseract, AWS Textract, Google/Azure OCR). Converts pixels to characters. Battle-tested and cheap, and it gives you exact text — but on its own it ignores layout, reading order, tables, checkboxes, and handwriting, and it propagates its own recognition errors downstream.
2. Layout-aware models (the LayoutLM family, Donut). Pre-trained on documents to fuse text + 2D position + visual features, so they understand structure — which text is a header, a table cell, a form field. LayoutLMv3 can even go OCR-free, reading image patches directly.
3. Vision-language models (Claude, Gemini reading a PDF). The generalist from L250 pointed at a page: it reads the rendered document directly to JSON, robust to weird layouts, with no separate OCR step to error-propagate. This is where most teams now start — it's the fastest path from document to data.
4. Agentic extraction (the 2026 frontier — LandingAI's DPT-2, Reducto's multi-pass agentic OCR). Treats the document as a visual object and runs a loop: plan the fields, extract, verify each against the source, correct, and escalate the ambiguous — trading more compute for the reliability the others can't guarantee.

A useful split: OCR-based pipelines (OCR engine → text + layout → LLM) give you exact characters but inherit OCR's mistakes; OCR-free VLM pipelines read the image directly and are robust to messy layouts but can hallucinate. The best production systems combine them — OCR for the text it nails, a VLM for structure and the hard cases, and an agentic verify step to catch both.

The reason a frontier model is so good at documents is that it doesn't treat a PDF as just an image or just text — it reads it as both, per page. Each page is processed as an image (so it sees layout, tables, stamps, signatures, and handwriting) and as an extracted text layer (so it can quote exact passages and cite the page they came from). A single request can carry up to ~600 pages — though a dense PDF can fill the context window long before that, which is your real limit.

The two things that turn this from a chatbot trick into an extraction system:

• Constrain the output to a schema. Don't ask for prose — demand a JSON schema (structured outputs, L232) with exactly the fields you need, typed, so you get parseable data and the model can't ramble. Crucially, give every field an explicit null/legible escape so it can say ‘I couldn't read this’ instead of inventing a value.
• Ground it with citations. The Citations API makes the model attach, to each answer, the exact source passage and page it used — so a human (or a downstream check) can confirm the value against the document instead of trusting it blind. (Note: citations and the strict JSON-schema format are requested separately — many pipelines do a structured extraction pass and a grounded verification pass.)

Here's the workhorse extraction call — a PDF in, a typed schema out, with an explicit ‘unreadable’ escape on every field:

Send a clean digital invoice to a good model and you'll get ~95% of the fields right and feel great. Send a thousand real documents — scans, photos, faxes, smudges, rotated pages, three-column forms — and reality bites. Studies of LLM field extraction put per-field accuracy across realistic documents in the 62%–88% range. That alone isn't the problem; the shape of the errors is.

Two failure modes you must design around:

• Confident hallucination. When the model can't quite read the total — the digit is smudged, the scan is low-res — it does not say ‘I can't tell.’ It returns a plausible number with high confidence, perfectly formatted, and your code happily writes $1,234.00 to the ledger when the real value was $1,284.00. A wrong-but-confident value is far more dangerous than an obvious blank.
• The missing-field trap. Ask for a purchase_order field that isn't on this document and a naive extractor will often invent one rather than return null — because the prompt implied it should exist.

The whole rest of this lesson is the antidote. You can't make the model never wrong; you can make it tell you when it's unsure and prove where it got each value — and then gate on that. The interactive at the end lets you watch a raw VLM ship a confident-wrong total, and then watch grounding and verification catch it.

You make extraction trustworthy with two mechanisms, applied per field:

Grounding — prove where each value came from. Every extracted value should carry a pointer back to the source: a citation (which page / which passage) or a bounding box (which region of the image). This does two things: a reviewer can verify in one glance instead of re-reading the whole doc, and a downstream check can confirm the value actually appears where the model says it does. A value with no source is a value you can't trust.

Per-field confidence — know which fields to doubt. A single ‘the extraction looks good’ is useless; you need a confidence per field, so you can auto-approve the invoice number and escalate the smudged total — field-level override, not all-or-nothing rejection. Where does the confidence come from? A few practical sources:

• Self-consistency / consensus scoring — run the extraction a few times (or at different resolutions); a value that comes back identical every time is trustworthy, a value that flickers is a likely hallucination (auto-filter it).
• Verify against the source — a second, focused pass: ‘Re-read only the Total box. What number is there?’ If the verify pass disagrees with the first, confidence collapses.
• Cross-field validation — the math must close: do the line items sum to the subtotal? does subtotal + tax = total? A failed check is a strong low-confidence signal.

Grounding answers ‘is this real?’; confidence answers ‘how sure?’. Together they're what let you automate the easy 80% and route the hard 20% to a human — which is the agentic loop.

‘Agentic extraction’ sounds fancy; it's just not trusting the first read. Instead of a single model call, you run a small loop per document (or per field):

1. Plan — what fields am I extracting, and what's the schema?
2. Extract — first pass: pull each field with a value and an initial confidence.
3. Verify — for any field that's low-confidence (or business-critical), do a focused re-read of just that field's region and compare. Agreement raises confidence; disagreement collapses it. (For bad scans, a multi-pass read — different crops/resolutions — is the analog of a human squinting and tilting the page.)
4. Validate — run the cross-field checks (sums, formats, ranges).
5. Gate — values above your confidence threshold auto-approve; everything else escalates to a human with its grounding attached. The agent never guesses a critical field it couldn't verify — it flags it.

That last rule is the whole point: a correct-but-flagged total is a win; a confident-but-wrong total is a bug. The loop turns ‘the model is ~85% accurate’ into ‘the auto-approved fields are ~99% accurate, and the rest are queued for review’ — which is a system a business can actually run on.

Full automation is the wrong goal for high-stakes extraction; calibrated automation is the right one. Surveys of production teams find roughly 74% keep a human in the loop — not because the model is bad, but because the cost of a silent error (a wrong payment, a missed clause) dwarfs the cost of a 30-second human check. The design is a review queue fed by your confidence gate:

• Set the threshold by the cost of being wrong. A marketing-tag extraction can auto-approve at 0.8; a payment amount might demand 0.98 or a human, always. The threshold is a business dial, not a model setting — and the interactive lets you feel it.
• Review at the field level. Don't bounce a whole 12-field invoice because one field is shaky — auto-fill the 11 confident fields and put a human on the one that's flagged, with its source region highlighted. That's the difference between a 30-second correction and a re-keying job.
• Close the loop — every correction is a test case. When a human fixes a flagged field, that (document, field, correct value) becomes a new eval case (L246) and a candidate few-shot example. Your accuracy climbs and your review rate falls over time. This is the same flywheel as the rest of the course: production failures → evals → a better system.

The human isn't a failure of the AI; the human is how the AI earns trust — and the system is designed so they only ever look at the few fields that actually need them.

The agentic pipeline is slow and expensive (it's doing multiple reads and verifications per document). Running every document through it is a waste, because most of your documents are easy — a clean digital PDF needs a fraction of that effort. So you route by difficulty, which is the parsing analog of model routing (L216):

• The clean majority — born-digital PDFs with selectable text, simple layouts — go through a cheap, fast parser (an OSS library like Docling or Unstructured, or a single VLM pass). Often near-perfect for pennies.
• The hard, high-value minority — scanned forms, financial tables with merged cells, handwriting, multi-column filings, anything regulated — get routed to the agentic / premium pipeline (a purpose-built service like Reducto or LlamaParse, or your verify-loop). It earns its ~3× cost by being ~20% more accurate exactly where accuracy matters.

A quick map of the tool landscape (you'll mix and match): OCR / cloud document AI — AWS Textract, Google Document AI, Azure Document Intelligence. OSS parsers — Docling, Unstructured (cheap, great for the clean majority and RAG ingestion). Agentic / premium — Reducto, LlamaParse, LandingAI (complex tables, scans, regulated workloads; on-prem / SOC 2 / HIPAA options). Native VLM — Claude or Gemini reading the PDF directly. The skill isn't picking the one ‘best’ tool; it's segmenting your documents by difficulty and sending each class to the cheapest thing that hits your accuracy bar.

You can't manage what you don't measure, and extraction has a specific way to measure it. Don't score ‘did the JSON look right’; score per field, against a human-labeled gold set:

• Field-level precision & recall. For each field across your eval set: how often was the extracted value exactly correct (precision), and how often did it find the field that was actually present (recall)? A model can be 95% on invoice_no and 70% on total — and you'd never know from an aggregate score. Track the fields that matter most the hardest.
• Confidence calibration. Your auto-approve gate only works if confidence is honest: when the model says 0.95, is it right ~95% of the time? Plot accuracy vs. reported confidence; if the curve is off, your threshold is lying to you.
• The long tail is the whole game. The clean documents pass; your accuracy and your incidents live in the weird 10% — the rotated scan, the handwritten correction, the unusual vendor format. Build your eval set from real production failures (the review-queue corrections from the last section), not from pretty samples.

This is the L246 eval discipline aimed at documents: a versioned gold set, run in CI on every prompt / model / pipeline change, with field-level and calibration metrics — so you can change anything and know instantly whether total accuracy moved.

Feel the whole spectrum in one place. You're extracting fields from an invoice; toggle the approach (Raw VLM / OCR+LLM / Agentic), drag the auto-approve threshold, switch a clean vs scanned document, and click any field to ground it to its region on the page. Watch the pipeline, the per-field confidence, and the auto-approve-vs-review split change.

Three things to try:

1. Watch a confident-wrong ship. On the scanned document with Raw VLM, the smudged Total comes back as a confident wrong number ($1,234 vs the true $1,284) — high confidence, so it auto-approves and ships wrong. That single red field is the whole reason this lesson exists.
2. See the safe failure. Switch to OCR+LLM: same document, but now the Total returns low-confidence and gets flagged for review instead of guessed. It didn't get more accurate — it got more honest.
3. Run the loop. Switch to Agentic: it verifies each field against the source, auto-approves the ones it can confirm, and escalates the ambiguous Total to a human — zero confident-wrong, 100% accurate on what it auto-approved, at ~3× cost. Then remember: you'd route only the hard docs to this. Drag the threshold to see the business dial between automation and safety.

The takeaway in your hands: you don't make the model perfect — you make it tell you when it's unsure, prove where it got each value, and escalate the rest.

Reason it through (then check in the lab):

1. Your extractor returns total: 1234.00 with confidence: 0.93 on a blurry scan — but the real total is 1284.00. Which is the more dangerous output: this, or a confidence: 0.4 on the same field? Why?
2. You're processing 100,000 documents/month: 92,000 clean digital PDFs and 8,000 scanned forms. How do you keep accuracy high and cost low?
3. A model extracts a purchase_order number for a document that has none. What design choice prevents this, and what should the output have been?
4. Your auto-approve gate is set at 0.90 but you're still shipping wrong values that were marked 0.95. What's broken, and how do you find out?

Answers:

1. The 0.93 confident-wrong is far more dangerous — it auto-approves and ships a wrong number silently. The 0.4 is safe: it falls below your gate and gets reviewed. A calibrated low confidence is a feature; a confident hallucination is the bug you build the whole pipeline to prevent.
2. Route by difficulty. Send the 92k clean PDFs through a cheap parser / single VLM pass (near-perfect, pennies), and route only the 8k scanned forms through the agentic verify pipeline. You pay agentic prices for 8% of volume and capture almost all the accuracy benefit.
3. An explicit null / unreadable_fields escape in the schema, plus a prompt that says ‘set missing fields to null — do not guess.’ The output should have been purchase_order: null. Without that escape, the model invents one because the schema implied it exists.
4. Your confidence is miscalibrated — 0.95 doesn't actually mean 95% correct on your documents. Find out by plotting accuracy vs. reported confidence on your gold eval set; if 0.95 is really ~80% accurate, raise the threshold (or add a verify pass) until the auto-approved bucket hits your bar.

• “Just send the PDF and ask for JSON.” → That's a demo, not a system. Real documents are scanned, rotated, and smudged; naive extraction is ~62–88% per field and confidently wrong on the rest. You need schema + grounding + confidence + a review gate.
• “If it can't read a field, it'll leave it blank.” → No — it hallucinates a plausible value. Give it an explicit null/unreadable escape and tell it not to guess.
• “Higher confidence means it's right.” → Only if confidence is calibrated. An uncalibrated 0.95 that's really 80% accurate will ship wrong values through your gate. Measure calibration.
• “OCR is obsolete; VLMs do it all.” → OCR still gives exact characters and is cheap; VLMs add layout + robustness but can hallucinate. The best pipelines combine them, plus an agentic verify step.
• “Run every document through the best (agentic) pipeline.” → Wasteful. Route by difficulty — cheap parser for the clean majority, agentic only for the hard, high-value minority.
• “A human in the loop means the AI failed.” → A human in the loop is how the AI earns trust — and a good system only shows them the few flagged fields, with the source highlighted. Their corrections become your evals.
• “One accuracy number tells me how good it is.” → Score per field. Aggregate accuracy hides that total is 70% while invoice_no is 98% — and total is the one that matters.

• Document AI is a spectrum: OCR → layout models → VLM → agentic, climbing in robustness and cost. OCR-based gives exact text; OCR-free VLMs give layout-robustness but can hallucinate; combine them.

• Frontier models read a PDF as hybrid image + text per page (layout + citations, up to ~600 pages) — start with a VLM + a JSON schema and an explicit ‘unreadable → null, don't guess’ escape.

• Naive extraction is dangerous: ~62–88% per-field accuracy and confident hallucination of fields it can't read. The fix isn't a better prompt — it's a system.

• Make it trustworthy per field: ground every value to its source (citations / bounding box), score per-field confidence (consensus, verify-against-source, cross-field checks), and run a confidence gate.

• The agentic loop = extract → verify → validate → gate → escalate. Auto-approve the confident, send the ambiguous to a human — never guess a critical field. Auto-approved accuracy goes to ~99%.

• Keep a human in the loop at a field level, with a threshold set by the cost of being wrong; every correction becomes an eval case (the flywheel).

• Route by difficulty to control cost (the parsing analog of model routing), and eval per field with calibration on a gold set built from real failures (L246).

• Next — L252: Audio — Speech-to-Text & Text-to-Speech. You can read documents; next you give your AI ears and a voice — transcription, TTS, and the latency/quality tradeoffs of audio, the foundation for the voice agents in L253.