Multimodality: Text, Vision, Audio

So far, our models have lived on text alone. But modern frontier models can also see images, read documents, hear audio, and even watch video. A model that handles more than one kind of input is multimodal — and it unlocks an enormous class of applications: turning a photo of a receipt into structured data, answering questions about a chart, transcribing a meeting, reading a screenshot, describing an image for accessibility.

The beautiful part is that, under the hood, this isn't a different kind of AI — it's the same transformer you already understand, fed a clever trick: everything is turned into tokens. Once you see that, multimodality stops being magic and becomes just another input to shape.

You'll learn:

• What multimodal actually means (and what models output)
• How it works — images and audio become tokens in one shared stream
• Why images cost tokens (and money)
• The 2026 landscape: who's best at vision, audio, video, documents
• What you can build, how to send an image, and the limits to respect

A modality is a type of data — text, image, audio, video. A multimodal model accepts (and sometimes produces) more than one.

The most common and mature capability beyond text is vision: you send an image alongside your text, and the model 'understands' it — describes it, extracts text from it, answers questions about it. Some models also take audio (transcription, audio understanding) and video.

One clarification that saves confusion: most multimodal LLMs take images/audio as input and respond in text. Show it a photo and ask 'what's wrong with this circuit?' → it answers in words. Generating images or audio is usually a separate kind of model (image generators, speech synthesis); this lesson is about models that perceive multiple modalities and reason in text.

Here's the key idea, and it builds directly on Section 2 (tokens, embeddings, attention). Recall that the transformer doesn't really operate on words — it operates on tokens turned into embedding vectors. Multimodality works by turning images (and audio) into vectors in that same embedding space.

For an image, the recipe is:

1. Split the image into patches — a grid of small squares.
2. Encode each patch with a vision encoder into an embedding vector.
3. Project those vectors into the same space as text tokens — now they're effectively 'image tokens.'
4. Concatenate image tokens with the text tokens into one stream, and feed it to the same transformer.

From the model's point of view, an image is just more tokens sitting next to your words. The same attention mechanism that relates words to words now relates words to image patches — which is how it can answer 'what color is the car on the left?'

Early multimodal systems bolted a separate vision model onto a text model with a connector. The 2025–2026 frontier did something better: native, early-fused multimodality — a single transformer trained from scratch on a mixed stream of text, image, audio, and video tokens together.

Why it matters: a natively multimodal model doesn't translate an image into words and then reason about the words — it reasons over the image and text jointly, in one shared representation. That's why today's best models can do genuinely cross-modal tasks (reason about a diagram while reading its caption, follow a chart's trend and explain it) rather than just captioning. When you hear a model called 'natively multimodal,' this early-fusion design is what's meant.

Because images become tokens, vision is not free. A larger or higher-resolution image is split into more patches → more tokens — which means it consumes more of your context window and costs more (recall the pricing and context lessons).

Practical consequences:

• A single high-res image can cost hundreds to a couple thousand tokens — sometimes more than the text around it.
• Many APIs offer a detail/resolution setting (low vs high): low detail uses far fewer tokens for simple images; high detail is for fine print and small objects.
• Downscale images to the smallest size that still shows what matters; don't send a 4000×3000 photo to read one line of text.

Text + image is now table stakes — GPT-5.5, Claude Opus, Gemini 3.x, and Qwen Omni all handle it strongly. But the modalities diverge once you go past images:

Key caveat: Claude is text+image only; OpenAI's GPT line handles audio natively (the omni models / Realtime API) but not video; Gemini is the one with native text+image+audio+video in a single API. For video — or audio on Claude — use a model that supports it, or a separate speech model (ASR) to convert audio → text first. Match the model to the modality your product needs.

Multimodality opens doors that text-only models can't:

• Document AI / OCR — invoices, receipts, forms, contracts, scanned PDFs → structured data. One of the highest-value real-world uses.
• Visual Q&A — "what's wrong in this photo / diagram / X-ray?" "summarize this slide."
• Charts & diagrams — read a graph, extract the trend, explain an architecture diagram.
• Screenshots & UI — understand an app screen (the backbone of computer-use agents).
• Accessibility — describe images for visually impaired users (alt text at scale).
• Audio — transcription (ASR) and voice interfaces (often via a dedicated speech model feeding an LLM).

Sending an image is just another content block in your message — text and image side by side:

resp = client.messages.create(
    model="claude-opus-4-8", max_tokens=512,
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "What's in this image?"},
        {"type": "image", "source": {"type": "base64",
            "media_type": "image/jpeg", "data": img_b64}},
    ]}],
)

(OpenAI uses the same idea with an image_url content part.) The mental model from the API lesson holds — multimodal input is richer content blocks, same call.

Vision is powerful but not flawless — and it hallucinates just like text:

• Fine detail & small text can be misread — verify OCR on critical fields (amounts, IDs).
• Counting and precise spatial reasoning ('how many people?', 'exact pixel position') are weak spots.
• It will confidently describe things that aren't there, especially for blurry, unusual, or adversarial images (the plausibility-not-truth problem, now in pixels).
• High-res = high cost — balance detail against token budget.
• Privacy & sensitivity — images can contain faces, PII, or medical data; treat them with the same care as any sensitive input (and consider local models for privacy).

For high-stakes extraction, validate the model's output (regex/schema checks, confidence prompts, or a second pass) rather than trusting it blindly.

• It hugely expands what you can build — document automation, visual assistants, and voice interfaces are all in reach with the same skills you've learned.
• Choose by modality. Need audio/video? Lean Gemini. Heavy document OCR? Claude. Charts/code-with-vision? GPT-5.5. Just image understanding? Anything frontier works.
• Budget for image tokens — they hit your context window and bill; downscale and use detail settings.
• Mind the same failure modes. Vision hallucinates, miscounts, and fumbles fine print — validate high-stakes outputs.
• The mental model carries over. Multimodal input is just richer content in the same stateless call — no new architecture to learn, just new content blocks.

Try document AI — and find its limit. Send a photo of a receipt and ask for {"merchant":..., "total":..., "date":...} as JSON. It'll mostly nail it — then deliberately send a blurry or low-res one. Predict the failure → a mis-read amount or date, confidently wrong. The takeaway: vision is powerful but hallucinates on fine print, so validate the high-stakes fields (amounts, IDs) every time.

• Multimodal models accept more than text — vision (image) is universal; audio/video vary by model.
• How it works: images are split into patches → encoded → projected into the same token/embedding space as text, so one transformer attends across modalities (native/early fusion).
• Images cost tokens (and money): higher resolution = more patches = more tokens — downscale and use detail settings.
• 2026 leaders: Gemini (audio/video native), Claude (long-doc OCR), GPT-5.5 (charts/code-with-vision); Claude is image-only; GPT adds native audio (not video); Gemini does audio+video.
• Build: document AI, visual Q&A, charts, screenshots/UI, accessibility — but vision hallucinates, so validate critical output.

Next: capabilities always come with trade-offs. We tackle the central tension of model choice head-on — the Iron Triangle of latency, cost, and quality.