Image & Video Generation (Diffusion Basics)

Every lesson in this section so far has taught a model to read the world: L250 had it see (image understanding & VQA), L251 had it parse (document AI), L252 had it hear (speech-to-text), and L253 had it converse (voice agents). This lesson flips the arrow. Now the model creates — it turns a sentence of text into a photograph that never existed, or a video of a scene that was never filmed.

The engine behind almost all of it is diffusion, and its core idea is one of the most beautiful in modern ML: learn to sculpt an image out of pure noise. Start with a square of random static, and step by step, remove the noise that doesn't belong — guided by your text prompt — until a coherent image emerges. Run the same process across dozens of frames at once and you get video.

As an AI engineer you will rarely train one of these models. You will call them — and to call them well you need a real mental model of what's happening, what each knob (steps, guidance, seed) actually does, which model to pick for which job, how editing and control work, and how to ship responsibly (provenance, cost, safety). That's this lesson.

In this lesson:

• The big idea — forward noising (training) and reverse denoising (generation), and why it works.
• Latent diffusion & the denoiser — the VAE latent trick, and the U-Net → DiT → MMDiT evolution.
• Steering — how the prompt conditions the image, and classifier-free guidance (CFG).
• The knobs you turn — steps, samplers, seed, and distillation for speed.
• Diffusion vs. autoregressive — why GPT Image is not a diffusion model, and the 2026 landscape.
• Control & editing — img2img, inpainting, ControlNet, instruction editing.
• Video — diffusion across time, and the 2026 video models.
• Safety, provenance & cost — C2PA, SynthID, and shipping economically.

Scope: this builds on L250 (Vision) — generation is the inverse of understanding — and on the embedding/CLIP ideas from the retrieval lessons. It defers stitching generation and retrieval into a larger multimodal system to L255 (Multimodal RAG & Pipelines), the section finale. We cover the basics deeply; we are not training a model from scratch.

Diffusion is built from two processes that mirror each other.

Forward process (used only in training): destroy an image. Take a real photo and add a tiny bit of Gaussian noise. Add a bit more. Repeat hundreds of times and the photo dissolves into pure random static — every trace of the original gone. This is mechanical; there's nothing to learn here. Its only purpose is to manufacture training pairs: (slightly-noised image, the noise that was added).

Reverse process (used at generation time): create an image. Train a neural network to look at a noisy image and predict what noise to remove to make it slightly cleaner. Do that, and you can run it in reverse: start from a fresh square of pure random noise and ask the network, step by step, "what should I subtract to make this look more like a real image — specifically, one matching this prompt?" After enough steps, a coherent image emerges from the static. That's generation.

It helps to picture a sculptor told "there's a statue inside this block of marble — remove everything that isn't the statue." The noise is the marble; the prompt says which statue; each denoising step chips away a little more.

What does the network actually predict? Different formulations target different things: the original DDPM models predicted the noise (epsilon) added at each step; modern flow-matching / rectified-flow models (Stable Diffusion 3, Flux) predict a velocity — the direction to move through the space — which gives straighter paths from noise to image and therefore fewer steps. Some predict the clean image directly. The intuition is the same in all of them: learn to undo noise, then run it backwards from nothing.

This is why, in the lab at the end, dragging the steps slider literally clears static off the image: you are watching the reverse process resolve a picture out of noise.

Running the denoising loop directly on a megapixel image would be brutally expensive — millions of pixels, dozens of steps. The breakthrough that made diffusion practical (and that powers Stable Diffusion, Flux, and the rest) is latent diffusion: don't diffuse pixels — diffuse a compressed representation.

A pretrained VAE (variational autoencoder) encodes the image into a small latent — e.g. a 1024×1024×3 image becomes a ~128×128×4 latent, roughly 48× fewer numbers. The entire noisy↔clean diffusion process happens in that compact latent space, and only at the very end does the VAE decode the finished latent back into full-resolution pixels. Same quality, a fraction of the compute.

The network doing the denoising — the denoiser — has evolved through three generations, and knowing the names helps you read model cards:

• U-Net (2021–2023): a convolutional encoder-decoder with skip connections. Powered Stable Diffusion 1.5, SD2, and SDXL (~2.6B params). Convolutions are efficient but hit a scaling ceiling.
• DiT — Diffusion Transformer (2023+): chop the latent into patches, treat them as a sequence of tokens, and run transformer blocks (the same architecture as LLMs). It scales far better with data and compute. PixArt-α showed a 0.6B DiT could rival much larger U-Nets.
• MMDiT — Multimodal DiT (2024+): the current frontier (Stable Diffusion 3, Flux, Qwen-Image). It runs separate but interacting streams for the text tokens and the image tokens, letting them attend to each other directly. Flux.1 is an ~12B MMDiT.

The trajectory is the same one LLMs took: hand-crafted convolutional designs gave way to general transformers that scale with compute. When a 2026 model brags about a "DiT" or "MMDiT" backbone and "rectified-flow" training, you now know exactly what it means: a transformer denoiser, in a VAE latent, taking near-straight paths from noise to image.

A denoiser left alone produces some plausible image. To make it produce your image, you inject the prompt — this is conditioning.

Text conditioning. A frozen text encoder turns your prompt into embeddings the denoiser can use. Early Stable Diffusion used CLIP (which aligns text and images — the L-series embedding idea). Later models added or switched to T5, a large language-model encoder, for much better understanding of long, compositional prompts; SD3 uses both CLIP and T5. Those text embeddings steer the denoiser via cross-attention (U-Net/DiT) or joint attention (MMDiT, where text and image tokens attend to each other). Practically: a better text encoder is why newer models follow complex prompts ("a corgi to the left of a red cube") that older ones botched.

Classifier-free guidance (CFG) — the single most important knob. Here's the trick: during training the model sometimes sees the prompt and sometimes sees a blank (the prompt is dropped ~10% of the time), so it learns to predict both a conditional result (with your prompt) and an unconditional one (generic image). At generation time you extrapolate between them:

    guided = unconditional + scale × (conditional − unconditional)

The CFG scale controls how hard you push toward the prompt:

• Too low (≈1–3): the image barely follows the prompt — vague, washed-out, low-contrast.
• The sweet spot (≈5–9): strong prompt adherence with natural color and detail. ~7 is a common default.
• Too high (≈15+): over-pushed — over-saturated, high-contrast, "fried," with artefacts and less diversity.

(Distilled fast models — next section — are tuned for a much lower CFG, often 1–2, or bake guidance in.) A negative prompt is the same machinery aimed in reverse: push away from terms like "blurry, extra fingers, watermark." You'll feel all of this in the lab: slide CFG and watch the image go from ghostly to clean to garish.

When you call an image API, a handful of parameters decide the speed, cost, and look of the result. These are the levers worth understanding:

• Steps (a.k.a. num_inference_steps): how many denoising iterations. More steps = more detail, more time. A full model like Flux dev needs ~28; below ~15 it looks soft and unfinished. This is the dominant cost lever (cost ≈ steps × per-step compute).
• Sampler / scheduler: the algorithm that takes each step (Euler, DPM++, DDIM, …). Different samplers reach good quality in different step counts; it's a quality-vs-speed trade, not a correctness one.
• Guidance (CFG): prompt adherence, from the previous section.
• Seed: the initial random noise. Same seed + same prompt + same model = the exact same image, every time. This is your reproducibility lever — lock a seed to iterate on a prompt, or vary it to sample alternatives.
• Size / aspect ratio and strength (for image-to-image, next section).

Distillation — the speed revolution. A full 28-step model is too slow and pricey for many products. Distillation trains a student model to match the teacher's output in far fewer steps: LCM (Latent Consistency Models), SDXL-Turbo / Lightning, DMD2, and Flux schnell reach good images in 1–4 steps — roughly a 10× speedup — for a small quality cost (and they use little or no CFG). This is how models like Z-Image Turbo generate in ~1 second for ~$0.01. The lab makes the trade-off concrete: on schnell, steps past ~4 are simply wasted.

Here's a real call to a hosted diffusion model (Flux on fal), showing the knobs side by side:

An honest twist: not every "AI image generator" is a diffusion model. There are two paradigms, and the difference is now product-relevant.

• Diffusion (Flux, Imagen, Stable Diffusion, Midjourney): works in continuous latents and refines the whole image in parallel over many steps. Fast (parallel), great quality, many knobs (steps/CFG/seed).
• Autoregressive (AR) (OpenAI's GPT Image): generates the image as discrete tokens, one at a time, left-to-right — the same mechanism an LLM uses to write text. GPT Image is built into GPT-class models, so the network that understands your request also draws it. The payoff is outstanding prompt-following and text rendering and conversational editing; the cost is that it's sequential (slower, one image per call) and has no CFG knob. Some 2026 models (e.g. GLM-Image) go hybrid — AR for layout, a diffusion decoder for pixels.

The lab lets you see this: in GPT Image mode the picture fills left-to-right, not by clearing noise everywhere at once.

The 2026 image landscape (verify specifics — this moves monthly):

Where does Claude fit? Claude is a text + vision + document model — it reads images superbly (L250) but does not generate them. In a real product you pair Claude with an image model: Claude is excellent at writing and refining the image prompt, deciding when to call the generator (as a tool), critiquing the result it can now see, and orchestrating multi-step edits. The generation is a specialist API; Claude is the brain that drives it — exactly the cascaded pattern from the voice lesson (L253), applied to pixels.

Pure text-to-image is a slot machine: you describe, you pray. Production work needs control — starting from an existing image, constraining structure, or editing a specific region. The toolkit:

• Image-to-image (img2img): seed the process with an existing image instead of pure noise. A strength parameter (0–1) sets how much to keep: low strength = a light restyle, high strength = a near-total reimagining. The everyday "make this photo look like an oil painting."
• Inpainting / outpainting: mask a region and regenerate only that area (remove an object, swap a shirt) — or extend the canvas beyond its borders (outpainting) to widen a shot.
• ControlNet: condition generation on a structural map extracted from a reference — a human pose skeleton, a depth map, Canny edges, a scribble. You keep the composition and change the style/content. This is how you get a character in the exact pose you need.
• IP-Adapter / reference: use an image as a visual prompt — "in the style of this" or "this character" — via a separate cross-attention path.
• Instruction editing (the 2026 UX): the conversational "edit this image" — drop in a photo and say "make it night, add a red car" and get a coherent edit back. Nano Banana and GPT Image made this the dominant interface, and it composes naturally with a chat model orchestrating the edits.

For an engineer, editing is often the higher-value capability than generation-from-scratch: most real workflows start from a product photo, a brand asset, or a frame — not a blank prompt. Reach for img2img/inpaint/ControlNet before you reach for raw text-to-image.

Video generation is diffusion with a time axis. The dominant approach is a Diffusion Transformer over spatio-temporal patches: instead of denoising one image, the model denoises many frames at once, treating little cubes of (space × time) as tokens. Start from noise across the whole clip, refine step by step, decode to frames.

The hard part isn't any single frame — it's temporal consistency. A face must stay the same face, a car the same car, the lighting and camera motion coherent across seconds. Solving that (so objects don't morph or flicker) is what separates the leaders from the toys, and why video models are far heavier than image models.

The 2026 video landscape (fast-moving — verify):

• Google Veo 3.1 — most versatile, strong physics, ~$0.15/sec.
• OpenAI Sora 2 — high cinematic quality, ~$0.75/sec (premium).
• Kuaishou Kling 3.0 — multi-shot cinematic sequences with subject consistency, ~$0.10/sec.
• Runway Gen-4.5, ByteDance Seedance, open-weight Wan 2.6 (~$0.05/sec), and fast Pika (clips in 15–30s).

Two 2026 shifts matter for engineers: native synchronized audio (most top models now generate sound with the video — a research demo a year ago, a product feature today), and cost discipline — at $0.05–$0.75 per second, a 30-second clip is $1.50–$22.50, so you generate short, iterate on cheap/fast models, and reserve the premium tier for finals. Treat video like an expensive batch job, not a chat call: it takes seconds-to-minutes and real money per generation.

Shipping generative media responsibly is now a hard requirement, not a nicety — and increasingly the law.

Provenance — proving where an image came from. Two complementary standards have converged:

• C2PA Content Credentials (now an ISO standard, ISO/IEC 22144) — a cryptographically signed manifest attached to the file recording the model/device that made it and every edit since. Backed by 6,000+ organisations: Google, OpenAI, Adobe, Meta, Microsoft, and camera makers (Sony, Nikon, Leica). It's becoming the provenance language of the internet.
• SynthID (Google) — an invisible watermark baked into the pixels (and audio) at generation, designed to survive screenshots, crops, and compression. OpenAI adopted a layered C2PA + SynthID approach in 2026.

The honest limitation: provenance is verification, not detection. A missing credential proves nothing — and the tools used to make malicious deepfakes are exactly the ones that won't watermark. Treat Content Credentials as a trust signal for good-faith content, not a deepfake detector.

Regulation. The EU AI Act (Article 50) requires AI-generated content to be machine-readable marked and deepfakes disclosed, with obligations landing August 2026. If you ship generated media to EU users, labelling is mandatory. Plus the usual: safety filters on prompts and outputs (no CSAM, non-consensual imagery, etc.), and rights questions around training data and likenesses.

Cost & latency engineering ties it together: pick the cheapest model that clears your quality bar (often a distilled one for drafts, a frontier one for finals), cache aggressively, generate at the resolution you need (not the max), and remember images are seconds + cents while video is minutes + dollars — budget accordingly.

Time to turn the knobs yourself. The lab below starts a mountain-lake prompt on Flux dev at just 6 steps — so the image is still buried in noise ("under-cooked"). Drag STEPS up and watch the reverse process clear the static into a picture; the strip on the right shows the whole noise → image journey.

Then play with the other knobs: push GUIDANCE (CFG) below 3 (washed out) and above 15 (fried, over-saturated) to find the 5–9 sweet spot; change the seed to sample a different image of the same prompt (and confirm the same seed always reproduces). Finally, switch the model: Flux schnell resolves in ~4 steps (try going higher — it's wasted), and GPT Image fills the canvas left-to-right instead of denoising — the autoregressive paradigm, made visible.

Notice three things. One: steps clear noise — too few and the latent never resolves; a distilled model just needs far fewer. Two: guidance is a sweet spot, not "more is better" — past ~14 it fries. Three: diffusion refines the whole image in parallel, while the autoregressive model paints it sequentially — two genuinely different ways to make a picture.

Predict first, then check in the lab (or reason it out).

1. You run Flux dev at 4 steps and the output is a blurry, half-formed mess. Your colleague runs Flux schnell at 4 steps and gets a crisp image. Same step count — why the difference?

2. Your generations look over-saturated and harsh, with weird halos. Which single knob is most likely wrong, and which way do you move it?

3. A client says "I love image #3 — can you make ten more exactly like it but with a blue car instead of red?" Which parameters do you hold fixed, which do you change, and which technique (text-to-image vs img2img/inpaint) do you reach for?

4. You need to generate 5,000 product thumbnails tonight on a budget, then 3 hero images where quality is everything. Which model(s) do you pick for each, and why?

5. You're shipping AI images to EU users. What must you add to every output, and what's the difference between what C2PA gives you and what a "deepfake detector" claims to give you?

Answers.

1. Distillation. Flux dev is a full model that needs ~24–28 denoising steps to resolve — at 4 it's nowhere near done. Flux schnell is step-distilled to converge in ~4 steps, so it's essentially finished. (Bonus: running schnell at 28 steps wouldn't improve it — those steps are wasted.)

2. Guidance / CFG scale — turn it down. Over-saturated, harsh, haloed output is the classic "fried" signature of CFG set too high (≈15+). Bring it back to the 5–9 sweet spot (or lower for a distilled model).

3. Hold the seed (and prompt, model, steps, CFG) fixed; change only the relevant detail. Same seed + prompt + model reproduces the image; for a targeted change like the car color, reach for inpainting (mask the car, regenerate just that region) or an instruction edit ("make the car blue") rather than a fresh text-to-image, which would change everything. Pure text-to-image with a tweaked prompt would give you a different scene, not the same one edited.

4. Thumbnails: a distilled/cheap model (Flux schnell, Z-Image Turbo, Seedream) at low steps — ~$0.01 and ~1s each keeps 5,000 images cheap and fast. Hero images: a frontier model (Imagen 4 Ultra for photoreal, GPT Image 2 for precise prompt-following/text, Flux 2 pro) at full steps — quality over cost for just three. Match the model tier to the stakes.

5. Add machine-readable provenance — C2PA Content Credentials (a signed manifest) and/or a SynthID watermark — and disclose AI generation per EU AI Act Article 50. The difference: C2PA proves a good-faith origin and edit history (verification); it is not a detector — a missing credential doesn't mean an image is real or human-made, and malicious deepfake tools simply won't embed one.

• "The model paints an image like a person would." → It removes noise. It starts from random static and, step by step, subtracts the noise that doesn't fit the prompt until an image remains.
• "Diffusion works on the pixels." → It works in a compressed VAE latent; pixels only appear at the final decode. That latent trick is what makes it affordable.
• "More steps and higher guidance = better." → Steps have diminishing returns (and distilled models need ~4); guidance is a sweet spot (5–9) — too high fries the image. More is not better.
• "All AI image generators are diffusion." → No — GPT Image is autoregressive (discrete tokens, left-to-right, like an LLM). Diffusion refines the whole image in parallel; AR builds it sequentially.
• "The seed is just randomness I can ignore." → The seed is your reproducibility lever: same seed + prompt + model = the same image. Lock it to iterate, vary it to explore.
• "Real work is text-to-image." → Most production work is editing — img2img, inpainting, ControlNet, instruction edits — starting from an existing asset, not a blank prompt.
• "Claude can generate the image." → Claude reads images, it doesn't make them. Pair it with an image model: Claude writes the prompt and orchestrates; the generator draws.
• "A C2PA credential / watermark detects deepfakes." → It's provenance, not detection. It proves good-faith origin; a missing credential proves nothing, and bad actors won't add one.
• "Video is just image generation, repeated." → The hard part is temporal consistency across frames — and video is minutes + dollars, not seconds + cents.

• Diffusion sculpts an image from noise: train a network to undo noising (forward process), then run it backwards from pure static, steered by the prompt, for many steps — a picture resolves out of the noise.
• Latent diffusion runs the loop in a compressed VAE latent (not pixels); the denoiser evolved U-Net → DiT → MMDiT (transformers won here too), with flow matching / rectified flow giving straighter, fewer-step paths.
• The knobs: steps (more detail, more cost; distilled models like Flux schnell / LCM / Turbo converge in ~4), guidance / CFG (prompt adherence; 5–9 sweet spot — too low = washed out, too high = fried), and seed (reproducibility).
• Not all generators are diffusion: GPT Image is autoregressive (discrete tokens, left-to-right, in a GPT model) — superb prompt-following, sequential and slower. Pick per job from the 2026 field (Flux 2, Imagen 4, GPT Image 2, Nano Banana, cheap distilled models).
• Control beats luck: img2img (strength), inpaint/outpaint, ControlNet (pose/depth/edges), and instruction editing are how real products work — start from an asset, not a blank prompt.
• Video is diffusion across time (spatio-temporal DiTs; the hard part is temporal consistency); the 2026 leaders are Veo 3.1, Sora 2, Kling 3.0, now with native audio — and it's minutes + dollars per clip.
• Ship responsibly: C2PA Content Credentials + SynthID for provenance (verification, not deepfake detection), EU AI Act labelling (Aug 2026), safety filters, and cost discipline (cheapest model that clears the bar; right resolution; cache).
• Claude's role: it reads images (L250), it doesn't generate them — pair it with an image model and let Claude write the prompt and orchestrate edits.
• Next — L255 (Multimodal RAG & Pipelines): the section finale — stitch understanding, retrieval, and generation across text, images, audio, and video into one production pipeline.