Versioning Prompts, Models & Datasets

You shipped an AI feature, gave it the right API shape (L244 — Designing Your AI API), metered it (L245 — Rate Limiting & Quotas), and gated every change on evals (L246 — CI/CD for Prompts & Evals). Then a user reports a wrong refund answer from last Tuesday. Simple question, brutal follow-up: can you reproduce it?

You can only if you know the exact prompt, the exact model version, the exact params, and the exact document corpus that produced that answer. If any of those has drifted since Tuesday — the prompt was edited live, the model was called as latest and the provider updated it, the corpus was re-indexed — then the run is gone. You can't recreate the bug, so you can't confidently fix it, and you have nothing to roll back to. You're debugging a ghost.

This lesson is the cure: treat every behavior-determining input as a versioned artifact, pin them all into one immutable release manifest, and record that manifest on every output so any answer is traceable to exactly what produced it. That's what buys you reproducibility, rollback, and audit.

Scope: this is the artifact / reproducibility / lineage side of LLMOps — what you version and how you bind it. It's the twin of L246 (CI/CD for Prompts & Evals): L246 tests and gates a change; this lesson makes that change a pinned, rollbackable artifact with lineage. We defer the audit-log and incident mechanics to L237 (Model Cards, Audit Logs & Incident Response), live debugging to L223 (The Monitoring & Observability Layer), and the regulatory duty to L236 (AI Governance & Compliance — EU AI Act, NIST AI RMF) — versioning is what makes all three possible.

Here's the mental model that makes everything else click. In normal software, an output is a function of your code. In an AI system, an output is a function of several artifacts at once — and any of them changing changes behavior:

• the prompt (system + task templates, few-shot examples),
• the model and its exact version (a base model, a fine-tune, or a quantized variant are all different),
• the sampling params (temperature, top-p, max-tokens, seed),
• the RAG corpus and index (the documents, the chunking, the embedding model, the retrieval config),
• the tool definitions for agents, and
• the eval / golden set you measure against.

The trap is to version only your code and the model's name. You have to version the full inference identity — which prompt, which model snapshot, which params, which corpus. There are four reasons this matters, and they're the spine of the whole lesson:

1. Reproduce — recreate a past output (to debug it, or to prove a result).
2. Debug / root-cause — trace a bad answer back to the artifact that caused it.
3. Roll back — return to a known-good state instantly when something regresses.
4. Audit — answer “what exactly produced this decision?” for governance.

All four require the same thing: a record of the exact version of every artifact that produced a given output. As one practitioner guide puts it, “a single output should be traceable to: code version (git commit), model (provider + exact version), prompt (template + few-shot), data (snapshot id), and evaluation setup.” When even one artifact floats untracked, you get the AI version of “works on my machine” — a ghost bug nobody can reproduce.

A prompt is the most-edited, least-versioned artifact in most AI apps — someone tweaks a sentence in a dashboard and ships it with no record. Two patterns fix that, and mature teams use both:

1. Prompts as code. Store prompts as files in your repo and change them through pull requests — diffable, reviewable, versioned with the code that uses them, and (per L246) covered by the eval suite. This is the default for prompts that are tightly coupled to code.

2. A prompt registry. Tools like Langfuse, LangSmith (Prompt Hub), and PromptLayer store each prompt as a versioned object with labels. In Langfuse, every save auto-increments a version, a latest label tracks the newest, and a production label marks the version that's actually served; your code fetches by label, not by hard-coded text:

• langfuse.get_prompt("support-agent", label="production") returns whatever version production points at.
• Deploy = move the production label to a new version. Roll back = move it back to the previous one — under a minute, no code deploy. (LangSmith added prompt tags in late 2024 for the same agent:prod pattern.)

The big win of a registry is decoupling prompt changes from code releases and giving you instant, label-based rollback — the prompt version is just a pointer you can repoint. Use a human-readable semantic version (v2.3) so people can talk about it, and link the fetched prompt version to your traces so every output records which prompt produced it (that's the lineage thread we pull on at the end). Whichever pattern you choose, the rule is the same: the exact prompt that shipped is always recoverable.

Most providers let you call a model two ways: a floating alias (gpt-5.5) that always points at the current version, or an immutable dated snapshot (gpt-5.5-2026-04-30). For anything in production, always pin the snapshot.

Why? A floating alias can change weights, behavior, and quality with zero code change on your side — the provider updates the model and your output silently shifts. This is the exact failure we saw in L246 (CI/CD for Prompts & Evals): a model-version bump that silently regressed faithfulness while everything else looked fine. If you're on latest, that regression arrives unannounced and unreproducible — yesterday's bug may already be gone today. Pinning the dated snapshot makes every model change an explicit, deliberate choice that you push through the eval gate, instead of accidental drift. (Pinning also protects you from deprecation surprises — when a snapshot is retired, you schedule the migration and re-eval.)

A few more model-versioning truths:

• A fine-tuned or quantized model is a different artifact — version its weights/checkpoint, not just a name. “Capture the full inference identity, not just the model's name.”
• Self-hosted / open-weight models get versioned in a model registry: MLflow's Model Registry (stages like Staging/Production/Archived, and aliases such as @champion you resolve with models:/support-agent@champion; MLflow 3.0's LoggedModel even links a model to the code, prompts, and evals that produced it), Weights & Biases Artifacts with lineage graphs, or the Hugging Face Hub (pin a specific commit/revision, not main).

The one-liner: latest is a silent bug waiting to happen. Pin the exact snapshot, treat every bump as a reviewed change, and you decide when — and whether — quality moves.

Three kinds of data shape your system, and all three need versions: the training/fine-tuning data, the eval / golden set, and the RAG corpus. But you can't just git commit them — Git LFS caps files at ~5 GB and repos at ~10 GB, and datasets blow past that. Data is versioned by content-addressing instead: hash the contents, store the bytes once, and reference them by hash.

The tools built for this:

• DVC — a .dvc pointer file (a content hash) lives in git while the actual data sits in object storage; dvc add/dvc.lock pin the exact snapshot.
• lakeFS — git-like branches/commits over your data lake with zero-copy reads.
• Hugging Face Hub — dataset revisions/commits; its new Xet backend uses content-defined chunking + dedup so changing a few rows only re-uploads those chunks.

Two principles matter more than the tool:

Immutability. A version, once created, never changes. Even fixing a typo in the eval set should create a new version, not edit the old one — otherwise a score you recorded last month no longer means what you think it means.

The eval set must be versioned and immutable, or your scores lie. Comparing this week's 0.91 to last month's 0.86 is only meaningful if the test set is identical. If you quietly edit cases as you go, your “version history lies” — you can't tell improvement from a changed ruler. So pin the eval-set version on every eval run (LangSmith and Langfuse both version datasets for exactly this), grow the golden set by adding new immutable versions, and record which eval-set version produced each score. (This is the data backbone under L246's regression gate.)

RAG breaks naive versioning because a retrieval result is a composite of many coupled artifacts — and if you version some but not all, “the same document produces different answers and you won't know why.” To reproduce what retrieval returned, you must pin all of:

• the document corpus (which docs existed — a content hash of the set),
• the chunking strategy (size/overlap/splitter — change it and different context gets retrieved),
• the embedding model and its version,
• the index config (HNSW/IVF params, distance metric) and retrieval params (top-k, reranker, filters),
• the prompt template that assembles the context.

The killer detail is the embedding model. Upgrade it — say text-embedding-3-large@2025-09 → a newer version — and “the same text maps to different vectors.” Your old index and your new query vectors now live in incompatible spaces, so similarity scores become meaningless. There's no incremental fix: you must re-embed the entire corpus (a full re-index) and block cross-version queries. Treat embeddings like a schema — store the embedding_model_id with every vector and never mix versions.

You don't have to cold-store the raw vectors. “Storing the document hash set, the embedding model version, and the index config is enough to rebuild the index on demand.” That tuple is your RAG version. To deploy a new index safely, build it in parallel and alias-swap (blue-green) — instant roll-forward, instant rollback — which is exactly the deployment story we generalize next.

Individually versioned artifacts aren't enough; what makes a run reproducible is binding them together. A release manifest (a release.yaml / release.lock) is a single file that pins the exact version of every artifact — prompt, model snapshot, params, RAG index, eval set, and the code commit — into one immutable, content-addressed bundle. That bundle is your release: you promote it dev → staging → prod, and you can rebuild it byte-for-byte (artifact-wise) months later.

This is also where two ID schemes earn their keep:

• Semantic versions (v2.3, r2026.06.27-1) are for humans — they communicate intent and ordering.
• Content hashes (sha256:9f3c0aa…) are for machines — they guarantee uniqueness and immutability (the same hash always means the same bytes).

Most teams combine them (a semver tag plus a git commit / content hash). The manifest is the artifact the rest of the lesson hangs on: reproduce = re-instantiate this manifest; roll back = redeploy the previous manifest; audit = read the manifest attached to an output.

Here's the part too many guides skip: even with every artifact pinned, hosted LLMs are not bit-for-bit deterministic. Pinning gets you most of the way, but “temperature 0 does not guarantee deterministic outputs.” The same input can produce slightly different tokens because of:

• batched floating-point arithmetic (your request is batched with others; FP addition isn't associative, so results shift run to run),
• routing and infrastructure (mixture-of-experts routing, GPU model, batch size — changing the hardware or batch can swing accuracy by as much as ~9% in studies), and
• occasional ties in greedy decoding resolved nondeterministically.

What you can lean on: OpenAI exposes a seed parameter and a system_fingerprint (so you can detect when the backend changed); Anthropic does not expose a stable seed as of early 2026. Strict, repeatable determinism is only really achievable on open-weight models you run yourself (fixed seed, temperature 0, pinned inference backend and hardware).

So the realistic goal isn't identical — it's “pinned & traceable.” Pin everything you can, then record what actually ran — model id, version, params, and system_fingerprint — in the trace of every call. Don't trust temperature 0; log the run so you always know exactly what produced an output, even when you can't reproduce it to the token.

Tie it together. The moment every output carries its release manifest, you unlock the three payoffs that justified all this work:

• Lineage (debugging). Each production output records the exact artifacts that produced it, so a bad answer is traceable straight back to its prompt version, model snapshot, and corpus — root-cause instead of guesswork. (This is the record that L223 — The Monitoring & Observability Layer surfaces, and the evidence L237 — Model Cards, Audit Logs & Incident Response preserves.)
• Rollback. Because a prompt/model/release is a deployable, versioned pointer, recovering is repointing the production label (or redeploying the previous release.yaml) — seconds, not a hotfix.
• Audit / governance. Lineage is what lets you answer “what produced this decision, and on what data?” — exactly the traceability high-risk systems owe under the EU AI Act (L236), which requires being able to trace a model and its outputs back to their data and configuration.

In code, the whole discipline reduces to: resolve the pinned manifest, refuse a floating model, and stamp the manifest onto every output.

Assemble a release yourself. Pin or float each artifact and watch what the release lets you do — and then try to reproduce last Tuesday's bug:

The lesson is all-or-nothing across the behavior artifacts: float a single model (latest) or prompt (live-edited) and reproduce, roll back, and audit all flip red — the corpus or weights drifted and you're chasing a ghost. Pin every one and the release.lock materializes: the immutable bundle you stamp on each output as its lineage. (Note the eval set is special — it doesn't change a production output, but float it and your scores stop being comparable.)

Use the Version-Lab and what you've learned:

1. Hit Float all, then read the incident card. Why can't you reproduce last Tuesday's answer?
2. From there, flip Model to PINNED but leave the prompt floating. Does reproduce go green? Why not?
3. Float only the RAG corpus + index. Which of reproduce / roll back / audit break — and why does the embedding model deserve special fear?
4. Float only the eval set. Reproduce stays green but Compare evals over time goes red. Why is a production output fine while your scores aren't?
5. Pin all. What materializes, and what should you do with it on every output?

→ (1) The behavior artifacts (prompt, model, params, corpus) drifted, so the exact run is gone — no repro, no rollback target, a ghost bug. (2) No — reproducibility is all-or-nothing: a live-edited prompt means there's no record of the exact text that shipped, so you still can't recreate the output even with the model pinned. (3) Reproduce and audit break (and the run isn't fully traceable): the corpus changed and may have been re-embedded — and a new embedding model maps the same text to different vectors, making old and new vectors incomparable, so you'd have to re-index, not patch. (4) The eval set doesn't determine a production answer (so reproduce holds), but if the golden set isn't immutable, this month's score and last month's aren't measured by the same ruler — not comparable. (5) The release.lock manifest — one immutable bundle pinning every version; record it on every output so each answer is traceable to exactly what produced it (lineage).

• “I version my code, so I'm covered.” Code is one artifact. An AI output is a combination — prompt + model version + params + corpus + eval set. Version the full inference identity.
• “gpt-5.5 is the model version.” That's a floating alias — it can change under you with zero code change. Pin the dated snapshot (gpt-5.5-2026-04-30).
• “I'll just put the data in git.” Datasets blow past Git LFS limits. Version data by content hash with DVC / lakeFS / HF, and keep each version immutable.
• “I versioned my vectors, so RAG is reproducible.” Retrieval is a composite — corpus + chunking + embedding model + index config + prompt. A new embedding model invalidates the whole index (re-embed, don't mix).
• “Temperature 0 means it's reproducible.” Hosted LLMs aren't bit-exact (batched FP, routing). Aim for “pinned & traceable,” and record the run (model, version, params, fingerprint).
• “Editing the eval set to fix a typo is harmless.” It silently makes old scores incomparable. Make a new immutable version instead.
• “Rollback means a hotfix deploy.” With versioned artifacts, rollback is repointing a label / redeploying the previous manifest — seconds.

• An output is a combination of artifacts — prompt, model+version, params, RAG corpus+index, eval set. Version the full inference identity, not just the model's name, so you can reproduce, debug, roll back, and audit.
• Prompts: version as code (git + PR) and/or in a registry (Langfuse/LangSmith) with labels — production is a pointer you repoint for instant rollback.
• Models: pin the exact dated snapshot, never latest — a floating alias drifts silently (cf. L246 — CI/CD for Prompts & Evals). Register self-hosted models (MLflow/W&B/HF revisions).
• Data: too big for git → content-address it (DVC/lakeFS/HF), keep versions immutable, and version the eval set so scores stay comparable. RAG is a composite — a new embedding model forces a full re-index.
• Bind it into an immutable release.yaml manifest (semver for humans + content hash for machines), record it on every output as lineage, and accept “pinned & traceable,” not bit-exact. Next — L248: Scaling & Reliability Patterns — running this versioned system under real load.