The Probabilistic Nature of AI (Determinism, Seeds)

You've now met every knob that shapes a model's output. Time for an honest question that trips up even experienced engineers: how random is a language model, really — and can you make it stop?

The everyday experience says very random: ask the same question twice, get two different answers. We know why — sampling (Lesson 2). But here's the twist that surprises almost everyone, including people who've shipped LLMs to production:

Setting temperature = 0 does not guarantee identical outputs — and the real reason is not the one most blog posts give.

This lesson explains what's actually going on, using the most current understanding of LLM inference (2025 research). It will save you hours of chasing "flaky" behavior that isn't a bug in your code.

You'll learn:

• Why models are probabilistic by design
• How temperature 0 and the seed parameter reduce randomness
• The myth: why temperature 0 is still not truly deterministic
• The real culprit: batch invariance + floating-point math
• How to get as reproducible as possible — and its hard limits

Start with the intended randomness. Recall the pipeline: the model produces a probability distribution (softmax), then samples a token from it. Sampling is a weighted dice roll — so the same prompt naturally yields different tokens, and different tokens snowball into different responses.

This is a feature, not a flaw. It's where a model's creativity, variety, and natural feel come from (Lessons 2–3). A model that always said the exact same thing would be a worse writer and a duller assistant. So at any temperature above 0, expect variation — that's the system working as designed.

When you don't want variation — evals, tests, structured extraction, caching — you have two levers:

• temperature = 0 → effectively greedy decoding (Lesson 2): always take the highest-probability token. This removes the sampling randomness — no more dice roll.
• seed → fixes the pseudo-random number generator used for sampling. With the same seed and the same parameters, the dice land the same way every time, so a sampled run becomes repeatable. (Most major APIs added a seed around late 2023.)

A companion field, system_fingerprint, tells you which backend configuration served your request — because if that changes, reproducibility can silently break (we'll see why). Set temperature = 0, pass a seed, and you'd think you'd get the exact same output every time. Usually you mostly do — but not always. Here's the part the internet gets wrong.

Run the same prompt at temperature = 0 against a busy commercial API a few hundred times and you'll occasionally get a different answer. This shocks people, because temperature 0 removed the sampling randomness — the dice aren't even being rolled.

The common explanation — "it's floating-point rounding errors" — is only half-right and misses the actual mechanism. As 2025 research from Thinking Machines Lab put it bluntly: temperature 0 is necessary but wildly insufficient for determinism. To see why, we have to look below the model, into how inference actually runs on a GPU.

Here's the real chain, and it's a systems problem, not a sampling one:

1. Your request is batched with other users'. To be efficient, inference servers bundle many incoming requests and process them together. How many? Whatever happened to arrive at that moment — so the batch size changes with server load.
2. Batch size changes the order of GPU math. Core kernels (matrix multiply, attention, normalization) sum up many numbers, and they split that work across the GPU differently depending on batch size — so the additions happen in a different order.
3. Floating-point addition is non-associative. At the bit level, (a + b) + c ≠ a + (b + c). Different summation order → logits that differ by a hair (think 3.20001 vs 3.19998).
4. A near-tie flips. Usually that hair doesn't matter. But when the top two tokens are nearly tied, the tiny difference flips which one is the argmax → a different token → and from there the whole continuation can diverge.

So it's not that your request is "randomly" computed — it's that the result depends on who else you were batched with. Same input, different batch, slightly different logits, occasionally a different word.

The encouraging part: this is fixable, just not by default. Thinking Machines Lab built batch-invariant versions of those kernels — math that gives the same result regardless of batch size — and got 1,000 identical runs out of 1,000. True determinism is possible; it just requires inference infrastructure designed for it, which shared commercial APIs generally aren't.

Batching is the big one, but a few other things move outputs even when your code is unchanged:

• Model & backend updates. Providers patch and re-deploy models continually. A different system_fingerprint means the thing serving you changed — so identical inputs can produce different outputs across days/weeks.
• Hardware differences. Different GPU types/driver versions can compute slightly different floating-point results.
• Mixture-of-Experts routing. In MoE models, which "experts" handle your tokens can depend on the batch too, adding another batch-dependent wrinkle.

The takeaway: a hosted model is a moving target, not a frozen function. Pin versions where you can, and treat exact-output reproducibility as something you engineer for, not something you assume.

Practical ladder, from easiest to strongest:

1. temperature = 0 — removes sampling randomness. Necessary, not sufficient.
2. Pass a seed — makes any sampled runs repeatable; pair with checking system_fingerprint to detect backend changes.
3. Pin the model version (e.g. a dated snapshot, not a floating alias) so an update doesn't silently shift behavior.
4. For true determinism, you need batch-invariant inference — realistically a self-hosted or specialized setup (e.g. fixed batch size, batch-invariant kernels). Shared APIs under variable load can't promise it.

Set expectations accordingly: on a commercial API, treat reproducibility as best-effort, not guaranteed.

This single insight prevents a lot of wasted debugging:

• Don't assert exact-string equality in LLM tests/evals. Even at temperature = 0, outputs can vary slightly. Compare with tolerance or semantic/structured checks, not assert output == "...".
• "Flaky" temp-0 behavior usually isn't your bug. It's batch-variance in the provider's stack — chasing it in your code is futile.
• Caching on exact outputs is fragile. Cache on the input (prompt + params), not on a hash of the output you hope is stable.
• Log seed and system_fingerprint for anything you need to reproduce or audit later.
• If you truly need determinism (regulated settings, reproducible research), plan for self-hosted batch-invariant inference — it's a deliberate infrastructure choice.

And a bridge to the finale: this same probabilistic machinery — a model confidently sampling a plausible token that happens to be wrong — is the seed of the section's last topic: where hallucinations come from.

Predict: deterministic? You send the same prompt at temperature=0, twice, to a busy production API. Identical outputs?

→ Usually — but not guaranteed. temperature=0 removes the sampling randomness, yet batch-variance (your request batched with other users → different float-summation order) can still flip a near-tie token. So if your eval 'flakes' at temp 0, that's not your bug — and you shouldn't assert exact-string equality.

• "Temperature 0 makes the model deterministic." No — it removes the sampling dice, but batch-variance (system-level) can still flip near-tie tokens.
• "The nondeterminism is just floating-point rounding." Incomplete — float non-associativity is the mechanism, but changing batch size is what triggers different summation orders. At fixed batch size, kernels are deterministic.
• "seed guarantees identical outputs." No — it fixes the sampler's RNG, not the batch your request lands in or the backend serving it.
• "A hosted model is a fixed function." No — providers update models/hardware; watch system_fingerprint and pin versions.
• "Determinism is impossible for LLMs." No — batch-invariant kernels achieve it; it's just not the default on shared APIs.

• LLMs are probabilistic by design — sampling makes the same prompt give different outputs (a feature for creativity).
• temperature = 0 (greedy) + a seed reduce randomness, but they're necessary, not sufficient, for determinism.
• The real culprit is batch invariance: your request is batched with others, batch size shifts the order of floating-point GPU math, (a+b)+c ≠ a+(b+c) perturbs the logits, and near-tie tokens flip → divergent outputs.
• Other drift: model/backend updates (system_fingerprint), hardware, MoE routing. Treat hosted models as moving targets.
• True determinism needs batch-invariant inference; on shared APIs, reproducibility is best-effort — so test with tolerance, not exact equality.

Next — the section finale: we follow this probabilistic nature to its most talked-about consequence — where hallucinations come from, and why a model can be so confidently, fluently wrong.