Ask an engineer how to make a language model predictable and the answer is almost always the same: set the temperature to 0. At temperature 0 the model stops sampling and takes the single most probable token at every step. On paper that is a deterministic algorithm. In practice the endpoint still answers differently next time you ask.
Temperature 0 removes the dice. It does not remove the machine, and the machine is where the real story is.
The usual explanation is half wrong
The standard account goes: GPUs do floating-point math in parallel, threads finish in random order, additions happen in a different sequence each run, and the tiny rounding differences snowball into different outputs.
The first half is true. Floating-point addition is not associative. With finite precision, (a+b)+c can differ from a+(b+c), and the order of a sum changes the answer. In one demonstration from Thinking Machines Lab, summing the same eight floating-point numbers in random orders produced 102 distinct results (source).
The second half is where the account breaks. The same researchers ran one 2048x2048 matrix multiplication on a GPU a thousand times and got bitwise identical results every time. Modern GPU kernels avoid the nondeterministic primitive that would break reproducibility, the atomic add, in almost everything a language model's forward pass runs. Two identical forward passes, given identical batch shapes, produce identical logits. The inference server, viewed from inside, is deterministic.
What it is not is batch-invariant.
Your output depends on your neighbors
Every production inference engine uses continuous batching: your request is processed alongside whatever other users' requests happen to be in flight, and the batch grows and shrinks from step to step. That is how providers keep per-token prices low.
Here is the problem. GPU kernels for matrix multiplication, attention, and normalization pick their internal strategy based on the shape of the batch they are running on. At large batch sizes it is fastest to assign each row to its own core. At small batch sizes the cores would sit idle, so kernels split the reduction across multiple cores, switch to smaller tensor-core instructions, or (in decode attention) use Split-KV strategies like FlashDecoding, where the number of splits is chosen to saturate the GPU at the current load.
Because floating-point sums are order-dependent, each strategy change reorders additions, which shifts logits by a rounding unit or two. Individually irrelevant. Except the batch size a request lands in depends on server load, and server load is, from your side of the API, a random variable.

As Thinking Machines put it: compose a property the kernels are not invariant to (batch size) with nondeterminism of that property (server load), and you get a nondeterministic system. One flipped logit late in a long generation can flip the argmax token. From there the two runs have different prefixes, and every subsequent step diverges (analysis). vLLM's own FAQ concedes the mechanism: outputs can vary because the same requests "might be batched differently due to other concurrent requests, changes in batch size, or batch expansion in speculative decoding" (docs).
None of this is GPU-specific. The authors note that CPU and TPU serving stacks carry the same batch-size dependence, and mixture-of-experts routing over a batch is one more input that changes with your anonymous batch neighbors.

How far off can "off" be?
The divergence is not academic. Thinking Machines sampled an open-weights Mixture-of-Experts model (Qwen3-235B-A22B-Instruct) a thousand times with temperature 0 and a fixed 1,000-token budget on the prompt "Tell me about Richard Feynman". The runs produced 80 unique completions, and the most common single answer appeared only 78 times. With the batch-invariant kernels they wrote, a dense Qwen3-8B model produced bitwise identical output across a thousand runs.
On benchmark scores, a systematic study across five LLMs, eight tasks and ten repetitions per configuration found that models "configured to be deterministic" showed accuracy variation of up to 15% across naturally occurring runs, with best-to-worst gaps of up to 70% on some tasks. No model delivered repeatable accuracy on every task (arXiv:2408.04667).
What the vendors promise
The documentation is honest, in a hedged way. OpenAI's API docs describe Chat Completions as "non-deterministic by default" and offer a seed parameter plus identical other parameters for "(mostly) deterministic outputs". The system_fingerprint response field exists to tell you when OpenAI changed the configuration under you, because that, too, will change your outputs (docs). Anthropic has no seed-style guarantee on the hosted product at all: asked for deterministic behavior in Claude Code, an Anthropic engineer replied in July 2025, "We don't plan to support deterministic LLM-based behavior," citing routine system prompt and tool changes (GitHub issue).
Determinism costs throughput, which is why nobody ships it
The fix exists and it is understood: write batch-invariant kernels, forcing every operation to use one reduction strategy regardless of batch shape. Thinking Machines' batch-invariant matmul runs about 20% slower than the vendor library cuBLAS. End to end on a single GPU serving Qwen3-8B, their unoptimized deterministic stack took 55 seconds for 1,000 requests where stock vLLM took 26; a better attention kernel brought it to 42. That overhead is real money at hyperscale, which is why serving stacks default to the fast, shape-dependent kernels (post).
The pressure to change is coming from training rather than chat. Reinforcement learning assumes the rollout engine and the training engine compute the same policy, but different numerics silently turn on-policy RL into off-policy RL. With batch-invariant kernels, Thinking Machines ran RL training where sampler and trainer agreed bitwise and reward curves stayed stable without correction terms. A follow-up paper has extended the recipe to tensor-parallel layouts, where kernel behavior also changes with the number of GPUs a model is sharded across (arXiv:2511.17826).
What to do with your own stack
- Treat
seedas a hint, not a guarantee. OpenAI says "mostly", and vLLM notes that even request seeds do not erase precision discrepancies. - Version-pin. A changed
system_fingerprint, a deprecated model alias, or a silently updated hosted model breaks reproducibility faster than any rounding error. - Run evals N times, report the spread. If a benchmark gap under 15% matters to your decision, one run tells you almost nothing.
- If bitwise reproducibility is a hard requirement (audit trails, regression tests, RL pipelines), you will likely have to self-host and accept the throughput tax, or build with batch-invariant kernels.
The models are not changing their minds about you. They are just being batched with strangers.
Sources
- Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference" (Horace He, September 2025) - https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference
- vLLM documentation, "Frequently Asked Questions" - https://docs.vllm.ai/en/v0.7.0/getting_started/faq.html
- OpenAI API documentation, "Advanced usage: Reproducible outputs" - https://developers.openai.com/api/docs/guides/advanced-usage
- Anthropic Claude Code repository, issue #3370, "Claude CLI produces non-deterministic output for identical inputs" - https://github.com/anthropics/claude-code/issues/3370
- Atil et al., "Non-Determinism of 'Deterministic' LLM Settings" - https://arxiv.org/abs/2408.04667
- Zhang et al., "Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch" - https://arxiv.org/abs/2511.17826
