Determinism and Reproducibility in Automated Eval Pipelines

Software engineering has always rested on a quiet assumption: run the same test against the same code twice, and you get the same result. AI models break that assumption at the architectural level, so a single eval run tells you almost nothing about whether a system works. What it tells you is what happened once, under one draw from a distribution of possible outputs, and treating that draw as a verdict is the central error running through how most organizations evaluate their AI systems today.

The mechanism starts inside the forward pass itself. GPU floating-point addition is not associative: the order in which partial sums get combined during parallel computation can change the final value enough to flip an argmax decision between two runs of the identical model on the identical input. That means even a deterministic-sounding setting like temperature=0 doesn't guarantee a fixed output, because the hardware path the computation takes can itself vary run to run.

Production systems add further layers of drift on top of this. None of this is a bug waiting to be patched. It is the architecture behaving as designed. Pinning temperature to zero solves only one small piece of a much larger problem.

The consequence for engineering process is structural. CI/CD gates, regression tests, and release approvals all assume that running the same check twice produces comparable results, and that assumption is what makes a gate meaningful. The pass@k versus pass^k distinction captures the stakes precisely: measured success across agentic benchmarks includes a share of results, with gaps reaching up to roughly 25 percentage points, that comes from stochastic exploration across multiple attempts rather than from any deterministic capability the system reliably possesses. A release gate built on a single eval run cannot see that tail.

How badly the reproducibility gap is documented in practice

The scale of the gap is now measurable, and the measurements are not encouraging. ReproEvalCard, published by Pattnayak and Bhatia at ACL, audited 55 pipeline-based LLM papers published between 2022 and 2025 and found that randomness controls were missing from the majority of them, with intermediate execution traces missing from a majority as well. Without logged randomness controls and execution traces, nobody, including the original authors of the papers, can reproduce the reported results.

The audit also surfaced a broader definitional problem: the artifacts actually required to reproduce an LLM pipeline evaluation extend well past model weights and datasets. Prompts, judge configurations, retrieval snapshots, and intermediate traces all belong on that list, and their availability across published work had not been systematically examined until ReproEvalCard did it. That absence of scrutiny matters because each of those artifacts can independently shift a result, so a paper that publishes its dataset and model checkpoint while omitting its judge configuration has published half an experiment.

None of this is confined to academic publishing. An eval pipeline that passes a model on Tuesday and fails it on Thursday without any code change is a predictable symptom of a system whose sources of variance were never controlled.

A second, less obvious dimension of the same gap comes from research by Ning et al. at Carnegie Mellon on what they call the implementation lottery. Seed control alone does not fix this, because the variance here comes from implementation choices made before any seed enters the picture. Reproducibility requires controlling both layers at once.

Controls needed before comparing eval runs

Making two eval runs comparable means pinning every identifiable source of variance in the stack, because any element left unpinned can silently shift the distribution being sampled without announcing itself. Think of the required controls as a ladder, where each rung buys a marginal improvement in reproducibility, and even at the top rung, some drift should still be expected and caught.

The first rung is the model version itself. Calling a specific, dated snapshot rather than a floating alias like "latest" applies across every major provider, because an alias can point to a materially different model from one week to the next without any visible change in the calling code.

The second rung is explicit decoding configuration: setting temperature=0 and top_p=1 directly in the call. Explicit values beat implicit ones even when the numbers happen to match, because a provider can change its defaults without notice while a pinned value in your own configuration cannot drift underneath you.

The third rung is the seed. Using the same integer across runs is necessary but not sufficient on its own; for OpenAI's API, teams also need to record and validate the system_fingerprint returned with each response. A changed fingerprint signals that the backend itself changed, and reproducibility cannot be assumed once that has happened, regardless of how carefully the seed was pinned.

For teams running self-hosted inference, the fourth rung covers the full serving stack: the vLLM or TGI version, the model checkpoint hash, the dtype, and the batch configuration all need to be pinned, with deterministic flags such as vLLM's VLLM_BATCH_INVARIANT=1 set and speculative decoding disabled.

Reasoning models add a further complication at the top of the ladder: both the model version and the reasoning budget need to be pinned, and even with every other control in place, these models should be expected to show more run-to-run drift than non-reasoning models do. Finally, the eval dataset, metric definitions, grader configurations, and policy slices all need to be versioned as artifacts in their own right. A changed prompt or a changed judge configuration produces a changed experiment, not a re-run of the prior one, and treating the two as equivalent is how teams end up comparing results that were never comparable to begin with.

Pinning alone is not enough: the grader's own non-determinism

Pinning the model under test solves only half the problem, because the grader sitting on the other side of the evaluation can introduce its own non-determinism even when everything upstream has been locked down. A perfectly pinned model can still produce a misleading eval result if the grader relies on surface-level matching rather than checking what actually happened, or if the grader is itself an LLM subject to the same sampling variance as the system it's judging.

Surface-level grading, checking a transcript to see whether an agent claimed to have completed a task, is how evaluation pipelines end up passing agents that report success confidently while the underlying system state never changed. Outcome verification, actually checking the end state of the system rather than the agent's account of it, is what separates an eval that protects users from one that protects a score on a dashboard.

LLM-as-judge grading compounds the issue further. A judge model that is itself non-deterministic adds a second sampling layer on top of the model under evaluation, so the composite result reflects the product of two distributions. ReproEvalCard's audit identifies judge configurations as exactly the kind of artifact that must be versioned and logged alongside model weights and datasets, since omitting the judge configuration makes even a carefully pinned model-side eval non-reproducible overall.

The ReproRepo framework from Li et al. at Carnegie Mellon illustrates a related distinction from a different domain. The lesson transfers directly: semantic matching and exact matching are different capabilities, and eval design has to state which one a given check actually requires before the grader can be trusted to deliver it.

Scoring eval results as distributions, not point estimates

A single number from a single run is one draw from a distribution, and reporting it as if it settles the question misrepresents what was actually measured. The honest approach is to run multiple rollouts per prompt and report the mean, the standard deviation, and the worst case for each metric, because a wide tail signals an unsafe system even when the mean looks perfectly acceptable.

In practice, this takes the shape of a repeatable pattern. Report the mean plus or minus the standard deviation, alongside the worst-case result, for every metric that matters to the release decision. Then plot the distribution itself, because the shape of the tail carries diagnostic information that a single mean conceals entirely; a narrow, well-behaved distribution and a bimodal one with a thin failure spike can produce the identical mean while representing two very different systems.

The obvious objection is cost: running N rollouts instead of one multiplies the compute and time spent on every eval. The implementation lottery research from Ning et al. offers a direct answer rather than a hand-wave, by modeling the tradeoff explicitly. A single-run score that happens to clear a threshold is a guess dressed up as a policy decision.

Semantic graders and verification requirements

Because non-deterministic models produce outputs that are semantically identical but textually different across runs, grading on byte-exact match conflates genuine regressions with harmless surface variation inside the same pipeline. A model that answers correctly in different words on two separate runs should not register as a failure, and a grader built around literal string matching cannot tell the two apart from an actual change in correctness.

Designing for semantic equivalence rather than byte equality means the evaluator has to recognize that two different phrasings of a correct answer are both correct, while also recognizing that a confident, fluent, well-phrased answer that is simply wrong stays wrong regardless of how convincing it reads. Those are two distinct judgments, and a grader that only measures fluency or surface similarity to a reference answer will systematically miss the second one.

Outcome verification extends this principle into agentic systems specifically. Checking the actual end state, did the record update, did the lead route to the correct queue, did the downstream workflow fire, grounds semantic evaluation in something the model cannot talk its way around. Grading on the transcript alone measures how confidently a system describes its own actions rather than whether those actions actually occurred, and those two things are not the same measurement even when they produce identical-looking transcripts.

ReproRepo's evaluation methodology offers a useful template for how to structure this distinction formally, by treating exact match and semantic match as two separate metrics. The best agent in that study surfaced at least one semantically related, human-reported blocker for approximately 90% of papers evaluated, while still falling short on exact localization of the specific problem. That gap between the two metrics is itself the argument: graders should operate at the semantic level as the primary check, and track localization precision separately, as its own metric, wherever localization actually matters to the decision being made.

Structuring CI/CD pipelines for trustworthy probabilistic gates

A trustworthy AI eval gate inside a CI/CD pipeline is not a single threshold check sitting at the end of a build. It functions as a staged policy that separates deterministic invariants from probabilistic quality checks and applies a different enforcement rule to each category, because treating every check as either a hard pass or a hard fail ignores the distributional nature of the thing being measured. Deterministic invariants, properties that must hold regardless of sampling variance, such as a response respecting a schema or avoiding a banned category of content, should fail the build directly and block without negotiation. Probabilistic quality checks need a different structure entirely: an absolute floor below which no result is acceptable, an accepted-baseline tolerance that allows for normal run-to-run variance, repeated generations rather than a single sample, and human alignment on where those tolerances should sit before the gate is trusted to block anything automatically. This staged architecture, laid out in Confident AI's 2026 guide to CI/CD evaluation, also recommends running critical blockers on every pull request, running broader variance and slice checks on a nightly cadence, and running full coverage before any promotion to production. That staging reduces friction on fast-moving day-to-day changes while keeping rigorous gates in place at the promotion boundaries where the cost of a missed regression is highest. Ownership belongs inside the policy as well, not bolted on afterward: engineering maintains execution of the pipeline, while QA, product, and domain experts own the review and the underlying quality decisions about where tolerances and floors should be set.

None of this holds together without an evidence trail. Versioned release evidence is the artifact that makes a gate meaningful over time: pinning the dataset, the metrics, the expected behavior, and the policy slices, while preserving the outputs, the score reasons, the traces, the deltas, the annotations, the approvals, and any overrides, so that any failed or approved release can be reproduced and audited later. Without that trail, a passed gate from three months ago is an unverifiable claim.

Several tools now build CI/CD-native evaluation around these principles. Noveum AI offers automated agent evaluation and remediation, with a free tier and paid plans starting from $69 a month, built on an SDK and the NovaEval framework, covering automated scoring, synthetic agent sessions, root-cause analysis, and backtested fix proposals. Arize Phoenix provides an observability and evaluation platform, free to self-host with an AX tier from $50 a month, released under an ELv2 open-source license, wiring pytest, Vitest, and Jest criteria directly to trace and experiment evidence. Each represents a different point of entry into the same underlying discipline: pinning, statistical scoring, and semantic grading working together as one system, rather than as three separate improvements layered on top of an eval pipeline that still trusts a single run to answer a question only a distribution can.