Determinism Controls in Agent Replay Environments

Setting temperature to zero is the most common fix teams reach for when they want an LLM agent to behave the same way twice. It doesn't work, and understanding why it doesn't work is the entry point into a much larger engineering problem: determinism in agent systems is not one setting; it's a stack of separate controls, each governing a different layer of the system, and each one failing in its own particular way.

Temperature governs how a model samples from its output distribution. Setting temperature to zero produces greedy decoding: the model always picks the highest-probability token. That sounds like it should produce identical outputs every time, given identical inputs. But the logits feeding that sampling step aren't fixed. Cloud inference providers batch requests together on shared GPUs, and the floating-point math involved in matrix operations shifts slightly depending on what else is in the batch. Swapping an H100 for an A100 changes the numerical behavior of the underlying operations, even when executing what is nominally the same model. OpenAI's own documentation on its seed parameter and Anthropic's documentation on temperature=0.0 both state directly that neither guarantees reproducibility. This is an admission that the sampling knob doesn't reach the actual source of the variance.

Research measures the scale of this directly. Research testing five LLMs configured for deterministic output found accuracy swinging by as much as 15% across runs of the same prompt, with the total gap between the best and worst outcome reaching 70%, a spread too wide to treat as noise. That's not noise you can average away. In a single-call setting, this kind of variance is survivable, even ignorable. In a multi-step agent, where each step's output becomes the next step's input, small divergences don't stay small. They compound, so by the fifth or sixth tool call, an agent replaying a "known" trajectory can be somewhere else.

What replay requires: recording as the foundation of control

Systems engineering solved a version of this problem decades ago. Record-replay debugging, the technique behind reproducing bugs in game engines and diagnosing failures in distributed systems, rests on a simple premise: you can't control everything about a live system, so you capture what actually happened and play it back exactly. Applied to agents, this splits into two modes. Record mode runs the agent normally, against real LLMs and real tools, while intercepting and logging every external interaction it makes. Replay mode takes that log and feeds it back in the exact sequence it occurred, substituting recorded stubs for live calls so the agent receives verbatim what it received the first time.

What gets captured has to be exhaustive. For LLM interactions, that means recording the complete prompt, every sampling parameter, the model identifier and version, and the exact response tokens returned. For tool calls, it means recording the function name, the arguments passed, and the complete response, including errors and timeouts as well as successful cases. System state needs recording too: timestamps, random seeds, environment variables. And because agents increasingly route through planners and classifiers before ever touching a tool, the inputs and outputs of that decision logic need to be logged as well.

In practice, each recorded event gets a monotonically incrementing step ID, tied to a run ID, with a structured input/output payload attached. An append-only JSONL format handles this well: simple to write, simple to diff, simple to replay in order. None of this is exotic. It's the same discipline that distributed-systems engineers have used for years to reproduce race conditions, just pointed at a different kind of nondeterminism.

Controlling LLM sampling variance: the limits of model-layer controls

Model-side controls exist, and they help, but each one has a boundary past which it stops guaranteeing anything. Temperature=0.0 forces greedy decoding, but it says nothing about the raw logits the model produces before that step. Seed parameters, as offered by OpenAI, improve reproducibility, but the documentation is explicit that neither seed nor temperature=0.0 guarantees full reproducibility. Pinning a model ID matters for a more basic reason: if the model itself changes between the record run and the replay run, that change is a source of divergence with nothing to do with sampling.

ContractBench, a benchmark referenced in recent agent-evaluation research, treats this as a record-keeping discipline rather than a runtime flag. It pins model IDs, runs at temperature zero, and logs the pinned model ID into every reward.json file it produces. The pinned model ID recorded in every reward.json file lets anyone later verify which model actually produced a given result. A pinned setting that isn't recorded alongside the result it produced is a setting you can't later prove was actually in effect.

The more uncomfortable finding comes from a research effort that examined over 4,700 agentic runs and found the correlation between decision determinism and task accuracy to be r = -0.11, statistically indistinguishable from zero. Small models in a lower parameter range hit near-perfect determinism while scoring only 20 to 42% accuracy. Frontier models showed moderate determinism, in the 50 to 96% range, paired with accuracy that varied just as widely. No model in that research achieved both perfect determinism and high accuracy at once. That's not a tuning failure waiting for a better prompt or a lower temperature. It looks like a property of how current architectures work: the same mechanisms that let a model generalize and reason flexibly are the mechanisms that make its outputs sensitive to tiny numerical perturbations.

Controlling tool and API interactions: mocking, transport interception, and the noise-aware diff

Even a perfectly replayed LLM call doesn't solve the problem, because agents don't just talk to models, they talk to the outside world. A live API called during replay will return whatever it returns that day: different data, different headers, sometimes a different error. If the goal is to compare a replay run against a reference run, that live variability has to be removed from the equation before any comparison means anything.

Tool call recording has to capture the function name and the exact argument payload sent, and it has to capture the complete response, not merely the successful case. Partial results, error bodies, and timeouts are all part of the tool's behavior and belong in the trace just as much as a clean successful response does.

The agrepl framework handles this at the transport layer rather than the application layer. A man-in-the-middle proxy intercepts every external interaction as it crosses the wire and serializes it into a structured execution trace. Replay then runs inside an environment with zero outbound network access, strictly isolated, so there's no path for live traffic to sneak in and contaminate results. That isolation is the point: it's not enough to intend to replay from stubs, the environment has to make live calls physically impossible.

Even with that in place, comparing record and replay traces with a naive diff runs into a subtler problem. HTTP responses carry fields that change legitimately from one call to the next without reflecting any real difference in behavior: timestamps, CDN routing headers, rate-limit counters ticking down. Diff those fields literally and every replay looks like a failure, even a perfect one. A workable comparison has to know which fields are signal and which are noise, or the whole exercise drowns in false positives.

Controlling execution environment state: time, randomness, and session isolation

Three categories of environment state sit outside the model and outside the tool call, and all three can quietly break a replay if left uncontrolled. Time is the most obvious: any timestamp that ends up embedded in a prompt, a tool argument, or a routing condition will differ between the record run and the replay run unless it's frozen. Randomness inside the harness itself, seeds used for retrieval, for shuffling, for sampling from memory, needs to be fixed and logged just as carefully as the model's own sampling parameters. Session state, meanwhile, covers things like A/B test assignments, personalization flags, rate-limiter counters, and cookie or session tokens, any of which can silently change what a tool does with the same nominal input.

ContractBench addresses the time problem with a virtual clock: a deterministic clock substituted for wall-clock time so the evaluation can run on any date and the agent still sees the original timestamp it saw during recording. That decouples reproducibility from when the test actually happens to run, so re-running a benchmark six months later doesn't change the outcome.

The same benchmark applies SHA-256 hashing to verify the byte integrity of artifacts moving through the pipeline. And it uses SHA-256 hashing to verify the byte integrity of artifacts moving through the pipeline, things like presigned URLs and session tokens, so that a virtual clock and a hash check together produce something closer to a byte-for-byte determinism guarantee rather than a looser standard of "looks about right." Output that's merely similar can be argued with; output that's provably identical is what makes a benchmark trustworthy.

Formalizing what determinism means across the three layers: trajectory, decision, and faithfulness

An agent can land on the same final answer twice while taking two completely different routes to get there. The DFAH research treats that as a critical distinction rather than a technicality, because whether the path matters depends entirely on who's asking the question.

Three formal dimensions come out of that framing. Action Determinism asks whether the agent calls the same tools in the same order across runs. Signature Determinism is the stricter version: it requires that both the sequence of tools and the specific arguments passed to them match exactly. Decision Determinism, the loosest of the three, asks only that the final output match the reference run, regardless of what path the agent took to reach it.

Which one you need depends on the question you're actually asking. A regulator or a downstream validation system asking whether an agent reproduced the same result cares about Decision Determinism and nothing else. An engineer debugging a production incident needs Signature Determinism, because the whole point is figuring out where the path diverged. A team trying to attribute the cause of a change, say, after modifying the harness or swapping a tool implementation, needs both measured side by side, since a match on one and a mismatch on the other is itself the diagnostic signal.

A fourth dimension sits alongside these three: evidence-conditioned faithfulness. This checks that the agent's stated reasoning is actually grounded in the evidence it retrieved through its own tool calls during that run, rather than drawing on something memorized from training data. The DFAH research measures this through what it calls EvidGround, checking that claims made in the final decision line up with the evidence the trajectory actually surfaced. An agent can be perfectly deterministic and still be unfaithful, reaching the same answer every time for reasons that have nothing to do with what it just looked up.

Layer interaction: why per-layer control is necessary but not sufficient for valid replay signal

None of the controls above work in isolation, and that's the crux of the whole problem. Think of each layer's determinism as a fidelity number somewhere at or below 1. Model sampling control gets you close to 1 but not quite there. Tool mocking gets you close to 1 but not quite there either. Environment freezing does the same. Chain enough near-1 numbers together across a long agent trajectory and the combined fidelity doesn't decline additively, it declines multiplicatively. A small imperfection at each of three layers compounds into a meaningfully unreliable replay by the time you reach step ten.

Three specific failure patterns follow directly from this. A slightly different token sampled at step N can change which tool the agent calls, or which argument it passes, at step N+1. If that argument now hits a live API instead of a recorded stub, the entire rest of the replay is compromised, regardless of how carefully the tool mocking was built elsewhere. Timestamp leakage causes a related failure: when wall-clock time isn't frozen and a tool's expected input is time-keyed, the recorded response no longer matches what the tool now expects, so the mock either fails outright or gets silently misparsed. Schema drift is the quietest of the three failure modes. When a tool's API changes its response structure between the record run and the replay run, nothing throws an error; the agent just parses a malformed response as though it were valid, and the failure appears somewhere downstream, disconnected from its actual cause.

The CAAF framework's answer to this is to treat the problem as having three separate failure surfaces that need three separate defenses: context isolation, deterministic grounding, and oscillation control. None of the three closes the gap by itself, and the ablation results behind that framework back this up directly, showing that removing any one of the three pillars degrades outcomes in a way the other two don't fully compensate for. Determinism in agent replay is a layered discipline built and maintained across every level of the system, not a property you switch on. It's a layered discipline, and a system is only as reproducible as its weakest uncontrolled layer.