Gating Agent Deploys on Replay Pass Rates From Production Trace Snapshots

Replay-based evaluation tests agent changes against real production traffic before deployment by freezing trace snapshots, running the candidate against recorded inputs and tool responses, and scoring its behavior against an approved baseline. Gating agent deploys on replay pass rates makes release approval a verifiable engineering decision. The gate blocks changes that fall short of calibrated quality, safety, or resource thresholds, while a canary catches failures that offline replay cannot show.

Why binary monitoring cannot catch agent quality regression

A production agent can return confident, well-formed, semantically wrong output while every span returns a 200. The request succeeds without the task succeeding. Unlike a deterministic service, an agent can select different tool sequences for the same prompt, and a syntactically valid action can still violate the user’s intent. Both can happen without setting off an error-rate or latency alert.

This gap leaves prompt edits, model updates, and tool-schema changes free to degrade quality without changing conventional health signals. The evidence sits inside the execution, including what the agent received, which tools it selected, and what those tools returned. A final response alone cannot explain that chain. If the regression signal lives in production traces, the deploy gate has to be built on traces.

What "replay" actually means for a non-deterministic agent

Replay does not mean reproducing an exact trajectory. At temperature > 0, the same inputs can lead to a different tool sequence, so a gate that requires the same sequence every time is flaky. The stable element is the fixture. Replay freezes the prompt inputs, supplies recorded tool responses without live API calls, runs the candidate change, and scores the result against a baseline.

That distinction determines what the fixture must preserve. It records every user message, tool-call argument, and tool response, together with latency, retry counts, and the exact model, prompt, and tool-schema versions. Capture happens at the boundary outside the model’s control, where the process receives its inputs, preserving what the agent actually observed.

A candidate can take a different path and still satisfy the task, provided its actions meet the defined constraints. Even when the wording looks familiar, a wrong state change is still wrong.

What a valid production trace snapshot set looks like

A snapshot set is valid only to the extent that its sampling represents the production work the gate must protect. Happy-path conversations are not enough for that purpose. Sampling should cover the full task taxonomy and give more weight to actions that change state, including writes, policy-gated operations, and multi-tool compositions. Adversarial and no-action tasks must also appear, because refusal or escalation can be the correct outcome. Without them, the suite rewards eagerness.

Account numbers and case IDs become stable, rehydratable placeholders through identifier stripping and tokenization. Each task also needs its initial system state, including every field the agent may modify, so that the resulting change can be evaluated. For multi-turn work, a simulated-user script represents the goal. This allows the task to require clarification when the opening message is underspecified.

Coverage needs to account for tool trajectories as well as topic labels. The set needs to test tool-selection boundaries, multi-tool composition, unsafe-tool denial, and recovery from tool timeout. Its size follows from that coverage requirement. Once a release cycle begins, the gate set stays frozen so test changes cannot masquerade as agent changes. A separate held-out set, together with periodic replay of sampled production traces, supports drift detection without altering the release reference mid-cycle.

How to score non-deterministic replays consistently

Consistent scoring begins with conditions that a test runner can evaluate directly. Structured assertions check whether the correct tool was called with valid arguments, the required state change occurred, a grounded citation resolves, and policy constraints were respected. These checks are cheap and deterministic to run. They cover the majority of attributable failures without inference cost, leaving semantic evaluation for questions that cannot be settled by inspecting state or structure.

The semantic layer uses embedding distance to flag shifted outputs, then an LLM judge to evaluate fidelity where that additional cost is justified. Distance alone cannot establish whether behavior improved or regressed, because a paraphrase may look different while a semantic error looks similar. The judge addresses that ambiguity against a grounded reference. Its temperature should remain at 0.0, with a fixed model version, so changes in the evaluator do not contaminate comparisons between agent versions. Sampling limits the more expensive evaluation to part of the fleet, so it does not require full-fleet replay.

Accuracy is only part of the pass condition. Ganglani’s harness guide specifies that CI gates must also cover step budget, tool error rate, latency, cost caps, and safety constraints because agent failures multiply across steps. Those constraints belong beside task assertions. Before blocking releases, the judge’s rubric scores must be tested against known regressions and known benign changes. Otherwise, the gate presents a pass rate without an established false-positive rate.

Where the replay gate lives in the deploy pipeline and what it blocks

The integration evaluation at Stage 2 runs after the pull request and before merge. It blocks candidate changes that fail replay requirements before they enter the merged release path. The trigger must include every prompt, model, tool-schema, or index change. Small edits can alter behavior without looking operationally significant.

The ordering of checks follows their cost and reach. Structured assertions run before semantic evaluation because they can reject clear failures without paying for judge inference. After merge, a canary with eval-gated rollback stands between the candidate and full rollout, covering behavior that the frozen offline environment cannot reveal. Both boundaries are required for this reason.

For a blocked change to be actionable, replay must capture the trajectory at each step. That record includes the tool name and full arguments, retrieval query and returned document IDs and scores, retry count, input and output tokens, step latency, terminal state, and final policy decision. These details turn rejection into an attribution mechanism. Empty retrieval hits point toward the index, while malformed arguments against a valid schema point toward tool descriptions and the system prompt.

Setting thresholds that catch silent regression without blocking legitimate improvement

A tight threshold can reject a legitimate improvement, while a loose one can admit silent regression. Neither error is harmless to the release process. The starting posture should therefore be a non-blocking dashboard, with the golden set running nightly and tracking judge-score delta and embedding centroid distance. Known regressions and benign changes form the calibration set. Blocking begins only after that evidence establishes how the scoring system responds to both.

Thresholds must differ by the requirement being evaluated. Safety requires exactly 1.0 on every vote, without a soft fail or a flag-and-watch exception. Factuality and tool-call accuracy instead use delta gating against main, with a per-rubric floor derived from observed judge variance on the calibration set. For borderline non-safety results, a change can be deployed behind a feature flag and watched through production monitoring for a defined window before full rollout.

Calibration itself remains a maintained engineering artifact. A miscalibrated enforcement layer can degrade the system by blocking useful changes or accepting harmful ones. Intentional behavior changes therefore require deliberate re-baselining, with approved new golden outputs appended as part of an explicit baseline update. That update must respect the frozen release cycle rather than change its reference while candidates are being compared. New production patterns can enter the held-out evaluation work without silently rewriting the active gate.

The failure classes the gate is specifically designed to surface

Tool schema drift exposes why output-only evaluation is insufficient. An agent may follow the contract it learned under the previous schema while the system enforces a changed contract, producing a silent mismatch without an error code. Production-length multi-step contexts can expose schema-compliance failures that benchmark scores understate. Replaying recorded tool calls and arguments against the new schema tests that contract directly. A plausible final answer cannot override a failed argument or schema assertion.

Workflow loops and runaway retries require a different mechanism. An agent can use up resources without raising an error, causing cost and latency to drift. A semantic judge may still approve the answer the agent gives in the end. Explicit step-budget and retry-count assertions catch the waste in the trajectory, while latency and cost caps keep task completion from excusing excessive execution.

Prompt chain drift builds up over a series of steps. Small errors add up until the final output no longer matches the task, even though each action may look plausible on its own. Trajectory-level checks between steps expose that accumulation where a final-output score cannot identify its progression.

Fail-plausible behavior needs a grounded semantic check. In this class, the agent converts an error into fluent, confident, false narrative, so successful transport and clean formatting provide no warning. The trace preserves the underlying tool response that the narrative misrepresents. A judge scoring against that grounded reference can reject the answer for semantic infidelity. This is where judge inference earns its cost, because structural correctness alone cannot establish that the account is true.

Building the gate as a repeatable engineering discipline, not a one-time setup

A gate built once and left alone decays. The snapshot set becomes stale, thresholds fall out of calibration, and its history becomes harder to interpret even while releases continue to pass. Scheduled replay provides the maintenance mechanism. The frozen golden set should run nightly against production at the same temperature settings, with judge-score delta and embedding centroid distance tracked over time. An upward drift in the delta is a reason to investigate.

The same review discipline that governs agent code must govern the harness. Changes to thresholds, golden tasks, and judge configuration need a reviewable change process because each can alter which releases pass. Informal updates erase that distinction from the record. Deliberate revisions preserve the meaning of comparisons and make an approved behavior change distinguishable from a weakened test.

Trajectory evidence must identify whether the responsible layer is the prompt, tool schema, retrieval index, or workflow logic, and the resulting incident should generate a new golden task. When a release ships without that replay validation, the failures seen in production have no bearing on release approval. The rigor is achievable because teams running production agents already generate the raw material in their traces.