Sampling Historical Conversations to Surface Agent Regression Blind Spots

To regression test agent changes before deployment, replay historical production conversations sampled by arc complexity and failure shape, rather than relying on isolated prompts or synthetic benchmarks. Attribute each recurring failure to its responsible layer, then test the proposed fix against the full affected arcs. Validation must show that the failure pattern is gone and that later state and nearby conversation types still work.

Why static eval sets cannot see cross-turn agent regressions

LLM-as-judge plus a growing eval set can appear to provide expanding coverage. That assumption fails when coverage is measured by the number of cases rather than the kinds of behaviour those cases expose. A static eval set is a snapshot, while production changes through model updates, prompt edits, and tool schema changes. Each can introduce silent drift between the tested conditions and the running agent. More cases do not necessarily close that gap.

Zhang et al. (2026) studied a deployed multi-turn food-and-beverage ordering agent using exhaustive human transcript review as ground truth. The built-in LLM judge caught well under a quarter of human-confirmed systematic problems. Its misses were not randomly distributed across defects. The judge reliably caught turn-local problems, including fabricated statistics and replies in the wrong language, but systematically missed confirm-gate lockout, cart hallucination, escalation lockout, and stale referents. Those failures required reasoning across turns.

The judge’s rubric had three coarse axes, which were intent, brand-voice, and personalization. It lacked axes for state-tracking, guardrails, and recovery, where most confirmed defects clustered. Cross-turn violations were therefore misfiled as brand voice and never reached the operational gate. A more sensitive judge cannot repair a rubric that has no operational category for the behaviour being tested.

Adding harder examples from already recognised defects can preserve the same omissions in a larger collection. The root cause is how cases enter the set. Teams build evals around problems they already know, rather than the arc-level failure shapes still hidden in historical production conversations.

The failure shapes that only appear across a conversation arc

Confirm-gate and transaction-state failures hinge on how later actions line up with earlier commitments. A reply can sound appropriate while contradicting what the user has already confirmed or what the cart should contain. The defect lies between those turns. In the ordering study, nearly every round whose raw judge note described a confirm-gate or cart-state defect was scored as brand voice, and none reached an operational failure. Detecting the inconsistency in prose was not enough to classify it correctly.

An API may add a required field or deprecate an endpoint while the agent continues using the old signature. Using the old signature then produces unexpected results across a sequence of calls. In a workflow loop, the agent repeatedly invokes a tool with the same incorrect parameters despite receiving an error. A single call shows an unsuccessful attempt, but repeated identical calls show that the plan is not changing.

Temporal blindness is harder to identify without the timeline. Cheng et al. (2026) identify failures to account for real-world time between messages when making tool-call decisions. The agent can then rely too much or too little on earlier context. A context reference may look reasonable when read beside its source message but become inappropriate once the elapsed time is considered. Its signature is a mismatch between the context being used and the timeline.

Memory and provenance degradation can leave the final wording apparently intact. The agent produces a correct-sounding resolution even though its source is degraded or absent. Outcome-only evaluation can score a grounded answer and a hallucinated answer identically. Only the relationship between the resolution and its supporting evidence distinguishes them. This makes provenance a property of the arc.

How to sample historical conversations to surface these failure shapes

The method starts with tail-based sampling. Retain every failed, expensive, or anomalous trace in full, while retaining a much smaller sample of happy-path conversations. Uniform or recency-weighted sampling does not set out to keep the conditions that make cross-turn failures visible. Keeping the full trace also preserves the earlier state needed to interpret a later anomaly.

Next, split the conversations kept into strata by arc complexity. Arc length and the tools used provide concrete dimensions for organising those strata. Within each stratum, select conversations by the failure signatures they contain. This separates the question of how involved a conversation became from the question of what may have gone wrong. A successful-looking resolution should not remove an otherwise suspicious arc from consideration.

When the same tool-call signatures repeat, they flag possible loops. A tool-call error followed by a continued successful-looking reply identifies a possible drift failure. References to context more than two turns back identify candidates for state-tracking or temporal review, while resolutions without a traceable confirming tool call or retrieval identify provenance candidates. Include a stratified slice of low-judge-score conversations as a confirmation layer for the turn-local defects the automated gate does detect.

A confirm-gate candidate needs the earlier confirmation, the later handling of transaction state, and the intervening turns that connect them. Record each candidate using its conversation ID, arc length, tentative failure shape, tools used, anomaly turn indices, and a link to the full trace. Keep transcript content only in the trace.

Replay constraints also affect what must be captured. Without the original environment, trajectories may need to be generated without live tool calls, with all available tool definitions hardcoded during data normalisation. Plan for this at capture time. Otherwise, a conversation kept for replay may show a failure but lack the tool context needed to test a proposed change.

What the sampled conversations actually reveal that the eval set was hiding

The ordering study’s second batch makes the coverage gap concrete. Human transcript review identified 23 distinct ledger defects and 7 new cross-cutting patterns that the automated gate had not flagged. Reviewers reasoned across the conversation arc where the rubric could not. The additional information was not merely a harsher assessment of individual replies. It was evidence of recurring relationships between earlier state and later failures.

A single confirm-gate lockout is a support ticket. The same lockout across sampled arcs becomes a systematic failure pattern whose cause can be investigated. Clustering by shape, rather than session ID or timestamp, brings those relationships into view. When loop signatures cluster around a specific tool and input range, the search narrows to its schema or the planner prompt. When provenance degradation clusters beyond an arc-length threshold, it points instead toward context-window management.

These clusters connect a failure to the conditions under which it recurs, making it possible to distinguish a transaction-state problem from a general weakness in answer quality. They also prevent every plausible but wrong reply from being treated as the same defect. Replies that sound alike can have different causes, while replies that sound different can reflect the same broken state transition.

Harness faults are important, but they are not universal. Failures surfaced this way can originate in prompts, tool schemas, workflows, or memory, with the model producing plausible output from degraded state. But the empirical taxonomy in the notes finds more faults on the model side. A cluster therefore supplies an attribution candidate. Investigating cluster exemplars makes the next step more tractable than treating every broken conversation as an unrelated incident.

Attributing the root cause to the specific harness layer responsible

Attribution requires finding where the state first became unsuitable for the task. An agent that fails at turn 8 may have received corrupted state at turn 3. The visible failure is only the starting point. Relevant evidence can be sparse and distributed across distant turns, so diagnosis must connect the observed failure to the prompt, tool schema, workflow, memory, or product logic that produced it.

Each failure shape narrows that search differently. Repeated calls to the same tool with the same parameters point toward the planner prompt or an error-response schema that fails to produce a state change. When a call succeeds in structure but returns unexpected data, it points instead toward a mismatch between the harness’s tool definition and the live API contract. Those signatures resemble each other only at the output level. One concerns how the agent responds to feedback, while the other concerns what the tool contract means.

Confirm-gate lockout directs attention to workflow state-tracking or the prompt’s handling of confirmation signals. A resolution without a retrievable source directs attention to memory or context-window management. Tool decisions that ignore elapsed time point toward missing time-aware context injection or planning logic. These are candidate locations. The full arc is needed to identify which layer introduced the defect behind each signature.

For tool drift, CI-side schema snapshot testing can work alongside checks of the traces. Record tool schemas at each pipeline run, compare them on each run, and fail on breaking changes. This catches schema mismatches before they become production attribution problems. For defects already present, the diagnosis must still name a reviewable change. “The agent failed at turn 8” does not do that, while “the planner prompt does not update its tool-call plan when the tool returns a non-2xx with a schema-mismatch body” identifies a specific behaviour to repair.

How do you validate agent changes against production traces before shipping?

Replay the proposed fix against the cluster of historical traces that exposed the failure. The validation target is the disappearance of the failure shape. For a loop fix, the relevant question is whether the agent stops repeating the incorrect call after receiving feedback. For a confirm-gate fix, it is whether confirmation and transaction state remain consistent throughout the arc. The sampled cluster supplies the conditions that the original eval set omitted.

The validation rubric must name the behaviour being tested. State-tracking, guardrails, and recovery cannot be inferred from a general quality score when those dimensions caused the original blind spot. A rubric needs a state-tracking axis to show that a confirm-gate repair holds. The pass/fail decision must also cover the whole conversation. A change that resolves the failure at turn 4 but breaks state at turn 7 has not closed the regression.

Replay design must account for unavailable tools. Hardcoding tool definitions during data normalisation prevents validation from depending on their availability at replay time and supports a controlled replay setup. Where the original environment is unavailable, the trajectory may need to run without live tool calls. That constraint must remain explicit when interpreting the result. A replay tests behaviour under its supplied tool conditions, rather than recreating an unavailable environment by assumption.

To test for adjacent regressions, use a golden-set replay of curated traces with expected outputs or rubric scores through the current production pipeline on a schedule. Once validation succeeds, add the corrected behaviour to that curated set. Historical sampling then does more than discover a blind spot. It gives the next release an arc-level test that can catch the same failure without rediscovering it in production.