Trace Pattern Triage Into Structured Prompt and Logic Fix Candidates

Production agent observability becomes a compounding improvement loop when teams turn recurring trace patterns into localized failures, typed harness attributions, bounded fix candidates, and replay-validated changes. This takes a careful search that goes beyond reading a trace once. Historical replay tests the proposed repair, while teams identify the next pattern to look into through continued production monitoring.

Why triaging production trace patterns is harder than it looks

Long-horizon agent executions accumulate actions, tool calls, retries, and intermediate outputs before a failure becomes visible. The decisive error may sit much earlier. Relevant evidence can also be distributed across interacting agents, separating the action that introduced a fault from the action that finally exposed it. The gap between a visible failure and an actionable fix therefore requires diagnosis.

Consider an agent that reports “no results found” after an upstream agent formulated an overly restrictive query. The reporting step exposes the failure but did not create it. A diagnosis centered on that final response assigns responsibility to the symptom, leaving the query decision untouched. In a multi-agent execution, the evidence needed to make that distinction may belong to different agents and distant parts of the history.

Raj et al. at Scale AI frame root-cause analysis as a search problem in which the attribution system tests plausible explanations and rejects them when contradictory evidence appears. This explains why reading a trace, spotting an error, and writing a fix right away can go wrong. Plausibility is not enough to establish causal responsibility. A trace may contain several mistakes, and the most conspicuous one need not explain the outcome.

TelemetrySuffBench shows that terminal status can support failure detection without supporting causal attribution. Failure detection and causal attribution remain distinct even when the final error is clear and consistently recorded.

What telemetry can and cannot tell you about failure origin

Detection evidence and attribution evidence are not the same fields. Span status and tool call outcomes can establish that something failed without preserving the decision that caused it or the links connecting that decision to later steps. More records of the same kind do not close this gap. Instrumentation choices determine which causal questions a later investigation can answer.

TelemetrySuffBench found that when decision content was removed, origin-step accuracy fell to zero for every model tested. Removing provenance links also caused large, model-dependent losses. These fields are load-bearing because attribution requires both the decision and its relationship to the execution around it. A tool outcome without that context can be evidence of a symptom.

For practitioners, the instrumentation requirement is explicit decision-to-provenance links at each step alongside terminal status and tool call logs. A trace must preserve enough evidence to distinguish competing explanations. Otherwise, an attribution process is being asked to recover distinctions that the recorded history does not contain.

Ambiguity also needs to remain a valid result. The same benchmark found that two of five frontier models answered every ambiguous case rather than deferring when multiple origins were equally consistent with the evidence. Model confidence cannot supply the missing distinction. A triage process needs to be able to abstain when telemetry leaves more than one possible origin, so that incomplete evidence does not become a diagnosis that looks precise.

Localizing the earliest decisive failure step before attributing anything

The first obligation in triage is to find the earliest erroneous step causally linked to the final failure. Layer assignment comes after that decision. The target is neither the first visible deviation nor the terminal error, and it is not automatically the earliest local mistake. When localization is wrong, every fix candidate that follows targets the system in the wrong place.

Failed trajectories contain errors with different consequences. Some are repaired and leave no terminal effect, while others remain inconsequential or merely expose a fault introduced upstream. TRAJDEBUG, from Tsinghua and Tencent Hunyuan, separates error trigger detection, error state classification, and causal attribution. By tracking resolution status and terminal impact, it can tell which local errors matter more to the diagnosis.

An agent can edit a forbidden file, compensate with a patch, and produce passing tests while still violating the task constraint. The critical step remains the edit because later success on a different check does not resolve the original violation.

LongRCA Bench contains 1,140 human-annotated failed trajectories whose reference root-cause steps precede completion by a median of 48 recorded steps. More than a quarter contain over 100 subsequent steps. These distances indicate the scale of the search. On the full benchmark, even structured methods are still far from finding most reference roots exactly.

Root-Cause Trajectory Attribution, or RCTA, also distinguishes errors inherited from upstream instructions from errors introduced during execution. That distinction changes the repair target. An inherited error directs attention to the instruction, while a newly introduced error directs attention to the execution step. Localization must show both where an error entered the trajectory and whether it still determined the outcome at the end.

Why one-shot trace reading misses what iterative search finds

A one-shot diagnosis can settle on the first plausible explanation before examining the evidence needed to disprove it. Long traces make this especially difficult to detect because supporting evidence may appear together while contradictory evidence sits elsewhere. A coherent explanation can still be wrong. The structural weakness is premature closure.

Continual Search, from Scale AI, addresses this by requiring the judge to challenge its standing answer at each successive turn. The judge must search beyond the evidence that supported the initial diagnosis and actively seek contradictions. This is more demanding than asking for another opinion on the same material.

Asking a judge to think again within the same session can leave it working from the same view of the evidence. A careful first reading is useful, but it is not a reliable stopping rule for a long execution.

MegaRCA-Mix provides 50 human-annotated failure trials spanning long-horizon, execution-heavy tasks, where iterative search produces meaningful attribution gains over single-pass methods. Within the same model family, lower-tier models using Continual Search can surpass higher-tier models using one-shot judging. Model scale is therefore not the only lever. For teams considering a stronger judge as the primary remedy, the finding points first to how the investigation searches and tests its explanation.

Attributing the localized step to the correct harness layer

Once the decisive step is localized, attribution must name the layer at fault, going beyond a broad label of agent failure for the execution. Fault assignment determines the next work queue. Model failures call for work on post-training or model selection, while environment or grader issues call for repairs to the benchmark or infrastructure. Harness failures direct attention to prompt, tool, workflow, memory, or routing design.

Within the harness, outcomes that look alike can call for changes to different artifacts. Tool schema drift occurs when a harness built against a particular input/output schema encounters an unannounced schema change, potentially producing silent data corruption. The decisive step is the tool call with schema-invalid arguments. That finding calls for a closer look at the schema mismatch, beyond a general instruction to use tools more carefully.

Prompt ambiguity requires a different inquiry. An agent may interpret an underspecified instruction in a locally reasonable way while violating a task constraint stated much earlier. Here the decisive step is the planning decision preceding tool use. By contrast, workflow loop and goal drift can originate when an agent accepts its own output as a completion signal instead of applying an objective verification condition.

Attribution should identify the earliest layer where intervention would have broken the causal chain. A prompt ambiguity that leads to a bad tool call and then a workflow loop does not by itself call for three changes at once. The downstream effects help explain the chain, but they do not independently establish where repair should begin.

TRAJDEBUG’s TRAJERRBENCH covers realistic tool-use and coding scenarios drawn from τ²-Bench and SWE-Bench Pro. Its application studies show that typed diagnoses provide actionable feedback that improves downstream agent success. Specificity has practical value beyond classification. It connects the causal finding to the kind of artifact an engineer can inspect and change.

Translating a typed attribution into a reviewable fix candidate

A layer label is not yet a proposed repair. “The system prompt needs to be clearer about constraint handling” identifies a broad location but leaves the instruction, replacement wording, and verification claim unspecified. An engineer cannot audit the intended behavioral change from that statement alone. The diagnosis must become a bounded, inspectable proposal before it qualifies as a fix candidate.

A reviewable candidate records the decisive step and responsible layer, then names the precise artifact to change. Depending on the attribution, that artifact may be a prompt instruction, a tool schema field, a termination condition expression, or a memory retrieval filter. The candidate also states a testable claim about the change. For the forbidden-file failure, that claim can be that the instruction should prevent the agent from editing files listed in the constraint block.

The proposal must also identify the historical traces against which it will be validated before shipping. Reviewers can then check whether the proposed artifact fits the diagnosed layer and whether the claim deals with the decisive step. The trace set establishes where that claim will face a test.

AgentDebugX organizes its closed loop around Detect, Attribute, Recover, and Rerun. Recovery converts the diagnosis into a rerun-able fix, while Rerun tests it. Attribution alone does not complete the loop. A correct explanation still needs a specific repair whose effect can be checked.

Scope bounding preserves that examination. A candidate that changes multiple prompt sections, two tool schemas, and a routing rule simultaneously obscures which intervention produced the result. When attribution identifies a single layer, the candidate should change only that layer’s artifact. Secondary findings need separate candidates so that each proposed repair retains an auditable relationship between evidence, change, and outcome.

Validating fix candidates against historical traces before shipping

A written fix remains a hypothesis until it is tested against the production traces that exposed the pattern. Resolving the labeled case is only one requirement. The change must also be assessed against traces where the agent previously succeeded, because a targeted repair can degrade adjacent behavior.

Ordinary logging is not enough to reproduce a non-deterministic LLM failure. A tool may return a different value, the model may sample differently, or the environment may have changed. Span-based trace replay with recorded tool outputs offers a stable, low-cost way to test a repair. It lets teams compare against the recorded execution, so they can do more than try again under changed conditions.

Forward validation asks whether the proposed repair causes the decisive step to behave correctly on the trace that exposed the failure. Regression validation asks whether previously successful behavior degrades. AgentDebugX’s Rerun step executes the repaired agent on the same task and environment, using the original trace as a baseline. Its repair criterion is a qualitatively different outcome at the step previously identified as decisive.

Replay is a necessary gate whose evidence is limited to historical cases. Continued production monitoring catches failures beyond the historical cases and shows when a repair fixed an example but left its pattern in place. A new cluster sharing the same failure type at the same layer is evidence that further triage is needed.

The toolkit’s opt-in Error Hub takes this process further with scrubbed failure-diagnosis-repair bundles that teams can share as debugging memory. A repair validated on one team’s traces can become evidence for a related repair elsewhere. Within each team, replay results and new production traces go back into the trace corpus. The improvement loop closes when teams identify the next pattern to investigate in that evidence, rather than when an engineer finishes writing the fix.