Closing the Loop From Production Failure to Validated Harness Fix

Closing the loop from production failure to validated harness fix requires four engineering stages, namely surfacing failures in structured traces, attributing them to the responsible harness layer, generating reviewable changes, and validating those changes against historical production traces. This workflow turns agent observability into a compounding improvement loop when recurring failures drive durable prompt and logic improvements that address their shared causes.

Why production agent failures are hard to attribute without the right trace structure

An agent can fail at a tool call because of an error introduced much earlier in its plan. Between the source and the visible failure, later steps may look valid while carrying the original mistake forward. Long trajectories interleave planning, tool calls, memory retrieval, and environment feedback, so the last unsuccessful action does not explain the whole run. Diagnosis requires preserving enough evidence to connect that outcome to the decision that caused it.

TRAJDEBUG, from Tsinghua / Tencent Hunyuan in 2026, formalizes how to detect the critical error that caused a trajectory to fail. A failed trajectory can contain several local errors, including errors that are repaired, errors that remain harmless, and errors that contribute to the final failure. The error that causes the failure may thus come after the first local error and before the one closest to the terminal failure. Judging it requires reconstructing context beyond the final log lines from distant task instructions, observations, and earlier decisions.

TelemetrySuffBench, from ECNU in 2026, shows how sharply detection and localization can diverge. When decision content is removed, origin-step accuracy falls to zero for every model tested, and removing provenance links causes further large drops. Systems that improve agents from production traces, including Moda, first need decision content and provenance links because automation cannot piece together a causal chain that the telemetry left out.

Stage 1 — Surfacing failures: what a trace must contain to make diagnosis possible

Actionable instrumentation records execution at the level where distinct decisions and interactions occur. Spans must cover model decisions, tool invocations, memory retrievals, inter-agent handoffs, and environment observations. This granularity separates what the agent chose from what a tool returned or the environment revealed. Without that separation, a trace may record the outcome while obscuring the sequence needed to explain it. Instrumentation must support both failure detection and failure-origin localization, which require different evidence.

Within those spans, decision content records what the agent decided and why, while provenance identifies the upstream instruction or retrieval from which the decision derived. Together, they let diagnosis compare a choice with the evidence available when it was made. A tool invocation alone cannot provide that comparison if the instruction that shaped it is missing. Moda validates these load-bearing fields before automated diagnosis, addressing the distinction between instrumentation built to report whether a run succeeded and instrumentation built to explain which layer failed.

Session continuity matters just as much because some failures are visible only in comparisons across turns. An agent that chooses the correct tool on turn one but an incorrect tool by turn four presents a pattern that isolated spans conceal. The trace must keep that link while allowing diagnosis without a full-history pass for every question. TRAJDEBUG addresses this scale problem through multi-granularity history compression, preserving history at different levels of detail for evidence retrieval. A well-structured trace supplies the raw material that attribution requires.

Stage 2 — Attribution: finding which harness layer caused the failure, not just where it appeared

Attribution must identify both the decisive step and the responsible layer, namely prompt, tool, workflow, memory, or model. A wrong tool call might reflect ambiguous instructions, a mismatched tool descriptor, faulty routing, or missing retrieved context. Treating all of these as prompt problems directs changes toward the visible behavior and leaves its cause unaddressed. Layer specificity is what turns diagnosis into an engineering decision.

Scale AI’s 2026 Continual Search framework treats root-cause attribution as an iterative search. Execution logs can reach millions of tokens, while the relevant evidence remains sparse and distant from the visible failure. A judge that settles on an early plausible explanation can leave decisive evidence unexamined. Continual Search instead challenges the standing diagnosis through iterative evidence search, substantially improving attribution accuracy. The important distinction is between finding a plausible explanation and continuing to test whether it accounts for the trajectory.

LongRCA Bench in 2026 shows why that distinction does not make attribution a solved problem. Exact root-step accuracy is still well below 50% for all baselines tested, even with full traces and strong models. Its RCTA method retrieves the original candidate evidence and checks it against upstream instructions to tell which errors were inherited and which were newly introduced. That improves accuracy, but RCTA still recovers fewer than one quarter of reference roots exactly. Even with more traces, automated diagnoses still need evidence to back up their judgments.

TRAJDEBUG provides a useful decomposition by separating error trigger detection, error state classification, and critical attribution. These establish whether a step is wrong, whether its error propagates or is repaired or harmless, and whether it causes the final failure. The method should also fit the category, since loops and malformed outputs have rule-detectable fingerprints, while prompt ambiguity requires instruction comparison and tool errors require checking arguments against the current schema. With a structured trace, Moda’s model uses the decision-to-provenance chain to isolate failures in prompt instructions, tool selection, workflow logic, memory retrieval, or environment interaction. This makes attribution a repeatable diagnostic step that can be tested and refined across production patterns.

Stage 3 — Generating a reviewable fix: what a harness change actually looks like at each layer

Once attribution identifies the responsible layer, the next output must translate that category label into an inspectable change. Harness controls fall into two functional families, which are guides and sensors. Guides are feedforward controls, including system prompts, constraint documents, and tool descriptors that shape what the agent attempts. Sensors are feedback controls, including evals, output parsers, and validation loops that catch problems in what it produces. This distinction clarifies whether a change should improve direction before an action or detection after it.

At the prompt layer, when guidance for empty inputs, invalid formats, or unclear situations is missing, the prompt text needs to spell out how to handle them. The resulting diff lets a developer read the instruction line by line and judge whether it clears up the ambiguity found in diagnosis. At the tool layer, schema drift calls for a different intervention because clearer task instructions do not reconcile an outdated argument signature with the current schema. Updating the tool descriptor or invocation wrapper produces an inspectable schema change.

Workflow and memory failures require equally specific edits. A wrong tool sequence or repeated loop calls for a routing-policy change or an explicit termination condition. When retrieved context is wrong or missing, the fix calls for changes to retrieval configuration, chunk boundaries, or relevance scoring. These are reviewable as distinct workflow and retrieval policy edits. The attribution taxonomy can also identify the model as responsible, but that label alone does not establish a justified harness change.

TRAJDEBUG’s application studies show that diagnoses can produce actionable feedback that improves downstream agent success. The engineering requirement is that the feedback explain both what should change and why that change addresses the attributed failure. Developers must be able to read the proposal and reject it when its reasoning does not hold. In Moda’s workflow, proposed improvements are presented as inspectable changes for developer review before application. Reviewability preserves the judgment needed for safe iteration, but it does not establish that the change works.

Stage 4 — Validating a fix against real traces before shipping

AgentDebugX, from UIUC et al. at EMNLP Demo 2026, makes validation concrete by rerunning the agent from the point of failure with the proposed fix. The new trajectory is compared side-by-side with the original failed run. If the rerun succeeds, the case is documented. If it fails again, that case’s loop remains open, and the primary sources do not describe a re-entry mechanism. The rerun establishes a necessary distinction between proposing a correction and showing a successful execution with it.

Replaying from the failure point is more efficient than fully re-executing every candidate change, but success on one failing case is not enough for release. A change can fix the root error found in attribution while changing behavior on inputs that passed before. Validation must therefore test both the original failure class and similar cases that passed in historical production traces. Synthetic tests alone cannot substitute for this evidence because they may not cover the input distribution that actually broke the agent.

Reliable agents need evaluation layers that work together, with unit evals on individual steps, regression suites for subjective output quality, and continuous production trace sampling. Before shipping, the proposed change must at least pass a sample of past cases that reflects the failures it affects and similar cases that succeeded. Moda validates proposed improvements against historical traces before recommending what to ship.

Scheduled replay of a golden trace set extends this gate into an early-warning mechanism for drift. Testing that set against the current production configuration can surface regressions from prompt or schema changes before they reach users. The historical cases become a continuing check on behavior, rather than records consulted only during an incident. At this point, the four stages form a complete loop, with structured evidence supporting attribution, attribution directing a reviewable intervention, and historical replay testing its consequences. Validation closes the loop only when the proposed improvement survives those tests.

How teams continuously improve agents by turning recurring failures into durable harness changes

A single validated change resolves a bounded problem, but repeated failures can reveal that the boundary was drawn too narrowly. When the same category appears across multiple traces, the signal may be a systematic gap in a guide or sensor. Prompt ambiguity and missing edge-case handling point toward guidance, while weak eval coverage and schema validation point toward detection. Patching one case after another leaves that gap in the guide or sensor in place. The compounding value comes from recognizing the shared cause and making a durable harness improvement.

Continuous production trace sampling supplies the evidence for that recognition by scoring a slice of real traffic independently of failure reports. It identifies distribution shift, edge cases missed by static eval suites, and slow drift in production traces as models, tools, and user behavior evolve. The workflow remains the same, but its unit of attention expands from one failed trajectory to a recurring failure pattern.

Harness Continual Learning in 2026 formalizes the basis for this accumulation. Prompts, memories, tool and skill specifications, and routing policies can carry over and change across interactions even while the foundation model stays frozen. Agent adaptation is therefore not confined to model weights because harness state can retain experience and shape future behavior. By contrast, relying on retraining for every harness edit repeatedly incurs a cost that is unsustainable at live production update frequencies. Harness iteration provides a shorter improvement cycle without requiring a weight update.

That shorter cycle still requires the rigor of code review and regression testing. Signal detection, pattern recognition, attribution, and validation are engineering work that requires engineering judgment. Moda implements this discipline through continuous trace ingestion, recurring-pattern detection, layer-specific attribution, and improvements that can be validated before shipping. Production observability compounds only when what it reveals becomes a reviewed, validated change to how the agent runs.