Harness Layer Attribution Methods for Production Agent Failures

AI agent failure attribution by harness layer traces a failed run to the earliest unrecovered interaction, then identifies the component responsible for repair. “Harness Layer Attribution Methods for Production Agent Failures” explains how to distinguish model faults from defects in prompts, tools, workflows, and memory. The governing principle is that the visible failure is a starting point for investigation.

Why production agent failures resist straightforward diagnosis

An agent can ignore an instruction because context compaction removed it several steps earlier. The final response exposes the violation, but not the removal. Similarly, a memory read can supply stale context, after which a tool call executes successfully and returns data that corrupts later decisions. Neither case requires an exception at the originating step. An engineer inspecting only the final answer sees the consequence without the interaction that produced it.

TRAJDEBUG, from Tsinghua/Tencent Hunyuan in 2026, formalizes this diagnostic difficulty through the different lifecycles of local errors. Some errors are repaired, others remain harmless, and others propagate into the final failure. Their order alone does not establish causality. The step that caused the failure may come between the first error and the terminal event. Fixing a later symptom can leave the original defect in place.

Why the model-vs-harness distinction is the first cut every diagnosis must make

Scale AI’s interaction-centric taxonomy (2026) defines the harness as the component managing the model’s context, memory, and tool access, distinct from the model policy. It organizes 41 failure modes by interaction edge and fault side, identifying where repair belongs. That distinction determines the intervention. If an instruction was lost during compaction, context management needs to be fixed. If the instruction remained available and the model ignored it, the same visible outcome requires model-level intervention.

The taxonomy’s practical criterion is whether a more capable model could have avoided or recovered from the failure under the same conditions. A more capable model cannot fix a harness-side root cause. CausalLoss-Fin (2026) shows the risk of ruling out that cause in a narrow setting with deterministic policies in finance. When the environment is a causal source but the attribution model contains no environment variable, it misfiles all infrastructure-caused episodes and recovers 0.0% of the available loss by repairing what it names. This result offers a structural warning while leaving prevalence in production agents generally unestimated.

The seven harness layers where production failures actually originate

The seven-layer map separates system prompt, tool description, tool implementation, middleware, skill, sub-agent configuration, and long-term memory. Each names a different repair location. System-prompt ambiguity leaves the task insufficiently specified, while an underspecified tool description can permit plausible-looking but invalid arguments. Tool implementation requires separate inspection when a structurally successful call returns corrupted data. A tool-call failure alone does not distinguish these layers.

Middleware governs context compaction, token budgeting, and summarization, so its characteristic signature is an instruction that existed at task start but disappeared before a model call. Skills encode task patterns in markdown files or structured prompts and can become stale as task shapes or tool interfaces change. Sub-agent configuration governs routing and permissions, where failure can appear as waiting without state progression. The taxonomy’s property-listings example shows why delivery must also be inspected. The agent completed phase one and waited through four 600-second notification cycles, while a human annotator attributed the failure to stale state delivery and the automated judge blamed the model.

Long-term memory introduces a quieter failure signature through memory-induced tool drift. The query executes and returns data, but that data reflects a prior schema or stale context. Success at the interface can conceal semantic failure. This separates memory faults from obvious validation errors and explains why an exception-only investigation cannot cover the harness.

How failure mode distributions in production reveal which layers break most often

Harness-Bench places contract/format failures as the largest class among failed runs, followed by tool/recovery, evidence/grounding, artifact commitment, and state/continuation failures. AgentFixer’s deployment on IBM’s CUGA system similarly found parsing-related failures to be a substantial, deterministically addressable category. These include malformed outputs, schema violations, and format contract breaks. Model changes are not required to repair that category. Together, these findings support checking structural contracts before beginning expensive semantic attribution.

The Kaggle GameArena chess competition illustrates the same distinction through action validity. A large share of Gemini-2.5-Flash losses were attributed to illegal moves, a failure mode removable through harness-level action validation. Better reasoning is not the only repair path. For an attribution workflow, contract and format checks offer the highest-frequency, lowest-cost starting point described by this evidence. That priority narrows the search without predetermining the cause of any individual failure.

The attribution rule — tracing a failure back to the earliest unrecovered step

The engineer begins at the terminal failure and works backward through the interactions that supplied its inputs and state. Candidate errors must then be tracked forward and their effects classified as repaired, harmless, or still propagating. Only unresolved effects support the final attribution. TRAJDEBUG breaks this process into error trigger detection, error state classification, and critical attribution. The assigned label belongs to the earliest unrecovered cause.

TRAJDEBUG’s TRAJERRBENCH contains 486 manually annotated failed trajectories from τ²-Bench and SWE-Bench Pro. Its published benchmark reports only 14% accuracy for step-level language model judgment, illustrating the difficulty of separating coexisting errors in long runs. More context is not automatically the remedy. DUOTRACE, by Jiayi Zhang et al. (2026), shows that hierarchical multi-agent behavior must be assessed against each node’s invocation context across the execution tree. A linear reading can miss the dependency that explains the failure.

DUOTRACE uses a detect-before-attribute approach to reduce that ambiguity. It first identifies anomalous execution nodes through semantic and structural signals, then passes focused candidates to attribution. This makes agent-level and step-level attribution better than it is with baseline LLM-as-judge approaches. Supplying an entire long trajectory can add redundant information that obscures the initiating error.

How do you pinpoint the responsible layer in a production trace?

A span hierarchy exposes the interaction boundaries hidden by a summary such as “flight_search failed”. The engineer can inspect the LLM’s context, the generated arguments, the tool’s response, and the enclosing agent’s progression separately. Each boundary offers a different test. A validation error calls for checking how the arguments were built against the tool description, while a call that succeeds but returns corrupted data calls for a check for schema-drift. Either finding needs more evidence before the model can be blamed.

Consider an illustrative trace fragment in which a retriever span returns non-empty but stale context, an LLM span receives that context, and a tool span completes without an exception. The agent’s final output is wrong. Reading backward establishes which later steps depended on the stale result. If its effects persisted unrecovered, the retrieval interaction becomes the attribution target, with memory or retrieval configuration investigated as the repair location.

A different trace might show an instruction absent from the LLM context after compaction, or repeated retries with no task-state progression. Those signatures direct investigation toward middleware and orchestration respectively. The taxonomy supplies the repair-assignment language for both. The engineer identifies the interacting component pair, determines the fault side, and assigns ownership on that basis.

Detection follows the same discipline of narrowing the search. Lightweight rules check malformed calls and schema violations first, statistical monitoring checks latency and token-usage drift next, and semantic analysis handles unresolved reasoning failures. AgentFixer used 15 validation tools on CUGA and achieved 64–88% issue detection rates, with deterministic parsing checks covering the largest recoverable category. Such tools provide evidence for attribution under the causal rule.

How HarnessFix and HTIR close the loop from trace evidence to attributed repair

HarnessFix (2026) turns trajectory diagnosis into changes to the harness. Its HTIR component organizes fragmented evidence into step-level relations suitable for attribution, identifying the responsible runtime step and explaining the root cause. The output extends beyond a failure label. Recurring diagnoses consolidate into accumulated flaw records, while the attributed layer directs the repair toward a concrete harness edit.

The repair operators differ by the diagnosed boundary. Tool-interface failures can be fixed through schema narrowing, argument validation, tool-candidate ranking, or repairs to the documentation. Lifecycle failures instead call for loop guards, explicit task-state models, bounded retries and timeouts, verification-gated finalization, or workflow-state checkpoints. Context and memory faults route to compaction-policy adjustments, memory-eviction changes, or retrieval re-ranking. Skill faults require refinement where existing coverage has gaps, or new skills where the task pattern falls outside that coverage.

An implementable attribution therefore connects the failure step, its explanation, and the proposed operator. The flaw record preserves the recurring diagnosis. When several traces show the same failure mode, the accumulated record supports a repair that covers the pattern as a whole.

Why replay validation against real traces is the gate before any harness change ships

A proposed repair remains unverified until replay tests it against historical traces, including those that exposed the original defect. Attribution determines which layer changed and which traces specifically stress it. The ReplayToolClient turns tool invocations into pure functions that use recorded traces, so live time, randomness, and tool-version drift no longer affect those interactions. That control distinguishes replay from rerunning an agent against a changing live environment.

Under the binary equivalence principle, unchanged agent code should not diverge during replay. A divergence signals a correctness issue such as a control-flow fork, prompt drift, or planning-execution misalignment. Changed code requires a different comparison. The proposed repair should fix the failure on affected traces without causing new divergences on ones that were clean before. The intended layer change supplies the basis for interpreting those differences.

Validation must check how the agent ran as well as grade its final answer. A correct-looking response can conceal a bad tool call, missed constraint, irrelevant retrieval result, hidden retry loop, or unsafe recovery path. Output-only grading misses these defects. Scheduled replay of a golden trace set, daily or on every deploy, makes performance drift visible over time. A change that degrades golden-set performance is a regression even if it repairs its motivating example.

Replay comes at a cost, and golden sets can grow stale. The corresponding discipline is to refresh them continuously with production traces showing interesting or boundary-case behavior. Attribution helps choose which traces to add because it shows which layer and interaction each trace tests.

Treating harness attribution as an ongoing engineering discipline, not a one-time debugging exercise

Production signals feed eval refinement, program optimization, sandbox replay, regression tests, controlled deployment, and new production signals. Self-Harness, described in Self-Harness: Harnesses That Improve Themselves by Zhang et al. (2026), formalizes recurring weakness mining, harness editing, and validation. Its weakness mining groups failures into patterns grounded in what the verifier finds. Skills can then accumulate task knowledge through additions and refinements driven by execution feedback.

That process requires reviewable changes and inspectable evidence. Engineers need to audit what changed, why it changed, and what validated it. Local improvement is not sufficient evidence. A prompt rewrite can repair one failure while creating another, and repeated iteration can overfit a held-out set. Validation must cover the production-trace distribution.

For teams establishing the discipline, the practical starting point is contract/format failures and deterministic parsing checks. A golden trace set from the first month of production traffic gives teams a base for validation, and they can make attribution more rigorous without instrumenting every layer at once. The process remains an engineering task throughout.