Change Attribution Across Prompt, Tool, and Workflow Layers

Change attribution across prompt, tool, and workflow layers shows what engineers should change when an agent run fails. Its central claim is that locating an error is not enough. A useful diagnosis must establish where the decisive error originated, assign responsibility accordingly, preserve the evidence for that assignment, and validate the proposed repair through replay.

Why knowing a run failed is not enough to fix it

The run broke, so the prompt needs a fix. That reflex selects an intervention before establishing a cause. An incorrect final answer may instead originate in stale memory, an invalid handoff, or an early planning constraint that later actions never recover. AgentDebugX (2026) identifies the distance between an error’s origin and its visible symptom as a central difficulty in agent debugging. The final step may expose the problem without introducing it.

TRAJDEBUG (2026) distinguishes local errors that are repaired, errors that remain harmless, and errors that contribute to terminal failure. The first visible mistake is therefore not automatically the repair target. Attribution must establish which error remained consequential through the rest of the execution.

What change attribution actually means — assigning fault to a layer, not just a step

Root-cause attribution transforms an outcome signal into an actionable diagnosis by finding the first decisive divergence from correct execution and assigning responsibility to a component. Layer assignment determines what can usefully change. Scale AI’s Continual Search paper (2026) distinguishes four fault domains, which are the model, harness, environment, and grader. Model failures can inform post-training objectives, harness failures guide harness design, and environment or grader faults require benchmark repair.

Within the harness, the distinction becomes more specific. The harness is the code that coordinates prompts, tools, subagents, control flow, memory, and workflow logic. A diagnosis that names a failed tool call but does not distinguish a bad instruction from a faulty implementation remains incomplete. TRAJDEBUG separates error trigger detection, error state classification, and critical attribution to establish whether a candidate error actually drove the failure. Change attribution adds the engineering destination to that causal account, identifying which layer owns the repair.

Why the harness is the primary failure surface, not the model

Most agent failures that attribution can act on belong to the harness. Most failures here are tool-call failures, context truncation, and runaway loops, but standard monitoring lacks the agent-aware instrumentation needed to show them. These components are directly changeable without model retraining. The harness therefore provides the main intervention domain for the prompt, tool, and workflow distinctions that follow.

In TRAJDEBUG’s example, all tests pass, yet the task fails because a governing constraint was ignored at step two. The constraint existed in the task instruction but was not salient enough in the system prompt to govern that decision. Passing tests did not establish compliance with the task.

That case sits within a broader set of harness failure sites. The seven components are the system prompt, tool description, tool implementation, middleware, skill, sub-agent configuration, and long-term memory. Each can introduce a different kind of error. Treating the whole run as a model failure obscures the component that can be changed and the evidence needed to justify changing it.

Prompt failures — when the instruction layer introduces the decisive error

A prompt failure occurs when the instruction introduces the decisive error while the model behaves correctly given what it received. A required constraint may be absent, ambiguous, or buried where it does not govern the relevant decision. In that case, attribution must connect the instruction available at the decisive step to the choice that violated the task, rather than infer a prompt defect from the final answer alone.

Long trajectories complicate that connection because governing instructions may become effectively invisible before the crucial action. In LongRCA Bench (2026), reference root-cause steps fall a median of 48 recorded steps before the run ends. The symptom can therefore be far from the relevant instruction. Another prompt-layer signature is a correct-looking tool call whose arguments are wrong because the prompt never fully specified the required format or parameter values. Schema validation cannot resolve that ambiguity. Once the evidence assigns the error to the instruction, the repair target is a reviewable system-prompt or task-instruction change.

Tool failures — schema drift, malformed arguments, and silent wrong-tool calls

Tool failures need not produce tool errors. A tool can return HTTP 200 with hallucinated content, accept malformed arguments, or run on the wrong input without raising an exception. Successful execution is not proof of semantic correctness. Attribution must follow the causal path from the instruction that generated the call through its arguments to its output.

Schema drift breaks that path when an interface changes but its prompt description does not. A renamed parameter, deprecated field, or narrowed type can leave a call syntactically acceptable but semantically wrong. Malformed argument generation creates a related signature when valid JSON contains logically incorrect values, such as an over-restrictive query. Wrong-tool selection is different again, because the chosen tool may be plausible without serving the intended task.

These signatures do not all imply the same repair. A misleading tool description belongs to the prompt layer, while a faulty implementation belongs to tool-level harness engineering. TelemetrySuffBench (2026) finds that removing provenance causes large, model-dependent losses in attribution accuracy. The upstream decision-to-call link helps show where to make the change, so a visible tool action alone does not put the blame on the tool implementation.

Workflow failures — loops, cascade errors, and inherited instructions

LongRCA Bench illustrates workflow failure through repeated checks that retain a nightly lodging price as a two-night total. A separate arithmetic error in the food/activity budget allocation is corrected, but the lodging error persists. One repair leaves another error intact. The diagnostic task is to recover dispersed evidence and separate the point of introduction from later propagation or missed opportunities for correction.

That distinction becomes essential across agent handoffs. A sub-agent may execute correctly while following an incorrect instruction from an upstream orchestrator, placing responsibility in the orchestrator’s planning logic. LongRCA Bench proposes Root-Cause Trajectory Attribution, or RCTA, which finds the original candidate records and checks them against the upstream instructions that are available. It reaches 51.1% responsible-role accuracy by addressing this separation between inherited and newly introduced errors.

Other workflow failures require a related distinction between detection and cause. AgentDebugX treats loop detection through deterministic rule packs as a separate stage from root-cause attribution. Repeated retries can show that a circuit-breaking condition is missing, while cascade analysis traces an early error that stayed in place even when later actions were valid. Detect Before You Attribute (Cascade Failure Attribution for Multi-Agent Systems) (2026) shows why isolated-agent attribution can mistake later symptoms for the source of an error when it has no access to upstream instructions.

Why telemetry format determines whether attribution is even possible

Telemetry determines what an attribution method can know. Standard distributed tracing spans can record latency, status codes, and request/response payloads without keeping the evidence needed to tell who was responsible. The gap is structural rather than a weakness in a particular diagnostic model.

TelemetrySuffBench demonstrates this separation across five frontier language models. When decision content is removed, origin-step accuracy falls to zero for every model tested. What the model was told and what it decided is therefore irreducible evidence, while provenance connects those decisions to the tool calls or sub-agent invocations they produced.

The instrumentation implication precedes any choice of diagnostic method. Hierarchical, replayable spans covering LLM calls, tool invocations, retrievals, and internal reasoning steps provide a baseline, but their structure alone is insufficient. That record must keep decision content and provenance available. Otherwise, a more capable judge receives a better-organized account of what ran without the links needed to explain who introduced the error.

How attribution methods navigate long traces without collapsing to a single-pass guess

Even sufficient telemetry can be difficult to search. Long-horizon execution logs can reach millions of tokens, with sparse evidence distributed across distant records. Single-pass LLM judgment often settles on a plausible explanation before examining the evidence that could overturn it. Continual Search addresses that failure through iterative diagnosis, requiring the judge to challenge its standing answer and search for unresolved evidence at each successive turn.

Inexpensive heuristics and whole-trace reading suit short or structurally simple failures, while binary search narrows an unknown failure region by locating the divergence between correct and incorrect execution. Per-step inspection and budgeted ensembles provide finer examination when evidence is ambiguous or contradictory. AgentDebugX’s DeepDebug uses multi-round diagnosis when it needs to understand the whole trajectory, follow its structure, and cross-examine the evidence, then writes a report that can be audited, with evidence and a suggested fix.

Search must also recover the right relationships. RCTA retrieves upstream instructions before judging candidate records, reaching 24.1% exact root-step accuracy on the benchmark. That result leaves substantial uncertainty despite improved localization. TRAJDEBUG contributes a complementary error-lifecycle analysis, distinguishing triggers from repaired, harmless, or propagated effects. Upstream retrieval identifies responsibility, while lifecycle tracing establishes whether the error was decisive.

Replay validation — confirming that a proposed fix addresses the attributed layer

Attribution identifies the layer to change. Replay tests whether changing it resolves the failure. These are sequential engineering steps. A fresh run that passes does not establish that a repair worked because inference is non-deterministic, tools read changing state, and the trajectory may differ from the failed execution. Validation must therefore isolate the proposed change from those other sources of variation.

Chronicle (Chawla & Koul, 2026) records agent runs at their non-deterministic boundaries as immutable envelopes and replays execution from that record. Its cut-point mechanism lets a developer choose which boundaries are served from recorded values and which run live with new code. A prompt edit, tool-schema update, or workflow-routing change can then be tested while the other recorded boundaries stay fixed.

Full replay can make zero model calls and stay bit-stable when run again. Recorded failures can then be used as deterministic regression tests in continuous integration, replacing manual spot-checks. Cut-point replay serves the distinct purpose of measuring a live change against the recorded execution. That separation connects the causal diagnosis to evidence about the proposed repair before shipment.

Making attribution a repeatable engineering discipline, not a post-incident ritual

Continuous attribution requires production trace sampling that preserves decision content and provenance, including for failures that do not trigger obvious alerts. Silent wrong-tool calls and inherited errors also need diagnosis, even when they lead to no incident. For teams adopting this discipline, the starting practice is structured sampling rather than relying solely on latency and error-rate metrics. Trace quality signals provide the evidence needed to assign failures to layers even when execution appears operationally successful.

Drift detection then tracks whether failure distributions shift at the prompt and use-case level. Those shifts can reveal changes in a given layer before they turn into incidents. Replay-based regression on a golden trace set helps teams use that monitoring to decide what to release by testing prompt edits, tool-schema updates, and workflow-routing changes against past failures. Attribution findings then set limits on future harness changes as well as explain past incidents.

Continual Search places this feedback loop within the move toward agentic self-improvement. Systems that learn from failed trajectories need a way to direct those lessons to the responsible component. Otherwise, efforts to improve the system can change a sub-agent that works correctly while leaving its faulty upstream instruction in place.