Root Cause Attribution as a Ship Prerequisite

An agent that ships without a confirmed root cause is a fix applied on faith. If the team doesn't know which layer of the system produced the failure, the change they push is a guess dressed up as an engineering decision, and it may leave the actual fault sitting untouched while introducing a new one somewhere else. Root cause attribution has to happen before the deploy button gets pressed, not after, because the deploy decision is the point where the guess becomes production behavior.

What "root cause" means when an agent fails, the layer taxonomy

The reflex when an agent misbehaves is to blame the model. Swap it for a newer version, upgrade the checkpoint, fine-tune on the failure case. The base model is the layer least likely to be the actual source of the fault.

What sits around the model, the harness, is where most of the damage happens. Lilian Weng's July 2026 post on harness engineering describes it as the system that orchestrates execution: how context gets presented, how tools get called, how memory gets stored and retrieved, how outputs get evaluated. That's four or five separate subsystems bundled into one word, and each one fails differently.

Take the prompt. Ambiguous instructions, role confusion, phrasing that happens to match another tool's signature more closely than the one it's meant to invoke, all of these produce failures that look like model error but aren't. Take the tool schema, the actual interface the model acts on. A schema mismatch throws a runtime error, which at least announces itself. A description mismatch is worse: the model drifts into calling the wrong tool, or skips a tool it should have called, and nothing in the system raises a flag. Take workflow, the sequencing and handoff logic between agents. A four-agent loop from November 2025 burned $47,000 over eleven days because a planner kept cycling checklist artifacts between two agents with no cycle detector in place to catch it. Take memory. Compaction mechanisms can silently drop rules that weren't anchored in the system prompt, and compaction can silently drop rules that weren't anchored in the system prompt, and nothing downstream knows they're gone. Environment and infrastructure round it out: rate limits, context overflows, cascading timeouts. Real, but a separate category from the harness itself.

The MAST failure taxonomy, applied at NeurIPS 2025 to annotate more than 1,600 execution traces, sorts failure modes into three buckets. Specification problems, meaning role ambiguity and missing constraints, account for 41.77%. Coordination failures, meaning communication breakdowns and conflicting objectives between agents, account for 36.94%. Verification gaps, covering missing validation and absent quality checks, make up the remaining 21.30%. Specification and coordination alone cover 79% of production breakdowns, and both sit squarely in the harness, not the model.

The Scale AI paper on continual search (arXiv 2609.13463) formalizes the same idea from a different angle: faults get assigned to a specific layer, and that assignment is what determines which intervention has any chance of working. Get the layer wrong, and the fix goes to the wrong address. Which layer failed is the entire ballgame. It's the entire ballgame.

How shipping without attribution creates a compounding liability

The counterargument sounds reasonable on its face: the team saw the failure, made a change, reran it, and it passed. Ship it. This is not an acceptable engineering practice, and the reasoning holds up under almost no scrutiny.

If the root cause was never confirmed, the fix is a hypothesis bolted onto a symptom. The actual fault, whatever it was, is still sitting in the system. Worse, the fix might suppress the visible symptom while leaving the mechanism intact, which makes the next occurrence harder to catch, because now it doesn't look like the same bug. And a fix aimed at the wrong layer can introduce new failure modes entirely: a prompt change meant to clear up one ambiguity can create a second one, or tighten a constraint in a way that breaks a downstream tool call nobody was watching.

Agent outputs are non-deterministic. One passing rerun after a change proves almost nothing, since the same input can produce a different sample on a different pass. This pattern recurs across production deployments: a team sees a failure, reruns the prompt, watches it pass, and closes the ticket. The same failure surfaces later for a different user, and nobody has an explanation, because nobody ever had one the first time either.

That's the compounding part. Each unattributed fix stacks another layer of untested change on top of a root cause that was never identified. Call it dark matter: changes accumulating in the system whose actual effect on the harness is unknown, because nobody measured it. Future failures get harder to attribute, precisely because the baseline is no longer clean, and there's no telling anymore whether a new symptom is the old bug resurfacing or a fresh one caused by last month's guess.

Multi-agent systems in production fail somewhere between 41% and 86.7% of the time, depending on the system. At failure rates in that range, a team shipping fixes without attribution is running in place, reacting to symptoms with no leverage over the cause. It's running in place, reacting to symptoms with no leverage over the cause.

There's a security dimension too. The Hugging Face incident cited in the Scale AI paper required reconstructing an intrusion timeline out of more than 70,000 agent messages and files. At that scale, misattribution in the forensic record isn't a debugging inconvenience, it's a safety failure. Attribution isn't a nice-to-have for the debugging workflow. Attribution is the prerequisite: without it, any intervention afterward has an unpredictable effect.

Why attribution is a search problem

The naive approach to debugging an agent trace is to look at the last few steps, spot the wrong output, and fix whatever prompt sits closest to it. This fails constantly for long-horizon agents, because the error usually originates early, propagates silently through several steps, and only becomes visible far downstream from where it actually started.

The Scale AI Continual Search paper (arXiv 2609.13463, September 2026) frames the problem exactly right: the relevant evidence is sparse, scattered across actions that are nowhere near the visible failure, and disconnected from it in any obvious way. That makes attribution a search problem. One-shot LLM judges, which are the current standard tool for this, settle on the first plausible explanation and stop looking. Evidence sitting deeper in a long trace goes unexamined, simply because the judge already had an answer it liked.

The paper's results back this up with numbers. On the MegaRCA-Mix benchmark, fifty human-annotated failure trials drawn from long-horizon, execution-heavy tasks, an iterative search framework lifted GPT-5.5's F1 score from 0.349 to 0.498, an improvement of more than 40%. The model didn't change. The search strategy did. That's the whole finding in one line: attribution quality is a function of how thoroughly the evidence gets searched, not how capable the underlying model is.

The TraceElephant benchmark (arXiv 2604.22708) supplies the companion result. Full execution traces improve attribution accuracy by up to 76% compared to partial-observation settings, and in at least 21% of the 184 failure cases in the Who&When benchmark, developers simply cannot attribute the failure reliably from output-only logs, no matter how carefully they look.

What this demands of the tooling is specific. Step-level traces need full inputs, not just outputs, because partial visibility leaves too many failures permanently ambiguous. Per-step evaluator scores need to attach to each span, because "agent fail rate went up" is not something anyone can act on without knowing which step degraded. Failed trajectories need to get diffed against successful ones on the same task, since the signal usually lives in the contrast between the two, not in the failed run examined alone. And the review process needs to be iterative, because a single pass through a long trace will miss the root cause more often than not. Human review doesn't scale to this. The TUM survey identifies the sheer volume of execution logs as the direct reason the field moved toward automated attribution methods.

Tool schema drift as the clearest case where attribution must precede the fix

Tool schema drift is where this entire argument gets concrete. A tool's expected input shape changes, a field gets renamed, a new required property appears, an enum shifts, a validator gets stricter, and the harness calling that tool has no idea any of it happened.

Two failure modes come out of this, and they need entirely different fixes. A schema mismatch throws a runtime error. It's visible, at least, though teams often still misread it as the model "calling the tool wrong" rather than the tool's interface having moved out from under it. A description mismatch is quieter and worse: the model starts calling the tool in situations it shouldn't, or stops calling it when it should, and there's no error anywhere. It does not appear in type checking. It won't show up in unit tests. It appears as behavior that's slightly off, for reasons nobody can point to.

The n8n postmortem from February 2026 is a clean illustration. Upgrading from version 2.4.7 to 2.6.3 broke both OpenAI and Anthropic integrations at the same time, because the upgrade generated invalid tool schemas, sending type: "None" instead of type: "object", and both providers rejected it. The schema for tool arguments had changed between versions, and there was no mechanism in place to surface that change to any harness relying on it. Teams hitting this without attribution to the tool layer would reach for the obvious levers: adjust the prompt, maybe swap the model. Neither one touches the actual fault, because the actual fault is in the schema, not in the model's reasoning.

This isn't rare. Even well-engineered systems see individual tool calls fail somewhere between 3% and 15% of the time, which makes schema drift a standing operational condition rather than an occasional surprise. And it splits into recurring subtypes that each need a distinct fix: wrong-args, where the argument shape fails validation and the agent just retries with the same broken shape (the fix is schema correction, not smarter retry logic); tool-hallucination, where the agent invokes a function that isn't even in the runtime catalog (the fix is registry hygiene); no-error-handling, where a tool returns a 500 and the agent fabricates a plausible-looking response on top of it (the fix is exposing the error, not tuning the prompt); and API-drift, where a third-party endpoint changes its schema while the CI mock keeps returning the old response (the fix belongs in the CI harness, not anywhere near the agent's reasoning).

The industry has settled on three main defenses against this: semantic versioning discipline, extended to cover description-level changes as major version bumps, tool surface hashing inside CI pipelines, and behavioral evaluation against critical user journeys. None of these help a team that hasn't first attributed the failure to the tool layer instead of the model layer. A prompt fix applied to a schema drift failure leaves the schema exactly as broken as it was, and now there's a new, untested prompt change interacting unpredictably with a tool interface nobody actually diagnosed.

What a rigorous pre-ship attribution workflow looks like in practice

Start with the failed trace itself. The output is the symptom. The bug is almost always sitting somewhere in the middle of the trajectory, several steps removed from where it finally becomes visible.

Step one is collecting full execution traces with complete inputs at every step. This is the difference the TraceElephant benchmark measured directly: full traces raise attribution accuracy by up to 76% over output-only logs. Every step needs to be captured as something like an OpenTelemetry span, carrying a step identifier, the tool name, the model used, the prompt, the output, any retrieved context, and a parent trace ID. Skipping the full inputs leaves at least 21% of failures ambiguous no matter how good the analysis afterward turns out to be.

Step two runs per-step evaluators to find which step actually broke. Task completion alone is a poor signal here, it tells the team something failed, not where. Scores for things like tool selection accuracy, groundedness, and reasoning quality, attached at the step level, turn a vague "fail rate is up" into a query someone can actually filter on. A low score at step three of an eight-step trajectory narrows the investigation to one place. Without it, all eight steps are equally suspect, which in practice means none of them get properly examined.

Step three diffs the failed trace against a successful trace on the same task. The root cause tends to live in whatever's different between the two runs. The diff surfaces which input, which prompt element, which tool call, or which context condition changed, and that difference becomes the attribution candidate to pursue.

Step four assigns the fault to a specific layer, model, harness (and within that, prompt, tool, workflow, or memory), environment, or grader. This step isn't done until someone can state the mechanism in a sentence: the planner's tool descriptions overlapped with an unrelated tool's signature whenever the input contained urgency language, for instance, rather than the vaguer "the planner picked the wrong tool."

Step five validates the fix against historical failed traces before anything ships. Replay the fix across the cohort of past failures to confirm the failure mode is actually gone, and check that passing traces still pass, since a fix that clears one failure class can quietly break another. A fix that skips this replay step is still a guess, even when the attribution behind it was correct.

Step six folds the attributed failure into a regression dataset, so the same root cause gets caught automatically if it ever comes back. That turns root cause analysis into a feedback loop instead of a one-time investigation, which is the framing FutureAGI's 2026 guide pushes toward.

The mistakes that short-circuit all six steps tend to be the same ones every time. Rerunning the prompt and calling a single pass a debug session is the most common one, and it proves almost nothing given how non-deterministic these systems are: one passing run after a change could just as easily be a different sample from the same broken distribution.