Failure Cohort Detection From Live Agent Traffic

Failure cohort detection groups semantically similar failure traces from live agent traffic into named, recurring types, and each type has a hypothesized cause and a set of member traces. When teams adopt it, they move from reacting to individual bad outputs to tracking a small number of named, attributable failure classes across releases, and that is the engineering discipline that turns production monitoring into actual improvement.

Why individual failure traces hide recurring problems

Most teams debug agents the way they debug any other software: something breaks, an engineer opens the trace, reads through the steps, and decides what went wrong. A single failed run reads like an isolated event: a bad input, an unlucky tool response, an edge case nobody anticipated. The natural response is to patch that one case and move to the next ticket.

The trouble is one of scale. If you have no shared, repeatable method for locating the failure point, conclusions drawn from manual review do not accumulate into anything a team can trust across dozens of incidents.

Research on multi-agent failure management describes this directly. Fixes land on symptoms. This reflects the absence of tooling that aggregates patterns across runs rather than inspecting them one at a time, not negligence on the part of engineering teams.

What failure cohort detection groups

A failure cohort is a named, recurring failure type backed by a set of member traces and a working hypothesis about its cause, not a tag applied after the fact or a filter on a dashboard. The raw input is the production trace itself: the complete execution record of a run, including every model call, every tool call with its inputs and outputs, retrieval steps, intermediate reasoning, and any retries the agent attempted along the way.

"Semantically similar" is the operative phrase, and it means something specific: two traces share a common failure mechanism, not just a common error message or status code.

This is a different exercise from anomaly detection, which flags individual traces that deviate from baseline behavior one at a time. Cohort detection takes those flagged deviations and asks which ones belong to the same family, aggregating them into a class that shares a cause. Cohort detection operates on behavioral and semantic content, which matters most in exactly the cases where no exception ever fires. You get no systematic way to aggregate these patterns across runs with manual review workflows, so the same failure class can resurface hundreds of times before anyone recognizes it as one problem rather than many. Platforms built to ingest production traces and surface recurring patterns address that blindness directly: they make the aggregation automatic and the causal attribution explicit, instead of leaving it to whoever happens to be on call.

The production failure modes that cohort detection is built to find

Several failure patterns recur so often in live agent traffic that a team running cohort detection will name them, and each one becomes its own cohort with its own member traces.

Silent tool-call failures are among the most common. If a tool returns malformed or partial data, the model improvises around the gap and keeps the workflow moving, so no alert fires, and the run completes looking healthy to anything watching status codes.

Tool schema drift takes two separate forms. A description mismatch produces no error at all: the model calls the tool in situations where it shouldn't, or skips it when it should have been called, and nothing in the system flags the behavior as wrong.

If an agent calls the same tool repeatedly without ever converging on a result, that is a workflow loop. Tool-call drift of this kind is often the least instrumented failure mode a team has, because most tracing setups capture the LLM call itself but not the tool-graph traversal beneath it.

If a small shift in how a user phrases a request causes a retriever to surface the wrong tool entirely, that is semantic tool discovery ambiguity. Each of these produces a distinct trace signature, and all three can cluster into the same cohort once enough instances accumulate.

None of these five appear in final-answer pass/fail scoring. Agent evaluation research confirms that scoring only the final output misses trajectory quality, tool-call correctness, looping behavior, and recovery from error, because the signal that distinguishes one failure mode from another lives in the step sequence, not in whatever the agent said at the end.

From raw traces to a named cohort

Turning raw traces into a named cohort is a four-step process, and each step depends on the one before it.

The first step is representation. Each step of a trajectory has to be encoded in a form that makes semantic similarity computable, not just visually apparent to a reader. One approach, OAT, represents each step as a latent vector extracted from an LLM, then models the sequence of those step representations as a continuous latent path using Neural Controlled Differential Equations. A complementary approach, EAGER, builds a reasoning trace representation that encodes a multi-agent system's decision logic at each step, letting an engineer manage failures based on reasoning patterns.

The second step is anomaly scoring. At inference time, each step in a failure trajectory gets a score based on how far it deviates from the pattern learned on successful trajectories. OAT trains exclusively on successful runs, with no step-level failure annotations required, and detects error steps by measuring deviation from the expected successful path. That design choice makes the method practical for any team running a live system, because successful traces are a natural byproduct of ordinary production traffic.

The third step is clustering. The output of this step is a candidate cohort: a set of traces sharing a failure mechanism, with the specific step or steps responsible identified inside each member trace.

The fourth step is naming and validation. An engineer reviews a candidate cluster, gives it a name that describes the failure mechanism it represents, and checks it against its member traces to confirm they share the hypothesized cause. Naming matters beyond convenience: a cohort without a name can't be assigned to an owner, tracked across a release cycle, or marked closed once a fix ships. This is where semantic similarity has to resolve the gap between a shared error message and a shared failure mechanism; tools built around full production-trace analysis handle it because attributing a failure to its true source in the harness, whether the prompt, the tool, the workflow, or memory, requires parsing the complete execution record all the way to its origin.

Cohort detection requires online traffic, not a held-out eval set

A fair objection at this point is that a team already running offline evals should be catching most of this. Offline evaluation answers a narrower question than it appears to: it confirms that an agent didn't get worse on yesterday's tasks. It cannot anticipate inputs the agent hasn't seen yet, or multi-turn conversations that drift over the course of a session, or tool failures that only occur against live systems under real load.

The failure modes most suited to cohort detection, silent tool behavior drift, prompt ambiguity surfacing after a model version change, semantic retrieval errors triggered by unexpected phrasing, are exactly the ones that show up in production traffic and are absent from any eval set written in advance. An eval set is written against known failure modes. Cohort detection exists to catch the ones nobody wrote down because nobody knew to expect them.

The cohort detection loop depends entirely on this continuous labeling: without a score attached to every production turn, there's nothing to cluster, and cohorts stay invisible no matter how carefully someone reads individual traces.

Gradual drift makes the case most clearly. An agent degrading slowly over days produces no spike in any single metric that a weekly dashboard check would catch. Its failure traces accumulate quietly until the cluster crosses a size threshold and becomes visible as a named cohort, and an eval suite run on a weekly cadence or tied to deploys will miss this by design, because it isn't built to answer the question drift raises.

What root cause attribution adds to a cohort

A named cohort establishes that a failure is recurring. Root cause attribution establishes which layer of the agent harness is responsible, and that answer is what determines where the fix actually has to land.

AgentRx, accepted to EMNLP Findings 2026, operationalizes this kind of attribution directly. It pinpoints the critical failure step in a failed agent trajectory by synthesizing constraints, evaluating them step by step, and producing an auditable validation log, so it improves step localization over prior approaches and also provides failure-category attribution. The target of this process is specifically the earliest critical failure from which the agent never recovers, which is the step that needs fixing, as distinct from the step where the failure first becomes visible to a human reading the trace. OAT contributes a related signal by comparing each failure step's latent representation against the expected successful path and surfacing which steps deviate most, pointing directly to the layer that broke the normal execution flow.

Attribution has to produce a specific answer to be useful. "Prompt ambiguity in the tool-selection instruction" gives an engineer something to act on. "Something went wrong in the model" does not. The practical output of attribution on a cohort is a layer tag paired with a candidate change: every member of the cohort points to the same layer, and the fix is a targeted edit to that layer rather than an open-ended investigation that starts from scratch each time.

Validating a fix against cohort members before shipping it

Once a fix is proposed, the cohort's own member traces become the natural way to test it, because replaying the fix against them checks the specific failure mechanism under the conditions that actually produced it, rather than against a synthetic case written after the fact.

Two replay modes serve different purposes here. Replay mode runs the fix against recorded tool responses inside a deterministic environment, which is useful for CI and for regression attribution, because the same inputs produce the same outputs every time the test runs.

Replay mode carries more weight for cohort validation specifically. It lets the fix run against every member of the cohort without needing to regenerate live traffic, and it produces a result that's reproducible and reviewable by someone other than the engineer who wrote the fix. The validation protocol follows a specific sequence: replay the exact failing run using recorded tool outputs, diff the trajectory at each step, tool choice, tool arguments, intermediate summaries, retries, and confirm that the step where the original trace first diverged now resolves correctly. That first divergence point is the check that matters, not the final answer. A fix that arrives at the right final answer by a different wrong path has not actually closed the cohort; it has just moved the defect somewhere less visible.

Regression risk needs checking at the same time, not afterward. A cohort should be marked closed only once the fix passes replay against all its member traces, or against a statistically sufficient sample of them, not simply because the fix has been deployed to production. This is the discipline that platforms built to replay and analyze the full behavioral record of failed runs are meant to support: grouping failures by hypothesized harness cause, then validating a proposed fix against those same production traces before anyone ships it, rather than shipping first and hoping the cohort doesn't reopen.

How cohort tracking changes the improvement cycle over time

Maintaining a named cohort registry across releases builds up the evidence base needed to improve the agent harness systematically.

Each cohort moves through a lifecycle: open while it's active in production, fixed once a change ships against it, closed once replay confirms the failure class is actually gone, or regressed if the class reappears after a model update or a tool change. When a closed cohort reopens after a model version change or a tool update, it points to exactly which component reintroduced the failure, with more precision than scanning a deployment diff would provide.

The loop that emerges from this practice runs in a consistent order: a new cohort gets identified, attributed to a harness layer, a fix gets proposed, the fix gets validated through replay, it ships, the cohort gets monitored for regression, and it's either closed or escalated depending on what the monitoring shows. A team with enough cohort history can run a genuine ablation: remove an instruction, replay it against the specific cohort it was written to prevent, and confirm the model no longer needs the guardrail or restore the instruction with evidence behind the decision.

A low cohort recurrence rate, tracked over time, becomes a reliability metric a team can actually trust, because it's grounded in production traces rather than in a benchmark score that may not reflect what real traffic looks like. Moda's approach to this problem closes the loop end to end: ingesting production traces, surfacing emergent intents and behavioral failures, attributing them to the harness layer responsible, generating fixes that are reviewable before they ship, and validating those fixes against historical traces before recommending the change go out. Run this way, improving an agent becomes a repeatable engineering discipline, with a registry, a lifecycle, and evidence behind every change a team decides to make.