Replay Eval Coverage Metrics for Agent Harness Changes

Most production agent deployments stall for a single, boring reason: someone changes a piece of the harness with no way to know if the change is safe. The field built agents faster than it built ways to test them, and that gap appears the moment a team touches a prompt template or a tool schema without a way to check the blast radius.

Agent failures don't behave like ordinary software bugs. A broken function throws an exception. A bad SQL query returns an error code. An agent can run start to finish, produce a fluent, confident, entirely wrong answer, and never trip a single alarm. Nothing crashes, nothing logs a stack trace, and the pipeline reports success because, technically, it finished. The output is just wrong, and nothing in the system said so.

That failure almost never lives in the model itself. It lives in the harness, made up of the prompt templates, the tool definitions, the workflow logic that sequences steps, and the memory or retrieval layer deciding what context the model sees. A survey of 55 papers on LLM agent trajectory analysis, out of TUM, names the problem directly: agents make non-deterministic decisions across long, language-heavy execution paths, and that combination makes it genuinely hard to pin down where a failure started. Changing one of those layers without a way to measure the change means a team is shipping on vibes, plain and simple. Coverage, in the replay eval sense, is the alternative: a concrete standard for what counts as tested enough before a harness change goes live.

The four harness layers where failures originate and the coverage logic each requires

Harness failures cluster into four layers. Treating them as one problem is the mistake most teams make, since each layer fails in its own way and needs its own kind of test.

The prompt layer fails through ambiguity, and ambiguity stays quiet. Renaming a field from user_id to member_id in a schema leaves the model generating the old key anyway, because every stored trace and every few-shot example still teaches it the old name. No type checker catches this. No unit test fails. The code is syntactically fine, and the drift becomes visible only in evaluator scores, well after the fact. Coverage at this layer means the replay corpus needs traces that capture the ambiguous encoding so a fix can be validated against the real failure.

The tool layer fails in two different ways, and only one of them announces itself. Schema mismatch throws a runtime error, so it gets caught fast. Description mismatch changes behavior with no error signal at all, which makes it the more dangerous of the two. The n8n incident is the clean case study here: an upgrade from version 2.4.7 to 2.6.3 generated invalid tool schemas, type: "None", inside the Vector Store Question Answer Tool, and it broke both OpenAI and Anthropic integrations at once, with no mechanism to surface the schema change to the harnesses relying on it. The industry has settled on three defenses: strict semantic versioning; hashing the tool surface in CI so a schema change can't slip through unnoticed; and behavioral evaluation against critical user journeys.

The workflow layer carries the highest variance of the four, by a wide margin. A documented postmortem describes a four-agent loop that burned $47,000 over eleven days because nothing was watching for cycles: an Analyzer agent and a Verifier agent kept handing requests back and forth to each other, indefinitely. Per-call rate limits didn't stop it, because each individual call looked perfectly legitimate on its own. Coverage at this layer means replaying traces that include retry paths, loop boundaries, and sub-agent handoffs.

The memory and context layer fails just as silently. Context failure is recognized as its own category: an instruction or file that's needed goes missing, goes stale, or gets dropped during compaction. A retriever that surfaces the wrong top-ranked chunk is a memory-layer failure, and it leaves no error signal anywhere in the generation step that follows. The model just answers based on the wrong context, fluently, and the mistake reads as confidence instead of error. Coverage here means the corpus needs traces where retrieval quality was the actual deciding factor between a pass and a fail, not traces where it was beside the point.

What replay eval does that offline test suites cannot

Offline test suites are snapshots, and snapshots age. Keeping one current against a moving harness is close to an impossible maintenance job, and none of those suites run on a per-turn basis in production anyway, so they miss the failures that only occur under real traffic.

Re-running a failed agent trace almost never reproduces the original failure because LLM inference isn't bitwise reproducible even at temperature zero. Re-running a failed agent trace almost never reproduces the original failure. LLM inference isn't bitwise reproducible even at temperature zero. Tools read from external state that has since moved on. Retries change step counts from one run to the next. Replaying a failure the naive way, by just running it again, gives a good chance the failure won't recur.

Replay eval gets around this with a specific contract: hold the recorded trajectory fixed, run only the piece of the system that actually changed, and check that the change fixes the original failure without breaking anything else the record captured. Chronicle is the most substantial recent primary work built around this idea. It records an agent run at its non-deterministic boundaries (model calls, tool calls, and similar decision points) as immutable envelopes. Cut-point replay then serves a chosen subset of those boundaries from the record and executes the rest live with new code, so a tool gate can run live while the model call underneath it gets stubbed from the recording.

Full replay, where every boundary is served from the record, issues zero live model calls and comes out bit-stable across 20 repetitions on 6 recorded failures. Recording overhead runs about 23 microseconds against an assumed 300-millisecond model call, roughly 0.008%, low enough to run inside CI without anyone noticing it's there. On the cut-point tests, faulty code fails and guarded or benign changes pass across all 6 incidents tested, and the method catches every mutant that lets a recorded unsafe action slip through. A baseline that stubs every single boundary catches none of them. That gap is the whole argument for cut-point replay over full stubbing.

Defining coverage: what the metric is measuring

Coverage here has nothing to do with line coverage, and treating it like a code-coverage problem is how teams end up testing the wrong thing. A harness change can touch every prompt template in the codebase and still leave the exact failure mode from the original incident completely untested, if the replayed traces never happen to include that failure mode.

Coverage needs three dimensions to mean anything, and skipping any one of them breaks the metric. Boundary coverage is the fraction of non-deterministic boundaries in a recorded trace that the cut-point test actually exercises: a boundary never crossed during replay is a boundary where a regression stays invisible, full stop. Failure-mode coverage is the fraction of known failure categories present somewhere in the replay corpus, since a corpus without edge-case traces won't include the failure modes that regressions tend to reintroduce. Layer coverage asks whether the corpus includes traces that trace failures back to each specific harness layer under change. Validating a prompt fix only against traces that don't exercise the changed prompt context leaves the actual failure mode untested.

The HARNESSFIX framework, from Chen et al. (arXiv:2606.06324), gives this layer-specific idea an analytical backbone. It introduces a Harness-aware Trace Intermediate Representation, HTIR, that normalizes trajectory evidence and lines up each runtime step with the specific harness artifact that shaped it. That alignment is what turns layer-specific coverage into something a team can actually compute, rather than something to sketch on a whiteboard and hope for the best.

Coverage is also conditional on something Chronicle makes explicit: only boundaries that are properly recorded and guarded can be checked during replay. A coverage metric that only counts guarded boundaries and ignores the unguarded ones is measuring the wrong thing, and doing so with total confidence.

Coverage thresholds by harness layer and the evidence behind them

No single number works across every layer, and hunting for one wastes time better spent elsewhere. The right threshold depends on how bad a failure at that layer actually gets, and on how much production trace volume a team has to draw from.

At the tool layer, a tool execution success threshold above 0.90 is one reference point from published evaluation frameworks, applied per deployment rather than as a universal fixed standard. Below that line, schema drift and description mismatch tend to recur, and in multi-step workflows those failures compound instead of staying isolated. The replay corpus needs traces where the tool was actually called against the schema version being changed, including cases where the call didn't succeed. The n8n incident is exactly the failure a corpus built this way would have caught before it shipped. CI needs at least one cut-point test where the tool boundary runs live against the new schema, since Chronicle's results show that a baseline stubbing every boundary catches none of the failures a cut-point approach catches.

At the prompt layer, published evaluation frameworks put safe-to-ship territory above 0.85 for context relevance across retrieved chunks. The corpus requirement is specific: it needs traces where the changed field, instruction, or few-shot example was the decisive factor in the outcome. Ambiguity fixes especially need checking against the traces that originally produced the ambiguous output, since that's the only way to confirm the fix addresses the thing that actually broke, rather than something adjacent to it.

At the workflow layer, the threshold for trajectory correctness (right tools, right order, within the step budget) should be calibrated carefully, given that some workflows already fail at a substantial rate. The corpus needs at least one trace with a retry or loop path in it, no exceptions. The $47,000 postmortem stands as the reference case for what happens when loop paths are simply missing from the test data. Every workflow change should also run against a trace that actually hits the step limit or timeout boundary, not one that stops short of it.

Root cause attribution as the prerequisite for coverage that means something

None of these thresholds mean anything if the traces feeding the corpus are mislabeled, and at scale, they often are.

As execution logs grow longer and more distributed across sub-agents, LLM judges used for root cause analysis tend to lock onto whatever failure looks plausible first, rather than working through the full trajectory of evidence. That's a real risk. A mislabeled trace ends up filed under the wrong coverage bucket, and every downstream check relying on that bucket inherits the mistake.

Who&When Pro is the largest labeled dataset built for this problem, with more than 12,000 labeled trajectories spanning different agent frameworks, domains, and modalities, and it's become the reference point for measuring attribution accuracy. On that benchmark, AgentDebugX's DeepDebug component reaches 28.8% exact agent-and-step attribution accuracy using a 9-billion-parameter backbone, against 21.7% for the strongest single-pass baseline. That's a 7-point gap, and 28.8% still sounds unimpressive standing alone. But the comparison is the point: single-pass attribution does worse, and an unattributed failure sitting in a coverage corpus is worse than either number suggests.

Continual Search, an iterative attribution approach, shows consistent gains over passive reconsideration on TRAIL and TELBench, the two benchmarks with the longest execution logs in the set. That matters directly for coverage, because long-trajectory agents are exactly where a replay corpus is most likely to carry mislabeled traces without anyone catching it.

Building the replay corpus: which traces to collect and how many are enough

Production traces are the right raw material, full stop. Reaching for synthetic data instead is a shortcut that costs more than it saves, since synthetic datasets tend to miss the long-tail inputs and the distribution shifts no offline suite was built to anticipate. A corpus built from them will pass tests that production goes on to fail.

A usable corpus has to clear a specific bar beyond a volume target. It needs at least one trace that actually captures the target failure, since a corpus missing the failure mode in question can't validate that a fix addresses it. HARNESSFIX is designed to improve on initial harnesses, and its trajectory-aligned approach depends on having a passing baseline to measure changes against. It needs traces where the boundary under change runs live rather than stubbed, since Chronicle's mutation study found that a baseline that stubs every boundary catches none of the failures that cut-point replay catches. And it needs edge-case paths represented directly: retries, loops, sub-agent handoffs, context compaction events.

Building this by hand doesn't scale, and that is where infrastructure earns its keep. The Harness-Evals framework offers importers, including one for OpenTelemetry traces, that map raw traces into typed EvalCase fields automatically, along with a store for saving score snapshots and a compare_to_baseline() function that flags regressions, improvements, and unchanged metrics against a configurable tolerance. For any team already collecting OTEL traces, that's a fairly direct path from raw production data to a typed, replayable corpus, without hand-transforming every trace along the way.