Trace Corpus Curation for Replay Eval Suites

Replay evaluation only works if the trace corpus behind it looks like real traffic, carries correct labels, and holds together well enough to rerun. None of that happens by accident. Production trace archives grow out of ordinary usage, so they skew toward the happy path, thin out fast on edge cases, and give no guarantee that any single trace even has the fields a replay needs. Diagnosing failures across an archive like that is still mostly manual: someone pulls a handful of traces, reads them closely, and moves on, missing whatever pattern is visible only when you look across the whole population. Building a replay suite on top of that kind of archive makes the suite test the agent against the wrong distribution, quietly skipping the exact failures it was built to catch. Most teams treat this as a tooling problem. It's a curation problem, full stop, and curation is where the whole discipline lives or dies.

What a trace contains and why structure determines curability

A trace is the full causal record of one agent run: every nested span tied together under a shared trace_id, in order, with parent-child relationships intact. Each span, in the OpenTelemetry sense, covers one timed operation and carries a timestamp, a status, a pointer to its parent, and whatever attribute metadata got attached along the way.

For replay purposes, a trace needs four kinds of spans to count as structurally whole. Tool-call spans record the tool name, the arguments passed in, what came back, how long it took, retry counts, and error state. Reasoning spans capture the plan, the action the agent picked, the observation that followed, and what it decided to do next. State transition spans log working memory before and after each step, plus any handoff payloads passed between components. Memory operation spans cover reads, writes, retrieval scores, and how fresh the retrieved data was.

If any one of those four is missing, the trace is a partial record. It can still get logged and stored, but it can't be replayed with real fidelity, because whatever's missing is exactly the detail a rerun needs to reconstruct what actually happened. A trace missing its memory-operation spans tells you the agent retrieved something, but not what, and not how stale it was. So a rerun can't tell whether a bad retrieval or a bad reasoning step caused the failure downstream, and that ambiguity is fatal to the whole exercise.

The sub-population problem: aggregate pass rates hide the traces you need

Scale AI's Insights Generator corpus (hle_gpt-5.4_20260430) makes this concrete. Out of 250 traces, the baseline pass rate is 23.6%: 59 correct, 191 incorrect. Looked at as one number, that seems like the whole story. It isn't, and that's the problem: the single number buries the 191 incorrect traces sitting inside it.

Break the same corpus down by tool use and a different picture appears. Of the 250 traces, 207 involve at least one tool call. The 43 that involve none score 9.3% accuracy. The 207 that use tools score 26.6%. A seventeen-point gap, sitting entirely inside a corpus whose aggregate number gives no hint it exists.

A replay suite built by sampling on overall pass or fail alone would badly underrepresent the zero-tool failure mode, simply because zero-tool traces are a minority and their failures don't move the aggregate enough to get noticed. A failure mode tied to a specific input type, task category, or execution path doesn't have to be common to matter. It has to be looked for directly, because the aggregate metric was never built to reveal it.

Selecting traces for coverage: the behavioral axes that matter

Coverage is not volume, and treating the two as the same thing is the most expensive mistake a team can make when building a corpus. A tightly stratified set of 250 traces beats a pile of ten thousand traces that are all variations on the same successful run. Volume without spread just buys more copies of what's already understood.

Build the corpus around tool usage first: whether a trace uses tools at all, which ones, how many calls. The zero-tool versus tool-using split above is the clearest possible argument for why this axis carries weight, since it's the one place in the Insights Generator data where a 17-point accuracy gap sat in plain sight, invisible until someone split the corpus that way.

Reasoning depth is the second axis to track: how many plan-act-observe cycles a trace runs through, whether it branches or backtracks, whether the task resolves in one step or five. A single-step trace and a five-cycle trace that both end up marked "correct" are not testing the same thing, and treating them as interchangeable throws away information a curator paid for.

Failure mode is the third axis, and the one curators skip most. Final outcome, correct, incorrect, partial, is only the surface. What matters more is the intermediate signal, such as a wrong tool argument, a context window truncated mid-task, a retry loop that never terminates, or a computation that fails silently without tripping any error. A corpus that tags outcome and ignores mechanism can't support the kind of diagnosis a replay suite exists to enable.

Labeling traces: what ground truth requires

Getting pulled from production does not make a trace ground truth. A trace becomes ground truth once it carries a verified expected outcome plus enough metadata to actually replay it, and both of those get added by hand. Neither can be assumed from where the trace came from.

A usable label set needs a task-level outcome (correct, incorrect, or partial, graded by a human or a checked automated grader, not a raw model self-assessment), a failure attribution naming which layer broke (the prompt, a tool, memory, the workflow logic, or the model itself, rather than just recording that the run failed), a harness snapshot capturing the exact prompt version, tool schema, and memory state active at execution time, and anomaly flags marking known edge cases, distribution shifts, or traces pulled from incident windows.

Attribution is the hardest of these to get right, and any team trusting automated attribution without checking it is building on sand. The ICML 2025 Spotlight paper "Which Agent Causes Task Failures and When?" found that the best automated method identified the responsible agent correctly only 53.5% of the time, and pinpointed the responsible step only 14.2% of the time. That gap between agent-level and step-level accuracy is wide enough that an automated attribution label should be treated as a hypothesis a curator still has to check, never as a finished answer sitting quietly in the metadata.

Fixing the corpus vs. re-collecting: the design choice that determines reproducibility

Borrow the academic convention directly here: collect traces under fixed, documented conditions and reuse that exact corpus for every downstream experiment, every ablation, every intervention test. Re-collecting traces for each new test is not a shortcut. It destroys the thing that makes the comparison mean anything.

The reasoning matters more than the convention itself. If the corpus shifts between one experiment and the next, a change in results can't be pinned on the intervention under test rather than on the new batch of traces simply looking different by chance. Holding the corpus fixed turns it into a controlled variable, and what's allowed to vary is the diagnostic system, or the patch, or the harness change being tested, never the ground the test stands on.

That has practical consequences for versioning. Once a snapshot goes into eval use, treat it as immutable: additions get appended, nothing gets edited in place. Every corpus version needs to carry a record of the harness snapshot it was collected under. When new failure modes or task types get added, that's a versioned release with its own identifier.

Sampling strategy: keeping the traces you need, not the ones that are easiest to keep

The most common sampling mistake is also the easiest to fall into by accident: production traffic is mostly successful runs, so sampling uniformly across it produces a corpus that's mostly successful runs too. That's downloading whatever's biggest and calling it representative.

Tail-based sampling is the better default, and the case for it isn't close. Keep every failed trace, every unusually expensive one, every anomaly, in full, and sample the successful, ordinary runs much more aggressively. Head-based sampling, at high traffic volumes, tends to drop precisely the traces a team needs most right when an incident hits, because those traces are rare by definition and a uniform sample has no reason to catch them.

Certain categories deserve full or near-full retention: every trace a scorer flagged as a failure or a quality regression, everything collected during an incident window or right after a harness change, traces representing task types or tool combinations underrepresented elsewhere in the corpus, and traces showing known structural problems like runaway loops, context truncation, or long retry chains.

Other categories are fair game for aggressive downsampling: structurally identical successful runs on common, well-understood task types, and traces drawn from stable, already well-covered regions of the input distribution. Keeping a hundred near-duplicates of the same clean success case adds storage cost and nothing else.

Cohort comparison as a curation tool, not just an analysis tool

Cohort comparison usually shows up after collection is done, as a way to compare groups of traces that differ along one axis: model variant against model variant, one prompt version against another, one task category against another.

The less obvious use is running that same comparison during curation, on the candidate corpus itself, to check whether the coverage axes a team thinks it built actually produce different behavior. If two cohorts supposed to differ along some axis turn out to look statistically identical, that's a signal the axis isn't load-bearing, and time spent stratifying the corpus around it was wasted.

Cohort comparison earns its keep here because it isolates behavioral differences tied to one controlled axis, a sharper signal than any aggregate number gives. Take the tool-using cohort against the zero-tool cohort from the Insights Generator data and compare them on more than pass rate: the behavioral dimensions that distinguish how each group fails, not just whether it fails. If the two cohorts differ only on pass rate and look identical on everything else, the corpus hasn't yet captured why the gap exists, and that gap should be chased down before calling the corpus finished.

Harness versioning as a corpus integrity problem

Replay eval's whole premise rests on comparing agent behavior before and after some harness change. That comparison only means anything if the baseline traces were collected under a harness state that's known and on record.

Multiple kinds of harness artifacts can change independently of each other, including prompts, tools, and memory configuration. Any one of them shifting changes what a "normal" trace looks like, so a trace collected before a tool schema update and one collected after are, in a real sense, measuring two different systems, even if the task looks identical on paper. Mix the two together in a replay corpus without flagging the difference and the resulting baseline is misleading before anyone even runs an experiment against it.

The fix is metadata discipline at collection time, not cleanup after the fact. Every trace should carry a prompt version identifier, a tool schema version, a memory configuration snapshot covering retrieval strategy and store version, and a skill library version wherever skills are part of the setup. If any of these are missing, a future curator has no way to tell whether two traces are really comparable or just happen to look alike on the surface.

Connecting corpus quality back to the improvement loop

The whole appeal of replay eval is that it lets a team test a proposed harness change against historical traces before shipping it. That promise is only as good as the traces backing it, and every section above has been making that same argument from a different angle.

Curation quality directly affects outcomes. In a controlled study, engineers given corpus-level diagnostic reports from the Insights Generator corpus improved scaffold performance by 30.4 percentage points over the unmodified baseline. That number makes the case for the discipline on its own: systematic, evidence-backed corpus analysis beats ad-hoc trace inspection by a margin large enough to matter, whether the yardstick is a production incident or a benchmark leaderboard.

A separate example points at the same mechanism from another angle. AgentDebugX's closed-loop approach repaired 13 of 73 failed tasks in a single rerun, against 4 to 6 repairs for single-pass baselines, and lifted overall task accuracy from 55.8% to 63.6%. The structural advantage came from reasoning across the full trajectory of a run rather than a single step in isolation, and that kind of reasoning is only possible if the underlying trace is complete enough to support it. Multi-turn diagnosis needs a multi-span trace. A multi-span trace needs deliberate collection and labeling. None of it happens by accident.