Wiring Replay Evals Into a CI/CD Pipeline for LLM Agents

Standard CI pipelines assume that running the same code twice gives you the same answer twice. That assumption falls apart the moment an LLM agent is in the loop, and replay evals exist to patch the hole: freeze the parts of a production trace you want held constant, and re-execute only the piece you're actually testing, live.

Why standard CI pipelines cannot reproduce LLM agent failures

The whole premise of a regression test is that you can rerun it and trust the result. Feed it the same input, get the same output, every time, forever. That premise holds for a sorting function. It does not hold for an agent that calls a model, reads a tool's response, and decides what to do next.

Three separate things break the loop. Model inference isn't bitwise reproducible even when temperature is set to zero, so the same prompt can come back with a different completion on two consecutive calls. Tools read from the outside world, and the outside world has usually moved by the time anyone reruns the trace, so a database query or an API call returns something new. Retry logic or routing decisions can change how many times a step fires in the first place, so the shape of the trajectory itself isn't fixed.

Test flakiness in the classic CI sense means an outcome depends on things outside the code under test, an assumption that collapses for agents. Here it is structural. It's structural: baked into what an agent is.

Existing observability tooling doesn't close this gap, it just documents it. Tracing tells a team what happened in a given run. Evaluation frameworks score whether the output looked acceptable. Neither one lets an engineer hold everything else fixed, swap in a single changed component, and check whether that specific change fixed the specific failure. That's the piece missing from the toolchain, and it's the piece replay evals are built to supply.

What replay evals do differently from re-execution

Replay doesn't try to reproduce the original run. It tries to test a fix against the incident that already happened, and that distinction matters more than it sounds like it should.

The mechanism is straightforward once you see it: freeze what you want to hold fixed, and run live only what you want to test. Every LLM call, every tool call, every routing decision an agent makes is a boundary where nondeterminism can enter the trajectory. A replay system records the envelope crossing that boundary, the inputs going in and the outputs coming out, without trying to capture or reproduce whatever internal computation produced them.

That is the part to sit with. A replay is a set of stubs standing in for the boundaries a run touched, not a snapshot of the whole run frozen in amber. It's a set of stubs, one per boundary, each one ready to serve back what was recorded there if a test doesn't need that boundary to run live.

Cut-point replay is what happens when an engineer decides which boundaries stay frozen and which ones get to run against new code. Say a prompt template changed. Every tool call and every model call gets served from the recorded trace except the one boundary that prompt actually touches, and that one runs live. The rest of the recorded trajectory holds the context stable around it, so the test isolates the change instead of testing the whole system at once.

The payoff is that a production incident, the kind that used to live in a postmortem doc and nowhere else, becomes a regression test that runs in CI without a single live model call outside the one boundary under test. Chronicle's implementation of this idea adds roughly 23 microseconds of overhead per boundary crossing, which against an assumed 300-millisecond model call works out to about 0.008 percent, effectively free https://arxiv.org/pdf/2609.20625. And in testing, Chronicle's full replay reproduced bit-identical results across 20 repetitions, which is the kind of stability an agent's live execution simply can't offer on its own https://arxiv.org/pdf/2609.20625.

Instrumenting production traces so they are replayable

Recording and logging are not the same discipline, even though they look similar from a distance. A standard observability log captures inputs and outputs for a human to read later. A replay-ready recording has to capture the full envelope at each boundary in a form another system can stub back deterministically, which is a much narrower and more exacting bar.

Each envelope needs a few things to actually be usable in a replay. It needs to record the boundary type, distinguishing an LLM call from a tool call from a routing decision. It needs every input that crossed that boundary: the prompt, the arguments, whatever state got passed in. It needs the recorded output, and for streaming responses, Chronicle records the assembled result rather than the token-by-token stream itself. And it needs a stable identifier, so that when a replay runs, the system can match a stub to the exact crossing it's meant to represent.

None of this is complete. Chronicle's current instrumentation has real limits, and any team building on it should plan around them rather than discover them mid-incident. Streaming responses are captured only in their assembled form, so the in-flight token stream isn't preserved for replay. Concurrent parallel tool calls aren't captured at all yet, which matters for agents that fan out multiple tool calls at once. Replay doesn't re-raise an exception that was recorded at a stubbed boundary. A test relying on exception-handling behavior at a stub won't see the original failure mode reproduced.

There's a security dimension here too, and it's not optional. A recording is, by definition, a copy of prompts, agent state, and tool arguments. It may contain secrets or PII. Chronicle handles this by applying redaction at record time, before anything is written or committed anywhere, stripping sensitive values while keeping the structural assertions a test actually needs intact. Any team standing up a replay pipeline should treat that redaction step as a pipeline requirement from day one, not something bolted on after the first near-miss.

The failure taxonomy that determines which boundaries to cut

Deciding which boundaries to freeze and which to run live isn't guesswork. It follows from understanding what actually breaks agents in production, and the research on this points in a consistent direction.

The MAST failure taxonomy sorts agent failures into three root categories. Specification problems, meaning role ambiguity, unclear task definitions, and missing constraints, make up the largest share. Coordination failures, meaning communication breakdowns between components, state that falls out of sync, or objectives that conflict with each other, come in second. Verification gaps, meaning inadequate testing and missing output checks, are the smallest of the three categories, which on its own is a useful thing to know.

It's an emphatic point. It means the model itself is rarely the thing at fault. The harness around it, the way tasks get specified and the way components coordinate, is where the failures actually live. Cut-point replay is built around isolating harness components rather than swapping out models.

Tool-invocation failures deserve their own line item because they recur often and in a consistent shape. Seven patterns recur: insufficient API calls, wrong argument values, wrong argument names, wrong argument types, calls repeated when they shouldn't be, function names that were hallucinated outright, and output formatting that doesn't match what a downstream step expects.

A real incident makes the abstract version concrete. What this kind of drift looks like from the outside should worry anyone running agents in production: it usually doesn't throw an error. It produces a response that's malformed but plausible enough to pass right through downstream, until a human eventually notices the output is wrong.

That's precisely the kind of failure cut-point replay is suited to catching, because it's a boundary-level problem: something crossing the tool-call boundary changed shape, and everything downstream of it kept running anyway. Together, specification and coordination problems account for roughly four out of five production breakdowns; the model is rarely the culprit, the harness is. Tool schema drift in practice: after upgrading n8n from v2.4.7 to v2.6.3, the platform began generating invalid tool schemas in tool calls, breaking both OpenAI and Anthropic integrations simultaneously (the schema for tool arguments changed between versions with no mechanism to surface the change to consuming harnesses); crucially, schema drift often does not produce an error, it produces a malformed-but-plausible response that passes downstream until a human notices wrong outputs.

How cut-point replay becomes a merge gate in CI

The job of the gate is simple to state even if it's not simple to build: block a PR if replaying a recorded failure against the candidate change shows the failure is still there, or if a trajectory that used to pass now fails.

That gate doesn't sit alone. It fits into a sequence of checks with different jobs and different budgets. Stage one is the PR-time layer, and it has to run in under 90 seconds or it starts costing engineers their flow state, so it sticks to schema lint, a small deterministic eval, and a check on token-budget delta, catching the obvious breaks before anything more expensive runs https://futureagi.com/blog/ci-cd-for-ai-agents-best-practices-2026/.

Configuring an actual replay test means making a handful of decisions explicit. Pick the recorded incident: the envelope library committed to the repo. Declare which boundaries run live against the new code and which get served from the record. For a prompt change, stub every tool call and model call except the one boundary that prompt touches, and let that one run live. For a tool schema change, stub the model calls and run the tool boundary live against the new schema in a sandbox. For a router change, stub the downstream model calls and run the routing decision itself live.

The pass criterion is easy to get backwards. The replayed run has to reach the corrected expected behavior, not the original recorded failure. The test validates that the fix actually fixes the issue. It's there to validate that the fix actually fixes it. Stage 3, Pre-deploy simulation, budget 10–60 minutes: persona-driven multi-turn simulation and guardrail checks complement replay by probing scenarios not yet in the recorded library. Stage 4, Canary plus rollback, budget 1–24 hours: online eval with auto-rollback catches distribution shift that neither replay nor simulation saw.

Interpreting results that mix stubbed and live boundaries

A replay result is a verdict that only means what it means in light of which boundaries were frozen and which ones weren't, and skipping that context is how teams end up trusting a green checkmark they shouldn't. It's a verdict that only means what it means in light of which boundaries were frozen and which ones weren't, and skipping that context is how teams end up trusting a green checkmark they shouldn't.

A pass tells an engineer that the live boundaries, the actual components under test, behaved correctly given the recorded context around them. It does not tell you whether the stubbed boundaries still represent how production behaves today. A stub is a snapshot of the past. If the tool it's standing in for has since changed its schema, a passing replay can sit right next to a failing production system, and the gate would never know.

A fail is more useful to sit with, because it comes with a real diagnostic question attached: did the failure start at a live boundary, or at the seam where a stub handed off to something live? That question does most of the attribution work by itself. Research on debugging tools like AgentDebugX's DeepDebug approach shows that attribution can get meaningfully accurate at pinning failures to a specific agent and step, and that feeding the system knowledge of the underlying code improves root-cause localization by a wide margin over earlier methods. A failure that appears at a live boundary points at the change under review. A failure at a stub-to-live seam points somewhere else entirely: the envelope itself, the recorded interface, no longer matches what the live system expects. That's often the fingerprint of schema drift, the same shape of problem the n8n upgrade produced.

None of this should be treated as ground truth just because a system produced a label. Diagnostic attribution can be wrong, and a results dashboard that hides its reasoning is asking to be trusted blindly. The evidence trail, not just the verdict, needs to sit next to the pass/fail flag so an engineer can actually audit the reasoning instead of taking it on faith.

Harness engineering practices that make replay gates sustainable over time

None of this works if the artifacts feeding it aren't taken as seriously as code. Prompt templates, tool schemas, workflow definitions, memory interfaces: these need version control and the same audit discipline as anything else in the repo, because a replay eval is only as trustworthy as the harness it's replaying.

Lilian Weng's framing of harness engineering from mid-2026 is useful here because it pushes the definition of "agent" past the old shorthand of model plus memory plus tools plus planning plus action. It folds in workflow design, evaluation itself, permission controls, and persistent state management as first-class parts of the harness. Under that framing, a replay eval is one of the evaluation artifacts that the harness is made of, on equal footing with the prompts and schemas it's testing. It's one of the evaluation artifacts that the harness is made of, on equal footing with the prompts and schemas it's testing.

A lot of agent failure, it turns out, comes from instability at the process level rather than incapacity at the action level. Steps get skipped. Operations run out of order. A task terminates early because nothing forced it to check whether it was actually done. Making the task skeleton explicit, breaking it into steps, phases, dependencies, and stopping conditions that are written down rather than assumed, turns what used to be implicit knowledge sitting in someone's head into a testable harness artifact that a replay gate can actually check against.

Tool schema discipline is the concrete version of all of this. The n8n incident is the object lesson: when a tool dependency gets upgraded, its schema needs to be re-validated against every harness consuming it before that upgrade reaches production, not after. It's a harness engineering issue, and treating it like a model problem misses the point entirely. It's a harness engineering checkpoint, as ordinary and as necessary as any other item on a release checklist, and skipping it is how a minor version bump turns into a production incident nobody sees coming until the outputs are already wrong. Pre-deploy simulation runs in 10-60 minutes testing persona objectives, guardrail trips, and unsafe tool calls https://futureagi.com/blog/ci-cd-for-ai-agents-best-practices-2026/.