Memory Layer Regression Testing for Long-Running Agents

A long-running debugging assistant reads the same README every Monday morning. Each time, it retries the exact fix that crashed the build the previous Friday. Nothing in that week's conversation is wrong on its own: the agent parses the file correctly, reasons about the fix correctly, and writes a coherent plan. The failure lives somewhere else entirely, in what the agent remembered from Friday and chose not to carry forward, or carried forward incorrectly.

Memory failures that standard evals miss

That scenario describes the article's core subject: memory layer failures in long-running agents form a distinct category of failure, one that compounds across sessions, evades single-turn evaluation, and demands its own regression testing discipline built around time, not isolated prompt-response pairs. Standard evals score an agent by whether a given prompt produces the right output. That framework works when the agent's state resets between tests, but a long-running agent doesn't reset. It carries working memory, episodic records, and semantic knowledge forward from one session to the next, so a bad write made in session 3 can sit dormant, invisible to every test that runs between sessions 4 and 16, and become visible as a wrong answer in session 17.

MemTrace, a 2026 benchmark from Long et al., makes this concrete. Pooled across 15,422 question rows, a memory system can post strong aggregate accuracy and still fail on the same underlying fact whenever that fact gets probed differently: as something aged by several sessions, as a trajectory of change rather than a snapshot, or under an evidence condition that's been altered or withheld. Holding the fact fixed and varying the conditions around it exposes the failures that pooled accuracy hid. The gap here is a mismatch between the unit teams use to measure agents (a prompt and its response) and the thing that actually breaks (state that only exists, and only fails, across time), not a shortfall in model capability.

What the memory layer owns across sessions

Testing memory well starts with naming what memory is in operational terms. Working memory holds the active context an agent reasons over within a session, and it fails in two characteristic ways: by overflowing when too much accumulates, and by carrying stale in-context state into a tool call that needed fresh input. Episodic memory stores records of past events and outcomes, and its distinct failure mode is encoding a flawed reflection, a post-mortem written after a failed attempt, that then recurs. Reflexion-style self-critique mechanisms, which generate a verbal lesson after a bad outcome, can lock in a wrong lesson just as durably as a right one.

Beyond these classical categories, the Always-On Agents survey by Ding et al. (2026) names a set of control-relevant state that most teams don't think of as memory at all, even though it behaves exactly like it: task ledgers, permission state, tool and credential state, trigger conditions, and side effects already committed to the outside world. All of it persists. All of it can corrupt what the agent does next.

That same survey identifies a gap in how the field studies the full lifecycle of this state. Research effort piles up around accumulating and retrieving memory, and far less goes toward governing it, recovering it, or letting it go. The stages that get the least testing in practice are exactly the ones that matter most once an agent runs for weeks or months: updating a fact, forgetting a fact, auditing what was stored, and rolling back a bad write.

How memory failures compound rather than isolate

A prompt-response failure is contained: one bad answer, isolated to one turn. A memory failure behaves differently. A single bad write doesn't produce one bad answer; it shifts the distribution of everything downstream, every later retrieval, every tool selection, every reflection the agent generates from that point forward. The error doesn't stay where it started.

The Multi-Layer Memory Framework paper, published in 2026, names the pathology directly: cross-session drift and unbounded context growth produce semantic drift and unstable memory retention over extended sessions. The slow decay of what the memory layer as a whole represents as true is the problem, distinct from a string of individual recall misses.

One mechanism makes the compounding effect concrete. When a system prompt contains ambiguous instructions, an agent resolves that ambiguity early, often in session one, and commits its interpretation to memory. Every later session retrieves the resolved interpretation, not the original instruction. An engineer debugging a failure in session 12 is looking at the downstream effect of a decision made in session 1, not at the instruction that produced it. Behavioral analysis done at the point of failure will consistently point to the wrong cause.

MemTrace's sharpest finding cuts against the instinct to fix this by improving retrieval. Across its evaluation of 13 memory-system configurations spanning four different design paradigms, the dominant source of failure was evidence use, not evidence availability¹. When these systems failed, the relevant fact was retrievable far more often than it was missing. Better retrieval infrastructure won't solve a problem that originates in what gets written to memory and how that written record gets applied afterward. Fixing the pipe doesn't help when the water going in was already wrong.

Three failure categories that require memory-aware test coverage

Memory regressions cluster into three categories, and all three share one structural trait: they are silent exactly where they occur and only become visible in behavior much later.

The first is memory-induced tool drift. Schema validation catches a malformed tool call the moment it happens, a missing field or a wrong type will fail immediately. It cannot catch an argument that's syntactically perfect and semantically wrong because the agent generated it from a stale episodic record. The agent calls the right tool with the right shape of input and the wrong content, and nothing at the point of execution flags it. Catching this requires watching how tool-call arguments shift over time rather than checking schema validity at a single point.

The second is selective forgetting failure. The Always-On Agents survey describes a governance gap in how "forgetting" gets implemented: many systems treat deletion as a retrieval-time edit that filters a fact out of direct queries while leaving it in storage. The fact survives a different phrasing of the same question, an oblique angle, an adversarial rephrasing. Testing for this means deliberately checking a known-outdated fact after an update, querying for it through the direct question that triggered the deletion, but also obliquely and adversarially.

The third is reflection encoding error. Reflective self-improvement mechanisms write verbal post-mortems after a failed task attempt, and those post-mortems shape every future retrieval in that domain. MemTrace operationalizes this failure with its "trajectory of change" question type: a system can correctly answer what's true right now and still fail to track how that fact got there, encoding an outdated state as though it were the current one. Testing for this means probing not just the current state of a fact after it changes, but the agent's grasp of the transition itself, what was true before, and what changed.

All three categories share the same blind spot. An eval that resets state between test cases cannot see any of them, because each one only exists in the space between sessions.

What memory regression testing requires

Four requirements separate memory regression testing from a standard single-session agent eval: state has to persist across test runs, test cases have to be ordered in time, the probe has to measure more than final accuracy, and the harness needs a way to check what was actually written to memory.

Persistent state across test runs is the non-negotiable structural difference. A standard eval harness resets state between test cases by design, because that's what makes each case independent and reproducible. Memory regression testing needs the opposite: state has to carry forward from one test to the next, the same way it carries forward from one production session to the next.

Temporal ordering follows from that. Test cases can't stand alone; later cases have to depend on what earlier cases wrote, mirroring the way a real agent accumulates sessions over weeks.

Probe dimensions beyond final accuracy come straight from MemTrace's design: memory age (how many sessions back a fact first appeared), question type (current state, earlier state, or trajectory of change), and evidence condition (present, missing, or contradicted). Two systems can post the same pooled accuracy and fail in completely different places once these dimensions get varied independently.

A write-path oracle rounds out the list: the harness needs to assert what got written to memory and when, not only what the agent eventually says out loud. MemTrace's finding that evidence is reachable far more often than it's missing points straight at the write path as the dominant source of regression, which makes checking writes directly more valuable than checking final answers alone.

The Always-On Agents survey's proposed Always-On Evaluation Protocol (AOEP-v0) reflects this same shift at the level of evaluation design. It's a pilot contract that scores state mutation and recovery obligations directly. Treating this as a sign that the field is converging on this approach, not as a specific tool to adopt, is the right way to read it.

A layered test strategy: what to run at each stage of development

Memory regression testing works best as a layered discipline, with different instruments operating at different points in the development cycle. The first layer fits inside a sprint. The later layers take longer to build out, and each one still delivers value on its own before the rest of the stack exists.

Layer 1 runs write-path unit tests at pre-commit. These test individual memory operations in isolation: a write followed immediately by a retrieval should return the fact that was written; a contradictory write should trigger whatever resolution policy the system defines; a delete should make the fact unrecoverable under every known variant of the query that might surface it. These tests run fast and deterministically, without needing a full agent execution loop, and they catch exactly the gap the Always-On Agents survey flags at the validate stage: write-time filtering and contradiction handling that most systems underinvest in.

Layer 2 runs multi-session regression traces before deploy. This means running a fixed sequence of sessions against a known starting memory state, probing the agent's behavior at defined checkpoints along MemTrace's three dimensions (age, type, evidence condition), and comparing the results against a baseline recorded before the change went in. The main obstacle here is non-determinism: re-running an agent rarely reproduces its original trajectory exactly. Chronicle, a cut-point replay approach from Chawla and Koul set to appear at the REALM workshop at EMNLP 2026, addresses this directly by recording the decision boundaries of a trajectory and replaying a chosen subset of those boundaries live against new code, while serving the rest from the recorded trace. That keeps the test both deterministic and scoped to whatever actually changed. Tier session sequences at this layer: a smoke test running two sessions with one fact update, a core test running five sessions with overlapping facts and one contradiction, and a torture test running extended sequences built to probe selective forgetting and trajectory tracking specifically.

Layer 3 samples production traces continuously. Sampling should include both the lowest-scoring traces and a random cross-section, labeled to catch drift that none of the pre-deploy test cases anticipated. Instrumenting the system with OpenTelemetry, capturing every memory read, write, and retrieval as a span, gives the observability needed to catch a distribution shift between what the agent writes and what it later retrieves, which is the signal that tends to precede a visible behavioral regression. Traces that surface a genuinely new failure pattern get promoted into the pre-deploy regression suite, so production observation feeds back into test coverage over time.

Layer 4 covers rollback and audit readiness; it's the slowest layer to build. The Always-On Agents survey finds rollback to be the rarest capability across the literature it surveys: most systems can suppress a fact at retrieval time but can't genuinely restore a prior state once it's gone. Testing a rollback path matters because, once a bad memory write reaches production, a team's ability to recover depends entirely on having exercised that path before needing it under pressure. The Long-Term Memory Security survey by Lin et al. (2026) frames this as a security requirement rather than an operational nicety: durable memory security has to be anchored in storage-time provenance, versioning, and policy-aware retention from the start, not patched in later at the point of retrieval or execution.

Validating that a memory change doesn't silently degrade behavior before shipping

Validating a memory change before it ships means running it through all four layers in sequence, not treating any single layer as sufficient proof the change is safe. A write-path unit test confirms the mechanics of the change are correct in isolation: the write lands, the contradiction resolves the way it should, the delete actually removes what it claims to remove. That confirms the component works, but it says nothing about how the change behaves once it's embedded in a real session history.

The multi-session regression trace is where a change gets tested against its actual operating conditions. Running the tiered sequences, smoke, core, and torture, against both the old and new memory logic, and comparing the agent's answers at each MemTrace-style probe point, reveals regressions that live in the gap between sessions. Chronicle's cut-point replay keeps this comparison honest even when the agent's own outputs are non-deterministic, because it holds the surrounding trajectory fixed while only the changed component runs live.

Production trace sampling after the change ships closes the loop. A memory change that passes every pre-deploy test can still shift the distribution of what gets written in production, in ways no pre-deploy sequence anticipated, simply because production sessions are longer, messier, and more varied than anything a test suite can script in advance. Catching that requires watching the gap between what the agent writes and what it later retrieves, not waiting for a wrong answer to appear downstream.

Rollback readiness is the final check, and it answers a different question than the first three: not whether the change behaves correctly, but whether the team can recover if it doesn't. A memory change that can't be rolled back cleanly carries risk that no amount of pre-deploy testing eliminates, because the failure mode it's guarding against is exactly the one regression testing is built to catch late: a bad write that looks fine in session 3 and doesn't show its cost until session 17.