Measuring Cycle Velocity to Prove Your Agent Improvement Loop Is Compounding

A compounding improvement loop from production agent observability turns real failures into layer-specific harness changes, validates them against historical traces, and reduces the work required in later cycles. Cycle velocity proves that improvement only when four metrics agree, namely one-shot rate, compound velocity, repeat-mistake rate, and session gap rate.

Why "we ship fixes" and "our agent compounds" are not the same thing

Logging is running, dashboards look busy, and fixes are merging, but nothing is measurably accelerating. New failures still take as long to diagnose, and familiar failures come back after changes meant to fix them. This is a structural gap. Shipping records that a change reached production, but it does not establish that the change removed the underlying failure pattern.

A compounding loop extracts a learning from each failure, validates the resulting fix against real traces, and makes the next cycle faster. If a failure mode that once took weeks to catch now takes days, and the same failure almost never appears twice, improvement is accumulating. If cycle time stays flat and repeat failures are still common, the loop is getting work done without making less work for the future.

What compounding actually means in an agent improvement context

Each completed cycle must reduce the cost of the next. Fewer failures reach production, those that do are diagnosed faster, and validated fixes raise the starting level for subsequent cycles. Linear progress is different because each fix costs the same effort and covers the same ground. Stagnation goes further, with fixes shipping while failures recur at the same rate. Compounding therefore describes a change in the loop’s efficiency.

RRSI from Google Cloud AI Research identifies a concrete way that apparent progress can mislead. Harness self-improvement loops that evolve without regularization can show large in-distribution gains that shrink or vanish on out-of-distribution tasks. The evolve set improves, but unseen tasks do not. That gap shows overfitting instead of compounding, because the harness handles familiar inputs better but is no more reliable on other tasks. Production improvement requires generalization to new traffic.

The four metrics that distinguish a compounding loop from a spinning one

One-shot rate is the percentage of agent tasks that pass human review without requiring a change after the cycle runs. It is a cycle-level ratio across all tasks in a sprint. Its target trend is upward across cycles. A rising rate indicates that fixes are landing correctly on the first attempt, while a flat or falling rate suggests that changes are not sticking or are addressing symptoms.

Compound velocity counts learnings extracted and validated per sprint. Its target is steady throughput. A collapse can indicate that fresh production signal has dried up because trace capture or the golden task set is stale. A spike is not automatically good, either, because additional learnings may be noise if one-shot rate does not improve with them. These first two metrics describe forward progress, but they cannot establish that failures are being eliminated.

Repeat-mistake rate supplies that test by measuring the percentage of failure modes encountered again after a fix ships for them. It should fall toward zero with each cycle. Session gap rate is the percentage of production sessions that have a failure mode outside the known inventory. That rate should also fall as the team’s understanding converges on the real failure space. High compound velocity with high repeat-mistake rate suggests that the team is learning but leaving root causes in place, while rising one-shot rate and falling session gap rate show that fixes are working beyond the same set of known problems.

Why root-cause attribution at the layer level is a prerequisite for any of these metrics to mean anything

An outcome-level signal says that a run failed, but it does not identify whether the prompt, tool schema, workflow control flow, or memory context needs to change. The owning layer determines the useful intervention. Editing a prompt cannot remove an underlying tool schema failure, so the wrong attribution leaves the repeat-mistake rate exposed to the same recurrence. RRSI shows how harness engineering has driven recent progress in agent products more than new model weights have. For these metrics, treating every failure as a model capability problem obscures the parts of the system that can be repaired.

Failure labels must preserve those distinctions without reducing them to vague symptoms. Context rot is the gradual loss of reliable instruction and evidence use as working context grows or accumulates competing state. False completion means declaring the task done despite failing tests, while tool hallucination includes nonexistent tools or malformed arguments. Goal drift is cumulative deviation from the original objective, which can include acting on files outside the intended scope. Each of these failures needs its own harness-layer fix and validation signal.

Attribution also requires the trajectory. Intermediate states shape later decisions, so the evidence may be much earlier in a log or spread across parts far apart. In multi-step agents, errors can compound as steps accumulate, making a late symptom a poor guide to the initiating cause. Step-level tracing must therefore include model identifier, token counts, duration, status, and span kind for every LLM span. Retriever, tool, and agent spans need duration, status, and span kind, but not necessarily model identifiers or token counts. Without that completeness, repeat failures cannot be reliably attributed to the changes intended to prevent them.

How to use production agent traces to instrument each metric

All four metrics must originate in production traces. Offline cases reflect how developers expect users to interact, while live usage exposes how users actually interact. The instrumentation must preserve that distinction across cycles. Otherwise, a clean dashboard can describe improvement on a curated workload while missing the failures entering through current traffic.

For one-shot rate, tag each shipped fix with its cycle and the trace cluster it addresses. After the cycle closes, replay that cluster against the new harness version and score it with the same rubric used in CI. A task qualifies as one-shot when it passes human review without further revision. Compute the ratio across the cycle’s tasks. For compound velocity, count a learning only when it has a layer attribution, a concrete harness change, and historical replay validation before shipping.

Repeat-mistake rate needs a way to track the same failures across versions. Assign each diagnosed failure mode a canonical identifier keyed to its layer and specific failure signature. After its fix ships, monitor subsequent production traces for that signature and increment the repeat count on recurrence. The cycle-level percentage should report failure modes that recur, rather than treating every repeated occurrence as a new mode.

Session gap rate needs an inventory of known signatures that is kept up to date. Classify each new production session according to whether its failures match that inventory or include something new. The numerator is sessions with at least one new signature, and the denominator is production sessions. Track the rate weekly against inventory size to distinguish coverage improvement from inventory growth. Across all four measures, shrinking time from failure to fix is the leading indicator of a working loop, especially when failures that once took weeks to catch now take days.

Reading the metrics together and what each combination means for the loop

In healthy compounding, one-shot rate rises, compound velocity is steady, and both repeat-mistake rate and session gap rate fall. The likely explanation is that production learning is reaching the right harness layers and removing the patterns it identifies. The operational response is to continue the sprint cadence and validation gate, rather than pursue a higher learning count for its own sake.

In a spinning loop, compound velocity is high, one-shot rate is flat, and repeat-mistake rate is flat or rising. Wrong-layer attribution is the likely cause, such as editing the prompt when the tool schema is responsible. A team in that position should revisit layer attribution and validate the resulting change against the failure traces.

A stale loop combines collapsing compound velocity with session gap rate remaining high. The signals no longer show what is happening in production. Input-distribution drift or retrieval-corpus drift may have left trace capture or the golden task set out of date. The intervention is to refresh the dataset against current production traces before the next cycle.

An overfitting loop combines high one-shot rate on known task types with rising session gap rate on new traffic. RRSI shows how unregularized harness evolution can lead to benchmark-specific fitting, noise chasing, and a buildup of complexity. The appropriate response is to refresh the production-grounded dataset and test whether gains transfer, rather than accept familiar-task performance as proof of progress.

Why replay validation against historical traces is what separates a validated fix from a guess

A proposed fix can remove its target failure, leave it unchanged, or introduce a regression in adjacent behavior. Without replay, those outcomes remain indistinguishable before production. Historical failure traces give the team the evidence to test whether the behavior changed as intended. This makes replay a gate for compound velocity and a foundation for trustworthy one-shot measurement. A shipped edit is not yet a validated learning.

Attributable replay also requires changes that can be distinguished. If the system prompt, retrieval strategy, and tool schema change together and the evaluation improves, the result does not reveal which change caused the gain. That ambiguity prevents reliable reproduction and leaves a post-mortem without a clear explanation. Counting learnings from gains whose changes are bundled and unexplained leaves the team unable to tell which harness layer improved or where to look when the next failure comes.

Replay reliability is as important as replay coverage. A flaky suite teaches engineers to ignore CI failures, so the gate gets weaker even when the process still requires it. The loop then risks counting unproven edits as learnings and mistaking accidental passes for one-shot success. Historical replay must show that the fix works as intended before it ships, while later production traces test whether the failure comes back or new gaps appear. Every fix shipped without replay validation adds noise to the metrics, and enough noise makes the loop unreadable.

Using the metrics as an operational cadence, not a one-time audit

A single snapshot describes the agent’s current position. Successive cycles supply the evidence of acceleration or stagnation. A practical cadence is to review all four metrics at the close of every sprint and gate the next cycle on declining repeat-mistake rate and rising one-shot rate. If session gap rate stays high, the team should refresh the trace dataset before moving on. Compound velocity belongs in the same review so that falling signal supply is not confused with a lack of remaining failures.

That cadence should produce a versioned, attributable history of fixes. Each entry records the failure addressed, the layer changed, replay performance, and the before-and-after metrics. This history connects monitoring to shipped improvements without relying on recollection or the volume of merged changes. Over quarters, the team-level result is shorter time-to-fix, lower session gap rate, and a reversal in the balance of known and unknown failure modes. The proof of compounding is a production failure making the next failure cheaper to understand and less likely to recur.