Ship vs. Hold Decisions in Agent Harness Releases

Most agent harness changes ship on a hope: the run passed in staging, so the team pushes it and watches the dashboards for a while. That's a coin flip dressed up as process. Research compiled at medium.com/@adnanmasood puts a number on it: 65% of enterprise AI project failures trace back to harness-level data defects, not to the model's reasoning. If that's where the failures actually live, a harness change deserves the same structured gate a code release gets, with defined inputs and a real verdict at the end. Not a shrug and a deploy button.

What the harness actually is and why each layer is a distinct release surface

The harness is everything around the model except the model. Tool execution, memory storage, state persistence, context management, error recovery, lifecycle control: all of it sits outside the LLM's weights, in code your team owns and ships. The model reasons. The harness acts. That split matters because it means harness changes are independently releasable, and independently breakable, from anything happening inside the model itself.

Break the harness down and you get several distinct pieces, and each one is its own release surface with its own way of breaking. System prompts and prompt files tell the model what to do at every invocation. Tool interfaces and schemas are the contract, name, description, and parameters, stamped into every request the model sends. Memory covers context compaction, retrieval strategy, summarization rules. Workflows and lifecycle control handle step sequencing, loop guards, branching logic, the stuff encoded in program flow instead of in a prompt. Hooks are the pre- and post-call scripts that enforce rules the model can't be trusted to remember on its own. Evals and observability form the out-of-band test suite that decides whether an agent is ready to go live, and whether it should stay live.

Fix a prompt and your tool schemas haven't been touched at all. Tighten a workflow guard and your memory policy is exactly as fragile as it was yesterday. A change to one layer doesn't validate the others, so each one needs its own evidence before it ships. This echoes the old Sculley diagram from classic ML infrastructure writing, where the model code itself was a tiny box buried inside a much bigger system of plumbing. The LLM call is the small part. The harness is where most of the engineering, and most of the risk, actually lives. MongoDB's agent harness series makes the same point.

Nothing on that list is minor, including something as plain as tool count. Vercel found this the hard way: cutting roughly 80% of an agent's available tools and replacing them with a single bash-execution capability took success rates to 100%. Tool count is a release variable with a measurable, testable outcome, and it has to be tuned with the same care as anything else you ship.

Why the visible symptom rarely points to the real cause of harness failures

The step where an error appears is often not the step that caused it. That's the core diagnostic trap in agent systems, and it's why teams that chase the loudest error message usually end up fixing the wrong thing.

Research from AgentTether found that one root error triggers 3.2 further violated checks on average, and that 76% of failures violate more than one check. The part of a trace with the most visible errors is usually just where the damage piled up, not where it started. A team that patches the densest failure cluster is patching inheritance, not origin, and the fix won't hold past the next release.

Agents also fail in ways specific to their structure. Several recurring failure modes appear across agent systems: tool misuse, context loss, goal drift, retry loops, cascading errors across multi-agent systems, and silent quality degradation. That last one deserves the most attention, because it's the hardest to catch. An agent can produce an answer that reads perfectly fine even though the tool calls underneath it are contract-invalid: wrong field names, wrong types, silently accepted. Nothing throws an error. Completion rates just drift down over weeks, and nobody can say why until someone opens the trace and actually looks.

AgentCompass (2025) sorts the failures most relevant to hold decisions into categories that include workflow and task gaps (goal drift, context loss, redundant actions, orchestration failures) and reflection gaps, where an agent fails to verify its own outputs or reasoning. Separately, the MAST taxonomy out of UC Berkeley attributes 41.8% of failures to Specification Issues and 36.9% to Inter-Agent Misalignment. Both are harness-layer problems. Neither is a model capability problem, and treating them like one wastes a fix cycle.

Reading "task failed" and stopping there tells a team nothing. Without tracing the failure to a specific layer, there's no principled way to decide what to ship next.

The three failure categories that most commonly force a hold

Tool schema drift is when a tool's expected input shape changes, taking forms such as a renamed field, a new required property, a stricter validator, or a different nullability rule. The insidious part is that calls can succeed and still return an unexpected structure without ever throwing an error. Completion rates drop, retries climb, output quality degrades, all quietly. An agent can pass every happy-path eval in staging and then fall apart in production because one argument changed from customer_id to account_id somewhere upstream. Most eval suites check whether the final answer sounds right. Almost none check whether the tool call that produced it was contract-clean. They miss this as a result. Any prompt or workflow change that touches tool invocations needs schema contract validation against the current production tool definitions before it ships. No exceptions.

Prompt ambiguity means telling a model to "remove outdated entries" without saying what "outdated" means, and watching it interpret that freely, then carry the interpretation into every action downstream. Long instruction sets, and instruction sets with internal conflicts, reliably degrade in production even when they clear a controlled eval without trouble. A prompt change that introduces an undefined term, drops a scope constraint, or adds an instruction that conflicts with an existing one shouldn't ship without a trace replay proving the new version actually handles the ambiguous cases the old one botched.

Workflow loops and unbounded execution describe an agent with no step limit, no token budget, no wall-clock timeout, no natural stopping point built in. One ambiguous task can burn unbounded compute. Cost spikes generally trace to one of three places: prompt length creeping up over time, retry storms where failures trigger their own re-execution loops, or context window inflation as the underlying knowledge base grows. Research from openlayer.com shows tool calling itself fails at meaningful rates in production, varying by model size and task complexity, with some agent workflows failing far above that baseline. Any workflow change that removes or loosens an iteration limit, or opens a new retry path, needs to show bounded execution across the production trace set before it goes near a release.

Why attribution must precede the ship/hold decision

Knowing a run failed tells a team nothing about which layer to change, which fix to test, or whether a proposed patch even touches the actual cause. Skipping past that gap is how teams end up shipping fixes for symptoms and calling it done.

Structured attribution, per research at arxiv.org/pdf/2606.06324, runs through a sequence of stages. Symptom localization identifies what failed, reading external eval results alongside the diagnostic evidence in the trace. Evidence backtracking follows data-flow and control-flow links backward through earlier steps, producing a ranked list of candidate responsible steps. Subsequent stages adjudicate among candidates and map the responsible step to a specific harness layer: prompt, tool, workflow, memory, or lifecycle control.

Some approaches propose building structured causal graphs and running counterfactual reasoning to find root causes in multi-agent systems, attributing them to specific agents and specific actions. The method's real value is separating the step that caused the failure from the steps that just inherited its effects downstream, which is exactly the distinction the cascade numbers above make necessary. Folding AgentFail's fine-grained failure taxonomy into LLM-guided localization reportedly improves identification accuracy by around 15% (2026), a meaningful gain when the alternative is guessing.

OpenRCA's analysis of 1,675 agent runs across five different LLM models found that Hallucination in Interpretation (71.2%) and Incomplete Exploration (63.9%) hold at comparable rates across every model tested, regardless of capability tier. A failure rate that doesn't move whether the model is strong or weak points to a harness failure wearing a model's face. It's a harness failure wearing a model's face.

None of this is complete without an output. Attribution has to produce a specific layer, a candidate responsible step in the trace, and a proposed change addressing that attributed cause. Anything short of that is a guess with extra steps.

What production traces make visible that evals alone cannot

A production trace records every tool selection, every tool argument, every model response, every memory read and write, every state transition, every branch the agent took. Enough detail to rebuild what happened, in what order, with what inputs and outputs at each step.

A trace shows three things a synthetic eval can't reproduce. It captures the actual output, the reasoning path behind why a particular branch got chosen, and the specific memory fragments and tool calls that fed the decision. Synthetic evals test the cases a team thought to write down in advance. Production traces reveal, in real usage, the cases nobody thought of. These include the ambiguous instruction that only breaks with one customer's specific phrasing, the tool argument the model assembled slightly wrong, and the retry loop that appears only once real task variety hits the system.

Hooks running at specific lifecycle points, before a tool call, after a file edit, before a commit, capture enforcement evidence. They show whether a constraint was actually applied, not just whether the model was told to apply it, which is a different thing to know. This is what observability is actually for: evals run after the fact, reading from the trace data observability collects, applying success criteria over and over across time. Without the trace as raw material, there's nothing for the eval to evaluate.

Traces show what happened. They don't attribute cause on their own, and they don't tell you whether a proposed fix works. Those two jobs belong to the attribution process above and the replay step below.

Replay validation: how to confirm a fix holds before shipping it

A proposed harness change gets tested against the specific production traces where the attributed failure actually occurred. That distinction is the whole point of replay, and skipping it is how teams ship a fix that looks clean and isn't.

The process runs in four moves. First, pull the trace set where the failure appears, meaning traces exhibiting the attributed failure signature at the attributed layer. Second, apply the candidate change (a revised prompt, an updated schema, a tightened workflow bound, a modified memory policy) in isolation, nothing else touched. Third, re-run those traces through the modified harness and compare outcomes both at the attributed failure step and at the downstream steps that previously inherited the damage. Fourth, confirm the failure no longer appears at the root, and confirm the downstream violations are actually resolved rather than just relocated somewhere else in the trace.

A fix that clears the attributed failure but opens a new one at a different layer isn't shippable. Replay has to cover the entire trace, not just the one step under investigation, or the team is trading one hold-worthy problem for another it hasn't spotted yet.

Research on agentic workflows has found that correcting a single root-cause mistake can flip an entire failing trajectory into a successful one. Targeted fixes on attributed causes become visible as trajectory-level gains, and that's what replay output actually measures, which is what makes replay worth doing at all.

Passing replay means four things hold at once: the attributed failure is gone from the trace set where it originated, no new failure signature appears in traces that were previously clean, execution stays bounded with no new loops or retry storms, and tool call contract validity checks out in every trace touching the modified tool.

Replay confirms behavior. It doesn't replace a human actually reading the diff, the new prompt text, the updated schema, the new loop guard, before any of it ships. Replay tells you the outcome improved. It doesn't tell you the change was the architecturally right call.

The ship/hold decision matrix: mapping failure layer to release criteria

The decision is a gate, and gates have defined inputs. No attribution, no replay evidence, no human review of the diff: the default answer is hold, not "ship it and keep an eye on the dashboard."

A prompt change needs attribution linking the failure to a specific ambiguous or conflicting instruction, replay on the traces where that instruction was mishandled, confirmation that previously passing traces don't regress, and human review of the actual revised text. Hold if replay improves the attributed cases but regresses adjacent ones, or if the change was written without a failure trace motivating it in the first place.

A tool schema change needs contract validation confirming the new schema matches what's actually running in production, replay on traces with schema-related failures, and confirmation that tool call validity holds across the entire replay set. Hold if a single trace that was previously contract-clean turns contract-invalid under the new schema.

A workflow or lifecycle change needs bounded execution confirmed across the full replay set (no new loops, no new retry storms), attribution tying the change to a specific task-gap or goal-drift failure, and confirmation that step limits and timeouts stay enforced. Hold if replay reveals any path reaching an unbounded state that didn't exist before, or if cost signals in the replay traces exceed the pre-change baseline.

A memory or context policy change needs replay on long-running and multi-session traces, confirmation that context compaction doesn't evict information a later step depends on, and attribution tying the change to a specific context-loss or goal-drift failure. Hold if any previously successful long-running trace now fails at a step that needed context the new policy would have summarized away or dropped.

When a candidate change touches more than one layer at once, each layer's criteria apply on its own terms. A change that clears prompt replay but fails tool contract validation is a hold on the entire change, full stop. And the default, always, is hold: a change proposed speculatively, with no attributed failure trace behind it, has no business shipping no matter how confident the engineer behind it feels.