Diff Formats for Reviewing Agent Harness Changes

Prompts, tool schemas, workflows, memory, and skills encode different behavioral contracts, so each needs a review format that exposes its particular risks. Matching the diff to the artifact is the foundation of reviewable harness engineering, with production-trace replay providing the final check on behavior.

Why the harness layer — not the model — is where agent behavior actually lives

The harness governs how a deployed agent uses its model. It controls context construction, tool interaction, orchestration, and verification, making it the primary lever for changing execution without changing model parameters. Stop Comparing LLM Agents Without Disclosing the Harness formalizes the Binding Constraint Thesis for long-horizon tasks across models with comparable frontier capability. The paper argues that, in this regime, harness configuration governs more performance variance than model choice, while evaluations systematically misattribute harness-level gains to model improvements.

A change to an instruction can change which language the agent uses, while narrowing a tool argument type can make a call that used to be valid return an error. Neither change requires a different underlying model. Both can change downstream behavior beyond the edited text, so the review surface must show how behavior changes as well as which file changed.

How each artifact type in a harness is structured, and why structure determines reviewability

The five artifact types encode different kinds of contracts. Prompts contain standing rules, reasoning procedures, and output constraints whose scope and ordering may be implicit. Reviewers need to judge whether a change to a clause changes an instruction or how it relates to another rule. Tool schemas set out typed contracts between the agent and an external API, with argument names, validation rules, and invocation formats. Their review must show whether a call that met the contract before would now break it.

Workflow review shifts attention from content to control. Their sequencing, branching, and retry logic determine which execution paths are reachable, so a reviewer needs to see changed transitions. Memory stores keyed observations, outcomes, facts, and procedures, but relevance and recency govern which entries reach context. Skills package reusable procedures into named, parameterized units whose callers depend on their inputs, outputs, and effects on state. These distinctions determine the useful unit of comparison, because a changed instruction, a narrowed field, a reordered step, a retrieval change, and a broken caller require different evidence.

What makes a diff format useful for reviewing a harness change

A useful harness diff satisfies three criteria, namely semantic visibility, scope legibility, and auditability. Semantic visibility means the reviewer can see how the change shown would affect behavior and reason about that effect. Scope legibility makes clear what changed and what remains outside the edit, including dependencies within the artifact. Auditability lets the reviewer approve, reject, or modify the proposal without reconstructing essential context elsewhere.

A unified diff of a long system prompt may contain every changed character while burying the decisive clause among surrounding lines. The reviewer then has to reconstruct both the instruction and its scope before evaluating the change. Code as Agent Harness treats inspectability as a first-class property together with executability and statefulness. Applied to review, that principle means a machine-readable representation is insufficient when the human reader cannot see the relevant behavioral delta.

Diff formats for prompt changes: surfacing instruction-level intent, not character-level noise

Prompt drift makes instruction-level review necessary. A small edit can degrade unrelated cases because prompts contain dependencies among role declarations, behavioral rules, output constraints, tool-use instructions, and fallback policies. A line-level diff can represent a neutral rewrite and a substantive policy change as the same deletion-and-insertion pattern. The textual operation is visible, but its significance is not. The useful review unit is therefore the instruction or clause, with enough context to establish its scope.

For teams reviewing prompts, section-level diffs separate changes to rules from changes to format, tools, or fallback behavior. Inline semantic annotation adds a closer view by highlighting clauses that were added, removed, or reworded while preserving surrounding text in a side-by-side comparison. An instruction-level diff can make the intent more explicit, showing that “respond only in English” was replaced by “respond in the user's detected language”. Even free-form prompts tend to cluster into stable functional blocks, so lightweight annotation at authoring time can provide this structure without treating every sentence as an independent rule.

Diff formats for tool schema changes: making contract violations visible at the field level

Tool schemas require a more rigid review surface than prose. A renamed required field or narrowed type can cause the model to emit invalid arguments, after which the agent may receive an error, loop, or fabricate a result. These are contract failures. A plain text diff of a JSON or OpenAPI schema may expose the edit, but the reviewer still has to infer its field-level significance. A structural diff makes that significance explicit.

The appropriate format renders the schema as a tree and labels added, removed, renamed, type-changed, and required-status-changed fields. Breaking-change annotations show which changes break the contract and which add an optional field, widen a type, or update a description. A call-site impact view can then show which recent tool-call patterns would fail under the proposed schema, linking the field change to what would happen in production. That connection is especially useful when model versions or tool APIs shift and the schema expected by the agent diverges from the interface encountered at call time.

Diff formats for workflow changes: reasoning about path and order, not just content

Workflow failures often originate in sequencing. Changing a condition or transition can open an execution path, close an existing one, or make a cycle reachable without substantially changing the steps themselves. A text diff of YAML, JSON, or code shows the definition edit but not its full control-flow consequence. The reviewer needs to reason about order and reachability. This includes newly reachable cycles in which the agent repeatedly calls itself or a tool with minor variations, a failure that standard monitoring can miss.

A graph diff exposes those changes as nodes and transitions, with added steps, removed edges, and changed conditions shown directly. A path enumeration diff offers a complementary comparison of reachable paths before and after the edit. For ordered workflows or directed acyclic graphs, an annotated step diff can show each step's incoming and outgoing connections, identifying a new incoming edge as a sequencing change. Path enumeration can be generated from the definition and scoped to the changed subgraph, keeping the affected paths visible without requiring a full-graph inspection for every small edit.

Diff formats for memory changes: showing what the agent will and won't recall

An entry may contain a prior observation, task outcome, retrieved fact, or reusable procedure, but its presence in the store does not establish when the agent will use it. Recency, relevance, and key matching govern what surfaces in context. A raw record diff shows that an entry changed without showing whether recall changed for a given task.

A retrieval-aware entry diff shows each entry that was added, removed, or changed alongside the query conditions that would cause it to be retrieved. A coverage diff shows this across a sample of recent task contexts, so reviewers can see which entries joined or left the retrieved set after the change. This connects memory review explicitly to trace-based validation without substituting for a full replay of the agent's behavior. Conflict detection adds another necessary signal by flagging contradictory entries, including the same key with different values or entries with overlapping scope. These contradictions can make behavior inconsistent without an explicit error, so the review surface needs to show both the competing content and when it may appear.

Diff formats for skill changes: reviewing the procedure, not just the code

Skills require review at the level of their behavioral contract. Their invocation signatures, expected preconditions, outputs, and effects on downstream state define what other harness components can rely on. A change to the implementation can change an output format or side effect while the skill's name and signature stay the same. A local code diff may not make that mismatch apparent. Workflow callers can consequently receive unexpected output and fail silently even though the invocation still looks valid.

A contract diff separates changes to the skill's inputs and outputs from changes within its implementation. A contract change is flagged as breaking and requiring caller review, while an implementation change that preserves the contract is lower risk. A caller impact diff identifies the workflow steps and prompt instructions that invoke the skill, exposing dependencies beyond the edited file. Where a prose description guides the model's decision to invoke a skill, a clause-level behavioral summary diff must also show whether the description broadens or narrows that decision.

Why validating a diff against production traces is the final check before shipping

A readable diff establishes what changed. A prompt or schema edit can appear correct in isolation while degrading an unrelated production case. Semantic visibility cannot show every way a change will interact with inputs during execution. Replay-based validation checks for those interactions by keeping other components the same and testing the changed artifact against recorded production inputs and tool responses. It asks whether the same run would have gone differently under the proposed harness.

Human diff review evaluates the semantics of the proposal, including its scope and likely consequences. Trace replay provides the automated check for regressions in known-good behavior and for whether the targeted failure is resolved. Neither gate substitutes for the other, because a readable change can fail in execution and a passing replay does not replace review of the behavioral contract. Versioning harness artifacts with git history allows any two harness versions to be compared and the proposed change to be replayed against a regression suite, connecting the observed failure to a validated fix.

Choosing the diff format as a first-class engineering decision

The practical next step is to audit the review workflow for each artifact type. That audit should establish whether prompt reviews expose instruction-level intent, schema reviews expose field-level breaking changes, workflow reviews expose altered paths, memory reviews expose retrieval consequences, and skill reviews expose contracts and caller impact. Where reviewers must reconstruct those signals from raw text, the review interface is leaving essential engineering work to inference.

The choice is between a representation that supports judgment and one that merely records an edit. Structured formats make behavioral contracts inspectable, artifact versioning preserves the basis for comparison, and replay tests the proposal against execution evidence. Treating those capabilities as review infrastructure gives harness engineering the same rigor expected of code review.