Judge Model Selection for Automated Replay Evaluation
Replay evaluation needs judges that can detect multi-step failures, not just score final outputs.
Darnell Fitch
Staff Writer
Darnell Fitch
Darnell spent several years embedded with developer-tooling teams at two venture-backed startups, where he developed a specialty in evaluation infrastructure and replay-based testing workflows. His reporting focuses on the systems and rituals teams use to assess changes before and after they ship.
7 stories
Replay Eval Coverage Metrics for Agent Harness Changes
Replay eval prevents silent agent failures by measuring test coverage across harness layers.
Darnell Fitch
Determinism Controls in Agent Replay Environments
Recording every interaction is the only reliable way to make agent behavior reproducible.
Darnell Fitch
Replay Eval Platforms for Production Agent Validation
Production agents fail silently in ways staging tests cannot detect or diagnose.
Darnell Fitch
Root Cause Attribution as a Ship Prerequisite
Unattributed fixes mask root causes and compound system liability in production deployments.
Darnell Fitch
Ship vs. Hold Decisions in Agent Harness Releases
Enterprise AI failures overwhelmingly stem from harness defects, not model reasoning itself.
Darnell Fitch
Evidence Thresholds for Shipping Prompt Changes
Prompt changes need the same testing rigor as code.
Darnell Fitch