Catching CrewAI Tool Exception Swallowing With Automated Return Value Regression Tests

Silent tool failure detection in production multi-agent pipelines requires assertions on what each tool returned. CrewAI can turn a tool exception into an error string that the agent reads and retries without surfacing a runtime failure. Automated return value regression tests expose that failure by checking recorded tool outputs for error markers, broken schemas, and unintended retries.

How CrewAI converts tool exceptions into opaque strings

Error executing tool: {e} is the recognizable artifact of CrewAI’s original exception-handling path. When a tool raised an exception, the runtime caught it, suppressed its propagation, and returned that string. The exception type and traceback were discarded at that boundary. The catch changed the representation of the failure. What had been a structured runtime exception became ordinary text returned from a tool.

CrewAI then injected that text into the agent’s context window as-is. The LLM received a flat English fragment with the distinctions between exception classes erased. The failure was turned into text before downstream consumers could inspect it. That behavior has since been patched with structured error information and retryability classification, but the original path remains the mechanism that a backward-compatible regression suite needs to recognize.

Why the agent loops silently instead of surfacing the failure

The returned error string does not necessarily stop the agent. CrewAI’s default behavior on a tool error is to add the string to agent memory and retry through the reasoning loop, bounded by max_iter. The crew thus seems to stay active as it uses up its iteration budget. In the trace, however, successive iterations can represent new attempts at the same broken tool call.

A transient network timeout should permit a retry, a parameter validation error should prompt an input correction, and an authentication failure should halt execution for reconfiguration. Under the opaque-string path, all three can produce the same looping pattern. Activity is not evidence of useful progress. Diagnosis depends primarily on whether the next tool attempt addresses the failure or merely repeats it, because a reasoning loop can continue without making that distinction visible to the operator.

Why standard logging and APM tools miss this failure class entirely

Standard APM can record HTTP 200 responses from the LLM provider throughout this loop. Each inference request succeeded, even though the tool operation that fed the next request failed. Latency and final crew output can reveal a slow or incomplete workflow, but they do not identify the swallowed exception. Issue #6262 notes that losing the original exception class prevents users from distinguishing whether to retry, fix input, or report the failure upstream.

CrewAI provides built-in tracing through TraceCollectionListener and tracing=True, while its full dashboard view requires a free CrewAI AMP account. Those facilities do not remove the need for a structured record of each tool return on each iteration. Even an OpenTelemetry tool span created through CrewAIInstrumentor records the returned representation of the raised exception, which may be the opaque string. Capturing text is not equivalent to classifying it. A log reader may recognize the fragment, but ordinary status-based monitoring does not turn it into an attributed failure.

Attribution becomes harder as the failure propagates through subsequent decisions. An incomplete crew result and the tool error that caused it can sit several steps apart in the trace. Monitoring the final artifact detects the consequence, leaving its origin unattributed. So what would a test that catches this actually assert on?

What a return value regression test asserts that a log-watcher cannot

A return value regression test evaluates the artifact delivered to the agent at the tool boundary. It can reject an output beginning with Error executing tool: or a structured response whose error field is true. Neither assertion requires recovery of the original traceback. The returned value supplies the evidence itself. Naming that assertion and making it fail the build turns an observable fragment into an enforceable contract.

Because LLM outputs are stochastic, exact snapshot equality is not a reliable general test for an agent workflow. Useful checks include JSON validity, required fields, types, constraints, semantic correctness, and the absence of error markers. A golden reference can then be a JSON Schema definition in place of a saved answer string. Schema validity alone does not establish semantic correctness, so the suite needs both where the task requires them.

This differs from an end-to-end evaluation in where it attributes the failure. A crew-level check asks whether the final result is acceptable, while a tool-level assertion identifies the returned artifact that broke its contract. Logs can expose the same text to a reader. The regression test adds a named condition that rejects it each time, without someone having to read the run.

Capturing tool return values from production traces to seed the test harness

Production traces hold the payloads that led to the swallowed exceptions. To capture them, instrument execution with CrewAIInstrumentor().instrument(tracer_provider=trace_provider) and verify that individual tool calls produce TOOL spans. Also verify per-task AGENT spans and the Crew.kickoff CHAIN parent with crew_agents and crew_tasks attributes. Coverage must be checked at the tool boundary. OpenInference and traceAI can auto-wrap crew, agent, and task boundaries without reliably capturing every individual tool execution.

Each recorded tool-call fixture needs the tool name and version, its JSON input payload, and its JSON output payload. Preserve an error string unchanged as the output value. The test needs what the agent actually received. Prioritize traces whose final crew output was degraded or incomplete, because those are likely to contain a tool return that explains the downstream failure.

For deterministic workflow replay, capture every user message, tool call, and tool response from the run. The replay engine can then use the recorded responses in place of calls to live APIs. Capture requires instrumentation of live execution, but replay assertions run offline. This keeps live API cost and delay out of the replay part of CI, while the test cases still come from production behavior.

Classifying tool return values into failure patterns before writing assertions

Swallowed exception strings form the first category. A trace may show a return beginning with Error executing tool: or a structured error envelope containing error, type, message, and retryable. An assertion for a healthy run should reject either form. An intentional failure-path test should validate the error contract.

Schema drift forms the second category. A recorded JSON return may lack a required field or contain a value that violates the tool’s expected return schema. The output may have no error marker. Its failure is a broken data contract, so prefix matching cannot detect it. A separate check of declared input schemas against the previous passing version can catch a breaking change before the agent runs into it.

Runaway retry artifacts form the third category. A trace may contain repeated returns for the same tool call across later iterations of one task. Repetition across tool returns is the relevant signal. The assertion should distinguish an unintended repeat from retries explicitly permitted by the task. Classifying fixtures this way lets new tools inherit suitable assertion templates instead of accumulating unrelated, tool-specific string checks.

Writing the regression tests: assertion patterns for each failure category

The error-marker check can begin with assert not raw_output.startswith("Error executing tool:"). For deployments using the structured format, parse the appropriate JSON return and apply assert result.get("error") is not True. These checks belong in healthy-run tests. A recorded trace containing a swallowed TimeoutError should make the relevant assertion fail, demonstrating that the suite detects the known failure during replay.

The schema check can use jsonschema.validate(json.loads(result.raw_output), golden_schema). Here, golden_schema sets out the required fields, types, and constraints that the tool’s return value must meet. Invalid JSON and schema violations both prevent the output from satisfying the contract. Keep the check of declared input schemas as a separate gate. Diffing it against the previous passing schema allows a breaking required-field change to fail CI before deployment.

The retry check counts tool-call occurrences within each task. A short assertion is assert call_count <= 1 or task_allows_retries, where the count is for the tool-call identity under test. The task specification supplies the exception to the rule. Without that qualification, a suite would mistake intended retry behavior for silent looping. The trace supplies the occurrences, so the check does not depend on interpreting the agent’s reasoning text.

All three patterns can run against recorded fixtures without live APIs. Store those fixtures as JSON so the tool identity, inputs, and outputs remain portable and diffable in code review. The failing assertion should identify the tool and relevant occurrence. That precision locates the violated contract. It does not, by itself, reconstruct an exception class discarded before capture.

Validating fixes against historical traces before shipping to production

A failing fixture establishes the defect, but replaying an unchanged response cannot establish that a tool implementation has been repaired. Recorded-response replay tests the assertions and downstream handling of that response. To validate a patched tool, run the recorded input against the patched execution path and inspect its new return value. The distinction is essential to an honest before-and-after comparison.

For a prompt or handling change, the suite can still use the recorded error response to check whether what happens next meets the task contract. When a tool changes, its new output must pass the return-value assertions that apply to it. Keep the original tool version label with the historical fixture so its provenance remains visible. Compare results per tool and iteration. A completed crew result does not establish which failing operation was corrected.

PR #6313 also has a contract for tests that are meant to exercise a failure path. Its structured envelope can represent an error with type set to TimeoutError and retryable set to true, allowing assertions on both structure and classification. That is different from accepting the envelope in a healthy-run test. Run the historical suite against the fix to catch schema mismatches in cases beyond the one repaired by hand. New production failures then become new fixtures, keeping the suite current beyond any one patch.

Keeping the harness current as CrewAI's error handling evolves

PR #6313, dated Jun 24, 2026, changed error returns from a plain string to a structured JSON envelope and kept the Error executing tool: prefix for backward compatibility. The harness must account for both representations. Retain prefix detection where that string is returned and inspect the error field where the envelope is used. Tests of the failure path should also check the exception classification and retryability.

Treat a CrewAI upgrade like an external API schema change. Run the full replay suite against the new version before deploying, because a changed tool return contract can break the workflow regardless of how the changelog describes it. Keep healthy-output checks separate from tests that are meant to trigger errors. Otherwise, a richer error response could look like proof that an operation was fixed just because it is easier to parse.

New failure patterns from live traffic should be added to the fixture collection automatically and become checks in the regression suite. The trace store supplies the observed input and return value, while the harness gives that evidence a repeatable test. Status codes and a final answer that sounds right cannot take the place of that record at the tool boundary. Production traces are the most direct account of what the agent received, and regression assertions turn that account into a reviewable engineering workflow.