Cost and Latency Alerting on Production Agent Traces

Cost and latency alerting on production agent traces works when a threshold breach identifies the responsible span. The compounding improvement loop from production agent observability is to capture real failures, attribute them to the right layer, revise the harness, and validate the change against historical traces before shipping. Each validated change turns production evidence into a repeatable improvement in agent behavior.

Why cost and latency signals in agent traces demand a different mental model than traditional APM metrics

A cloud invoice arrives days after a prompt change doubled completion length, but the error-rate dashboard still looks healthy. The agent returned well-formed answers with HTTP 200 responses and no stack traces. Nothing in binary pass/fail monitoring registered the extra spend. This is a silent blowout, where a technically successful response conceals economic damage.

Traditional APM records request duration and leaves cost to billing, but an agent request fans out into a hierarchy of operations. Model calls, tool calls, retrieval steps, sub-agent handoffs, and retries each contribute to the run. The root duration measures the time for the whole execution tree, beyond a single atomic operation. Token spend accumulates within that tree, while the planning decision that caused it may sit above the generation spans where it is recorded.

Latency follows the same pattern because delays compound across hops. P50 can conceal a long tail, and P99 of many seconds remains invisible when monitoring shows only averages. A tool sequence or retry loop that is not needed can make the whole run slow even when each call looks ordinary. A threshold breach on the total is therefore a symptom.

Tools such as Moda ingest production traces and pinpoint the span responsible for a cost or latency overage, making the alert a diagnostic starting point before the invoice arrives. That attribution matters because the remedy depends on whether the excess came from generation, planning, or tools. Alerting only on the trace envelope is like alerting on total server CPU without knowing which process is responsible.

The two foundational signals and the span hierarchy that makes them meaningful

OpenTelemetry GenAI semantic conventions define two foundational histogram metrics, which are gen_ai.client.operation.duration for latency in seconds and gen_ai.client.token.usage for separate input and output token counts. Export both so investigations can connect speed with consumption. As of v1.41, nearly all gen_ai.* attributes carry Development badges. Their names can change without a major version bump, so instrumentation must be a versioned artifact.

Build one span for each operation, with the agent invocation as a whole at the root, planning steps below it, and model calls and tool invocations below those steps. A trace ID that stays the same links the whole run, while span IDs and parent span IDs keep track of its hierarchy across services. Capture inbound requests, model generations, embedding calls, retrieval and reranking, tool calls, memory reads and writes, and the final response as distinct spans. Attach prompt context, token counts, latency, and status where applicable so investigations can distinguish missing context, tool failure, and generation problems.

For each model generation, record provider and model to identify the service used, operation name to classify the work, and finish reason to explain how generation ended. Record latency for elapsed time, status for outcome, and input, output, and total token usage for consumption. Estimate cost using the applicable provider price for that account and date, retaining the price version for billing reconciliation. MCP tracing, added in OTel v1.39, supplies mcp.method.name for the method, mcp.session.id for the session, and mcp.protocol.version for the protocol version. When instrumentation can reliably detect an existing execute_tool span, those attributes enrich it.

Application SDKs and an AI gateway provide complementary capture points. SDKs see prompts, retrieval context, and reasoning steps, while the gateway sees requests reaching model providers regardless of framework. Gateway attribution is therefore the most reliable capture point for complete model spend across teams. With both signals captured throughout the hierarchy, the trace becomes a durable production record that Moda can analyze for cost creep and compounded latency, then use to validate a fix.

What each layer of the trace can tell you about cost and latency responsibility

A growing root duration with ordinary model-call durations points first to the number of operations. In a runaway loop, steps per run and token cost rise without a corresponding increase in successful completions. Repetition drives the expense. That pattern points to planning and workflow decisions as the place to look, even though the final charge sits in a model-call span.

Tool failures leave a different signature because a run can complete at normal cost despite using invalid data. A tool returns an error string, the model treats it as valid input, and the final answer appears successful. Tool-call success rate then diverges from overall run success rate. Logging tool inputs, outputs, and outcomes as span events makes that divergence visible and locates the failure beneath the apparently healthy completion.

If an endpoint changes customer_id to account_id, the agent may retry until the loop times out and then fabricate an answer. CI can stay green because its mock still returns the old payload. Generation reveals the fabricated answer, while the tool-call spans reveal the initiating fault. Treating that sequence only as a model problem leaves the broken contract untouched.

Context drift instead appears as prompt-length growth across sequential spans. Early steps remain compact while later steps carry bloated context windows, increasing end-to-end latency without one conspicuously anomalous call. Aggregate token spend to the root for budget monitoring, but retain breakdowns by model, tool, and planning step to distinguish that growth from a prompt change that lengthens completions. These diagnostic signatures turn an overage into a specific candidate improvement, such as a prompt revision, tool elimination, or workflow restructuring.

The metrics worth tracking per run and why task-level cost is more honest than raw spend

Cost per successful task is more honest than raw cost per run because a cheaper agent that fails more often is not actually cheaper. Divide total spend by the number of completions that evaluation says succeeded. Across multiple steps, end-to-end completion degrades even when per-step success looks acceptable. Single-turn evaluation misses that compounding effect, so task success must be the denominator.

Track steps per run as an early signal of runaway loops or degrading planning. A rising average can expose repetition before token spend visibly spikes. Track tokens and cost per run with a model breakdown to show whether planning is unnecessarily routing work to a more expensive provider.

Keep time to first token separate from end-to-end latency because streaming responsiveness and total duration describe different service properties. Derive task-level success from evaluation because only evaluation establishes whether the answer was correct. Together, these signals distinguish a responsive agent from one that finishes the task reliably.

In the Google SRE framework, SLIs are measures of service level expressed as numbers, with care taken to define what they measure. Agent services need that discipline with cost treated as a first-class SLI and errors separated across caller, policy, provider, and gateway fault domains. Exclude caller-caused errors from the service error-rate denominator. Attribute cost and errors to teams, customers, applications, and API keys so the metrics identify an owner as well as a trend.

How to set alert thresholds that fire on the span responsible, not just the trace envelope

Set budget burn alerts at a high percentage of allocated budget and scope them to a team or virtual key. For latency, alert on a P95 SLO breach. The SLO must be set per operation type because a tool call and a multi-step planning run have different baselines.

Set the early loop threshold where steps per run go above the run-type baseline. Pair it with a separate threshold for when tool-call success falls below baseline, even if overall run success still looks healthy. Set a conservative error-rate threshold and scope it by fault domain so caller, provider, and gateway problems do not all page the same owner. Monitor cost-per-successful-task deviation to catch a prompt change that holds raw spend flat while reducing answer quality.

Attach provider, model, team, operation type, and fault-domain labels to alert rules. Carry the responsible span layer into the alert so the engineer can distinguish a generation overage from repeated planning or failed tools without first searching dashboards. The total establishes impact, while the span context establishes where to investigate.

Bifrost's Prometheus endpoint exposes per-request metrics with provider, model, and method labels, with team and virtual key available as governance labels on supported metrics. It also records retries and key rotations. A fallback can keep the user experience the same while resilience metrics and budget alerts show the trouble behind it. This gateway view complements the application trace rather than replacing the context needed to explain why the agent made those requests.

Route alerts to the channels on-call engineers already use, including Slack, PagerDuty, and webhooks, and configure cooldowns to keep repeated notifications from becoming a storm. Control alert fatigue through layered severity while retaining span detail. Treat steps-per-run and tool-call success deviations as low-urgency early warnings. Make P95 latency SLO breaches and budget burn high-urgency alerts, while reporting cost-per-successful-task deviations in a scheduled digest.

How do teams validate a harness fix before shipping better agents?

A prompt edit, tool schema correction, routing change, or memory update should be tested against the historical traces that exposed the problem. Without those tests, the change reaches users before there is evidence that it cuts the cost or latency it was meant to reduce. Those traces also preserve the failure pattern, making the validation target more concrete than a general request to improve the agent.

Harness components can change while model weights stay the same, so failures in production can guide a step-by-step effort to improve the harness. Prompts, tool schemas, routing logic, and memory can be changed and tested by replaying the same recorded cases. Deterministic replay is particularly useful for tool calls and schema contracts. The comparison must establish both that the responsible pattern changed and that the task still succeeds.

Unit evals on every CI run replay tool calls deterministically, check schema contracts, and catch plumbing regressions, making them the first check for a tool schema fix. LLM-as-judge evaluation per PR or release checks whether a prompt change reduced answer quality despite keeping cost flat. Continuous production trace sampling catches distribution shift and real-world drift that neither earlier layer sees, with sampling rate controlling its cost. Replaying a golden trace set against current production on a schedule gives an early warning of drift caused by model updates, prompt edits, and tool schema changes.

Moda closes this loop by ingesting production traces and attributing failures to prompt, tools, workflow, memory, model, or product logic. It generates reviewable harness changes and validates them against historical traces before recommending what to ship.

Replay adds engineering overhead, but that work is front-loaded and bounded. Skipping it risks reopening the same failure pattern or introducing another one that must be diagnosed again. The compounding loop comes from turning each production failure into evidence for the next prompt or logic improvement, then testing that improvement before release. When that loop is trace-driven, harness iteration becomes as rigorous and repeatable as code review.