Agent Observability Metrics That Actually Predict Failure

Most agent dashboards are ported straight from web services: request rate, error rate, average latency, tokens per day. All four can look perfectly healthy while the agent is failing half its tasks, because none of them are measured over the thing that matters.

An agent does not serve requests. It attempts tasks, using a variable number of requests to do so. Get the denominator wrong and every metric you build on top inherits the mistake.

Metrics and traces answer different questions

A trace tells you what happened in one run. It is the tool you reach for once you know something is wrong. Metrics tell you which runs to look at, and whether the situation is getting worse.

That division decides what belongs where. High-cardinality detail — arguments, results, the exact model output — goes in traces. Low-cardinality aggregates that you want to alert on and plot over weeks go in metrics. Trying to put run identifiers or user identifiers into metric labels produces a cardinality explosion and a large bill.

The two need to be joinable, though. Every metric dimension should also exist as a span attribute, so that when the error rate for one tool spikes you can immediately pull the traces behind that exact series. If your metrics cannot lead you to a trace, they are a wall chart. Tracing a loop you cannot reproduce covers the other half of this pairing.

Choose the denominator before the metric

Per-call metrics are almost always the wrong unit for an agent. A model call succeeding tells you nothing about whether the task succeeded, and a cheap model that needs three times as many calls looks cheaper on every per-call chart.

Define the task as the unit. One task begins when a user or a queue hands the agent a goal, and ends when it completes, is abandoned, hits a cap or errors out. Every headline metric is then per task: cost per task, steps per task, wall-clock per task, tool errors per task.

Keep per-call metrics for infrastructure questions — provider latency, rate limit rejections, HTTP error codes — because those genuinely are per-call phenomena. Just never let them be the top-line health signal.

The metrics worth defining

Task resolution rate. The fraction of tasks that reached a verified successful outcome, not the fraction that returned without throwing. Verification usually means a test suite passing, a schema validating, or a human accepting. If you cannot define success programmatically for most tasks, that is the first problem to solve, because nothing else on this list is interpretable without it.

Steps per task, as a histogram. The mean is nearly useless; the tail is the whole signal. Healthy runs cluster tightly and stuck runs sit in a long right tail, so the ninety-fifth and ninety-ninth percentiles are what move when quality drifts. A rising p95 with a flat mean is the classic early warning.

Cost per resolved task. Total spend divided by successful outcomes, not by attempts. This is the only cost metric that survives a model change honestly, because it prices retries, extra steps and failed attempts into the number. Why agent costs are unpredictable explains why the per-token view misleads here.

Tool error rate, split by tool. Aggregated across tools this is meaningless. Per tool it is one of the highest-yield metrics you can have, because a single tool failing a third of the time silently inflates step counts and spend across every task that touches it.

Termination reason. A counter labelled with completed, step cap, cost cap, timeout, unrecoverable error, or human abort. Watching the mix shift is more informative than any single rate. Step-cap terminations climbing from two percent to eight percent is a clear signal even if resolution rate has barely moved yet.

Repeated-call rate. The fraction of runs containing the same tool called with identical normalised arguments more than once. It is cheap to compute and correlates strongly with runs that are going nowhere, which makes it a good trigger for both alerting and trace sampling. Detecting agent loops covers the detection mechanics.

Context size at terminal step. A histogram of how full the window was when the run ended. If the distribution is piling up near the ceiling, your compaction is failing and quality problems are downstream of that rather than of the model.

Histograms, not averages

Agent workloads are heavily skewed. A typical distribution has a dense cluster of short cheap runs and a thin tail of runs that cost fifty times as much, and that tail is where both the money and the failures live.

An average sits in the empty space between the two modes and describes neither. Record steps, cost, duration and context size as histograms with buckets you have actually chosen — logarithmic buckets work well when the range spans two orders of magnitude — and read them at p50, p95 and p99.

The gap between p50 and p95 is itself a health signal. A widening gap means the tail is getting worse while the common case stays fine, which is exactly the pattern that a mean-based dashboard reports as no change.

Dimensions that earn their cardinality

Every label multiplies your time series count, so add them deliberately. Four usually pay for themselves: the model identifier actually served, the prompt version, the task type or entry point, and the tool name on tool-scoped metrics.

Those four let you answer the questions that come up in practice. Did resolution rate drop for every task type or one. Did it drop when the prompt version changed. Is the new model cheaper per resolved task or only per token. Carrying the prompt version here is why prompt versioning pays off operationally rather than just tidily.

Resist adding user, repository, session or run identifiers as labels. Those belong on spans, where high cardinality is expected and priced accordingly.

Alerting without noise

Alert on percentiles and ratios, never on raw counts, and give every rule a window long enough that a handful of unusual tasks cannot trip it. Agent traffic is bursty and small samples are wildly noisy.

Three rules cover most real incidents. Resolution rate down by more than a few points against the same window last week. Step-count p95 up by more than a third. Step-cap or timeout terminations above a fixed share of tasks. Everything else is better as a dashboard you look at during a change than as a page at three in the morning.

Start with the denominator and one metric: cost per resolved task, plotted weekly. If that number is not defined, no amount of additional instrumentation will tell you whether the agent is getting better or just busier.