Guides · 5 min read · October 27, 2026

Monitoring AI agents with OpenTelemetry: the warning signs, in KQL

A web service fails loudly: errors, timeouts, a red dashboard. An AI agent fails quietly. It takes more steps for the same answers, sends longer histories, repeats a tool call, or keeps hitting a limit. Every request still returns 200, and the bill grows.

This guide traces a Python agent with OpenTelemetry, sends the data to Azure Application Insights, and turns each warning sign into a KQL query. Then three of them become alerts. Every output below was captured from a real deployment in October 2026, in the agents package linked at the end.

Traces for one run, metrics for all of them

A trace is the story of one request. It's made of spans: timed steps, each nested under the step that started it. An agent's trace has one span for the run. Under it is a span for each model call and each tool call.

It works like a parcel's tracking page: every hand-off, with a time, under one tracking number.

Metrics tell you about all the runs together. Counters add up runs, tokens, tool calls and cost. Histograms keep a spread, such as steps per run. Each number carries labels, such as the stop reason or the tool, so you can split it.

Use OpenTelemetry's conventions for generative AI. Name the spans invoke_agent, chat and execute_tool, with details such as gen_ai.usage.input_tokens. Microsoft Agent Framework uses the same names, so your own agents and framework agents read alike.

Step 1: spans for the run, the model and the tools

An agent run is often a generator: it pauses at every event, and it may wait minutes for a person to approve an action. A plain with block can't stay open across those pauses safely. So start the run's span by hand, and make each model or tool span its child:

class RunTrace:
    def __init__(self, run_id: int) -> None:
        self.span = tracer.start_span("invoke_agent fieldnotes", attributes={
            "gen_ai.operation.name": "invoke_agent", "gen_ai.agent.name": "fieldnotes",
            "fieldnotes.run_id": run_id})
        self.parent = trace.set_span_in_context(self.span)

    @contextmanager
    def model_call(self, model: str):
        with tracer.start_as_current_span(f"chat {model}", context=self.parent, attributes={
                "gen_ai.operation.name": "chat", "gen_ai.request.model": model}) as span:
            yield span

The tool span is the same shape, with gen_ai.tool.name. After each model call, set the token counts on its span. When the run ends, set the stop reason on the run's span and end it.

configure_azure_monitor from azure-monitor-opentelemetry sends it all to Application Insights. It also traces every request to your API and every HTTP call, with no extra code. Until it runs, every instrument is a no-op, so your tests run unchanged.

What one trace shows

One question to the agent: "Is checkout-api healthy?" This query lists every span in the newest answered run:

let run = dependencies
    | where name == "invoke_agent fieldnotes"
        and tostring(customDimensions["fieldnotes.stop_reason"]) == "answered"
    | top 1 by timestamp;
union dependencies, requests
| where operation_Id == toscalar(run | project operation_Id)
| project timestamp, name, ms = round(duration, 0),
    tokens_in = tostring(customDimensions["gen_ai.usage.input_tokens"]),
    tool = tostring(customDimensions["gen_ai.tool.name"])
| order by timestamp asc

The result, without the timestamps and two small FastAPI spans:

name                             ms    tokens_in  tool
POST /api/agent/ask              2694
invoke_agent fieldnotes          2681
chat chat                        1448  505
POST /openai/v1/responses        1440
execute_tool get_service_status  0                get_service_status
chat chat                        1217  548
POST /openai/v1/responses        1211

The request took 2.7 seconds, and the two model calls took 2.6 of them. The tool took under a millisecond. In an agent, the model is almost always the slow part. The second call read 43 more tokens than the first: the whole history again, plus the tool's result. That growth is a warning sign, in one run.

The eight warning signs

Each sign has a number that moves first:

Warning sign	What it may mean
Tokens per task rising	Longer histories, bigger tool results, or more steps
Input tokens per step rising	Histories grow within runs, or a run is looping
Steps per task rising	The agent works harder for the same answers
More runs ending on a limit	Agents are getting stuck
Repeated identical tool calls	The earliest sign of a loop
One tool's errors or latency rising	That tool, or what's behind it, is failing
Cost per task or per day rising	Any of the above, or simply more use
More approvals denied or expired	Wrong actions proposed, or nobody sees the requests

Each row is one KQL query over your metrics. How runs ended, for example:

customMetrics
| where name == "agent.runs"
| extend stop_reason = tostring(customDimensions.stop_reason)
| summarize runs = sum(valueSum) by stop_reason
| extend share = round(100.0 * runs / toscalar(customMetrics | where name == "agent.runs"
                                             | summarize sum(valueSum)), 1)
| order by runs desc

We made the agent misbehave on purpose: a tight token budget for four questions, and a tool switched to fail for three. Then we ran all eight queries. Part of what came back:

── 04-stop-reasons ──
  stop_reason  runs  share
  answered     8     66.7
  token_limit  4     33.3

── 06-tool-errors-and-latency ──
  tool                calls  errors  error_rate  p95_ms
  get_service_status  10     3       30          9
  list_tickets        2      0       0           8
  read_note           5      0       0           8

── 07-cost ──
  day                   tasks  cost_usd  cost_per_task
  2026-10-27T00:00:00Z  12     0.0052    0.000433

A third of the runs ended on token_limit: those were the four. The failing tool stands out at 30%, and every other tool shows none. In production, more than a few percent of runs ending on a limit deserves a look.

Alerts that ask for you

Queries answer when you ask. Alerts ask for you. Three warning signs make good alerts: one tool's error rate, runs hitting a limit, and daily cost. Our tool-error rule, in Terraform:

  criteria {
    query                   = <<-KQL
      customMetrics
      | where name == "agent.tool_calls"
      | summarize calls = sum(valueSum), errors = sumif(valueSum, tostring(customDimensions.outcome) == "error")
          by tool = tostring(customDimensions.tool)
      | where calls >= 5
      | extend error_rate = errors / calls
    KQL
    time_aggregation_method = "Maximum"
    metric_measure_column   = "error_rate"
    operator                = "GreaterThan"
    threshold               = 0.2
  }

Split by tool, on purpose. Our first version added all tools together. One failing tool among healthy ones stayed under 20%, and the alert never fired. The calls >= 5 stops one unlucky call from paging anyone. After the misbehaving runs, both alerts fired on their own:

{ "fired": "2026-10-27T16:48:51Z", "rule": ".../scheduledqueryrules/agent-tool-errors", "state": "Fired" }
{ "fired": "2026-10-27T16:42:06Z", "rule": ".../scheduledqueryrules/agent-limit-hits", "state": "Fired" }

Errors you'll hit

Traces arrive, but no metrics. The exporter sends metrics once a minute. A short run, such as a CI job, ends before the first send. Set OTEL_METRIC_EXPORT_INTERVAL lower, and call force_flush() on the providers when the app stops. Our first run lost every metric this way.
Nothing shows for a few minutes. Telemetry takes two or three minutes to arrive. Wait before you decide it's broken.
The teardown takes half an hour. Application Insights creates a "Failure Anomalies" alert and an action group of its own, outside Terraform. Deleting the resource group waited on them for 34 minutes. Delete those two first.

The package below builds all of this into a real agent, then adds injection defences and an evaluation gate on top.