OpenTelemetry E2E Setup Guide for AI Agents
This guide shows how to set up end-to-end OpenTelemetry observability for AI agents, from local tracing to production export. It covers the OpenTelemetry Collector, Python and Node.js instrumentation, agent-specific spans, LLM and tool-call tracing, validation, and production recommendations.
1. What You Are Instrumenting
For an AI agent, the most useful trace structure is a top-level agent run with child spans for every meaningful operation.
agent.run
|-- agent.plan
|-- llm.chat
|-- tool.call: knowledge_search
|-- retrieval.query
|-- memory.read
|-- llm.chat
|-- memory.write
`-- agent.finalize
At minimum, trace these operations:
agent.run : one full user request or autonomous task
llm.chat : each model call
tool.call : each tool invocation
retrieval.query : vector search, database lookup, or document retrieval
memory.read : agent memory lookup
memory.write : agent memory update
agent.handoff : transfer to another agent or human
agent.finalize : final response construction
2. Recommended Architecture
Agent Application
|
v
OTLP traces, metrics, logs
|
v
OpenTelemetry Collector
|
v Exporter
v
Observability Backend
Common observability backends include:
Jaeger
Grafana Tempo
Honeycomb
Datadog
New Relic
Azure Monitor
AWS X-Ray
Google Cloud Trace
Elastic Observability
For local development, you can start with the OpenTelemetry Collector plus a logging exporter. For production, send data from the Collector to your preferred backend.
3. Local OpenTelemetry Collector Setup
Create a file named otel-collector-config.yaml:
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
grpc:
endpoint: 0.0.0.0:4317
processors:
batch:
exporters:
logging:
verbosity: detailed
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [logging]
Run the Collector with Docker on Windows PowerShell:
docker run --rm -p 4317:4317 -p 4318:4318 \
-v "${PWD}/otel-collector-config.yaml:/etc/otelcol/config.yaml" \
otel/opentelemetry-collector:latest
Run the Collector on macOS or Linux:
docker run --rm -p 4317:4317 -p 4318:4318 \
-v "$PWD/otel-collector-config.yaml:/etc/otelcol/config.yaml" \
otel/opentelemetry-collector:latest
The local OTLP endpoints are:
gRPC: localhost:4317
4. Python Agent Setup
Install dependencies:
pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp
Create telemetry.py:
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
def configure_telemetry() -> None:
resource = Resource.create({
"service.name": "agent-service",
"service.version": "1.0.0",
"deployment.environment": "local",
})
provider = TracerProvider(resource=resource)
exporter = OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
Instrument the agent:
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
from telemetry import configure_telemetry
configure_telemetry()
tracer = trace.get_tracer("agent-service")
def run_agent(user_input: str) -> str:
with tracer.start_as_current_span("agent.run") as span:
span.set_attribute("agent.name", "support-agent")
span.set_attribute("agent.version", "1.0.0")
span.set_attribute("agent.input.length", len(user_input))
try:
with tracer.start_as_current_span("agent.plan") as plan_span:
plan = create_plan(user_input)
plan_span.set_attribute("agent.plan.steps", len(plan))
with tracer.start_as_current_span("llm.chat") as llm_span:
llm_span.set_attribute("gen_ai.system", "openai")
llm_span.set_attribute("gen_ai.operation.name", "chat")
llm_span.set_attribute("gen_ai.request.model", "gpt-4.1")
llm_response = call_llm(user_input)
llm_span.set_attribute("gen_ai.response.model", "gpt-4.1")
llm_span.set_attribute("gen_ai.usage.input_tokens", llm_response.input_tokens)
llm_span.set_attribute("gen_ai.usage.output_tokens", llm_response.output_tokens)
with tracer.start_as_current_span("tool.call") as tool_span:
tool_span.set_attribute("tool.name", "knowledge_search")
tool_span.set_attribute("tool.call_id", llm_response.tool_call_id)
documents = search_documents(llm_response.tool_query)
tool_span.set_attribute("tool.success", True)
tool_span.set_attribute("retrieval.document_count", len(documents))
with tracer.start_as_current_span("agent.finalize") as final_span:
final_answer = produce_answer(llm_response, documents)
final_span.set_attribute("agent.output.length", len(final_answer))
return final_answer
except Exception as exc:
span.record_exception(exc)
span.set_status(Status(StatusCode.ERROR, str(exc)))
raise
5. Node.js or TypeScript Agent Setup
Install dependencies:
npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-http
Create telemetry.ts:
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
export const telemetrySdk = new NodeSDK({
serviceName: "agent-service",
traceExporter: new OTLPTraceExporter({
url: "http://localhost:4318/v1/traces",
}),
});
export async function startTelemetry() {
await telemetrySdk.start();
}
export async function stopTelemetry() {
await telemetrySdk.shutdown();
}
Instrument the agent:
import { SpanStatusCode, trace } from "@opentelemetry/api";
import { startTelemetry, stopTelemetry } from "./telemetry";
const tracer = trace.getTracer("agent-service");
export async function runAgent(userInput: string): Promise<string> {
return tracer.startActiveSpan("agent.run", async span => {
span.setAttribute("agent.name", "support-agent");
span.setAttribute("agent.version", "1.0.0");
span.setAttribute("agent.input.length", userInput.length);
try {
const plan = await tracer.startActiveSpan("agent.plan", async planSpan => {
const result = await createPlan(userInput);
planSpan.setAttribute("agent.plan.steps", result.length);
planSpan.end();
return result;
});
const llmResponse = await tracer.startActiveSpan("llm.chat", async llmSpan => {
llmSpan.setAttribute("gen_ai.system", "openai");
llmSpan.setAttribute("gen_ai.operation.name", "chat");
llmSpan.setAttribute("gen_ai.request.model", "gpt-4.1");
const response = await callLlm(userInput, plan);
llmSpan.setAttribute("gen_ai.response.model", "gpt-4.1");
llmSpan.setAttribute("gen_ai.usage.input_tokens", response.inputTokens);
llmSpan.setAttribute("gen_ai.usage.output_tokens", response.outputTokens);
llmSpan.end();
return response;
});
const documents = await tracer.startActiveSpan("tool.call", async toolSpan => {
toolSpan.setAttribute("tool.name", "knowledge_search");
toolSpan.setAttribute("tool.call_id", llmResponse.toolCallId);
const result = await searchDocuments(llmResponse.toolQuery);
toolSpan.setAttribute("tool.success", true);
toolSpan.setAttribute("retrieval.document_count", result.length);
toolSpan.end();
return result;
});
const finalAnswer = await tracer.startActiveSpan("agent.finalize", async finalSpan => {
const answer = await produceAnswer(llmResponse, documents);
finalSpan.setAttribute("agent.output.length", answer.length);
finalSpan.end();
return answer;
});
span.end();
return finalAnswer;
} catch (error) {
span.recordException(error as Error);
span.setStatus({
code: SpanStatusCode.ERROR,
message: String(error),
});
span.end();
throw error;
}
});
}
async function main() {
await startTelemetry();
try {
await runAgent("How do I reset my password?");
} finally {
await stopTelemetry();
}
}
6. Recommended Span Attributes
Use OpenTelemetry semantic conventions where possible.
GenAI Attributes
gen_ai.system
gen_ai.operation.name
gen_ai.request.model
gen_ai.response.model
gen_ai.request.temperature
gen_ai.request.max_tokens
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
Agent Attributes
agent.name
agent.version
agent.run_id
agent.session_id
agent.step.name
agent.step.index
agent.output.length
agent.input.length
Tool Attributes
tool.name
tool.call_id
tool.success
tool.error.type
tool.retry_count
Retrieval Attributes
retrieval.system
retrieval.index.name
retrieval.query_count
retrieval.document_count
retrieval.top_k
Memory Attributes
memory.operation
memory.scope
memory.result_count
7. What Not To Capture
Do not capture sensitive or high-risk data as span attributes unless your organization has explicit approval, redaction, access control, and retention policies.
Avoid storing:
Full prompts
Full model responses
User secrets
API keys or tokens
Raw documents
Email addresses
Payment information
Health data
Authentication headers
Full tool outputs
Prefer safe metadata:
prompt length
response length
token counts
model name
tool name
status
latency
retry count
document count
8. Metrics To Add
Traces show what happened for one request. Metrics show aggregate behavior.
Recommended metrics:
agent.run.count
agent.run.duration
agent.run.error.count
llm.request.count
llm.request.duration
llm.token.input.count
llm.token.output.count
tool.call.count
tool.call.duration
tool.call.error.count
retrieval.query.count
retrieval.document.count
agent.handoff.count
Start with traces first, then add metrics once your span structure is stable.
9. Logging Setup
Use logs for discrete application events, but correlate them with traces.
Recommended log fields:
trace_id
span_id
agent_run_id
agent.name
event.name
status
error.type
error.message
Keep logs redacted. Avoid logging raw prompts, completions, credentials, or document bodies.
10. Validation Checklist
Use this checklist to verify the setup end to end.
Start the OpenTelemetry Collector.
Run one local agent request.
Confirm the Collector receives spans.
Confirm the root span is named agent.run.
Confirm LLM calls appear as llm.chat child spans.
Confirm tool calls appear as tool.call child spans.
Confirm errors are recorded on failed spans.
Confirm token usage attributes appear when available.
Confirm sensitive content is not exported.
Confirm traces reach your observability backend.
11. Production Collector Example
For production, route data through the Collector and export to your backend.
otlp:
protocols:
http:
grpc:
processors:
batch:
memory_limiter:
check_interval: 1s
limit_mib: 512
spike_limit_mib: 128
exporters:
otlphttp:
endpoint: https://your-observability-backend.example.com
headers:
api-key: ${OBSERVABILITY_API_KEY}
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp]
Set environment variables securely through your deployment platform, secret manager, or CI/CD system.
12. Production Hardening
Before production rollout:
Add sampling if trace volume is high.
Redact sensitive inputs and outputs.
Add service names and versions.
Add deployment environment attributes.
Use secure OTLP endpoints.
Store backend API keys in a secret manager.
Configure retention policies.
Add alerts for agent error rate and high latency.
Track model cost through token metrics.
Validate compliance requirements before exporting AI data.
13. Sampling Guidance
For development, sample everything.
For production, consider:
100% sampling for errors
100% sampling for low-volume critical workflows
Lower probabilistic sampling for high-volume successful requests
Tail sampling when supported by your backend or Collector distribution
Example parent-based trace ID ratio sampling in Python:
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
provider = TracerProvider(
sampler=ParentBased(TraceIdRatioBased(0.10))
)
This samples roughly 10% of new traces while preserving parent-child trace consistency.
14. Framework-Specific Notes
LangChain
Instrument the chain or agent executor as agent.run, then wrap model calls, retrievers, and tools as child spans. If you use callbacks, create spans inside callback handlers.
Semantic Kernel
Trace kernel invocation as agent.run, function calls as tool.call, planner execution as agent.plan, and AI service calls as llm.chat.
AutoGen or Multi-Agent Systems
Use one trace for the full multi-agent workflow. Each agent turn can be represented as agent.run or agent.step, with agent.name distinguishing participants. Handoffs should be explicit spans named agent.handoff.
Custom Agents
Create spans directly around your orchestration code. This usually gives the best signal because you know where planning, memory, retrieval, tool execution, and finalization happen.
15. Minimal Rollout Plan
Add the OpenTelemetry SDK to the agent service.
Send traces to a local Collector.
Add a root agent.run span.
Add child spans for LLM calls and tool calls.
Validate trace shape locally.
Add retrieval and memory spans.
Add token and latency attributes.
Add redaction rules.
Export from the Collector to your observability backend.
Add dashboards and alerts.
16. Example Dashboard Panels
Useful panels for agent operations:
Agent run count by agent name
Agent error rate
Agent p50, p95, and p99 latency
LLM latency by model
Input and output tokens by model
Tool call failure rate
Retrieval document count
Handoff count
Estimated model cost
Top failing tools
17. Power BI Dashboard For Usage And Tokens
Power BI should usually read from a queryable store, not directly from raw OpenTelemetry traces. Use OpenTelemetry for observability, then write a normalized usage table for reporting.
Recommended flow:
Agent Application
|
v
OpenTelemetry traces and metrics
|
v
OpenTelemetry Collector
|
v Export
v
Azure Monitor, Log Analytics, SQL, Fabric Lakehouse, or Data Warehouse
|
v Power BI semantic model
v
Power BI dashboard
Recommended Data Source Options
Use one of these patterns:
Azure Monitor or Log Analytics if your traces already go to Azure.
Application Insights if your application telemetry is already centralized there.
Azure SQL Database if you want simple relational reporting.
Microsoft Fabric Lakehouse or Warehouse if you want scalable analytics.
Databricks, Snowflake, or BigQuery if your organization already uses one of them.
For most teams, the easiest production setup is to store one row per LLM request in a table named agent_llm_usage.
Usage Table Schema
Create a reporting table with stable, low-cardinality columns.
CREATE TABLE agent_llm_usage (
usage_id VARCHAR(100) NOT NULL,
trace_id VARCHAR(100) NULL,
span_id VARCHAR(100) NULL,
timestamp_utc DATETIME2 NOT NULL,
environment VARCHAR(50) NOT NULL,
service_name VARCHAR(100) NOT NULL,
agent_name VARCHAR(100) NOT NULL,
agent_version VARCHAR(50) NULL,
session_id VARCHAR(100) NULL,
user_id_hash VARCHAR(100) NULL,
model_provider VARCHAR(100) NULL,
model_name VARCHAR(100) NOT NULL,
operation_name VARCHAR(50) NOT NULL,
input_tokens INT NOT NULL DEFAULT 0,
output_tokens INT NOT NULL DEFAULT 0,
total_tokens AS (input_tokens + output_tokens),
estimated_cost_usd DECIMAL(18, 6) NULL,
duration_ms INT NULL,
status VARCHAR(30) NOT NULL,
error_type VARCHAR(200) NULL,
tool_name VARCHAR(100) NULL,
retrieval_document_count INT NULL
);
Do not store raw prompts, raw completions, secrets, full document text, or emails in this table. Use hashed user identifiers if user-level reporting is required.
Writing Usage Records
OpenTelemetry spans should still include token attributes such as:
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.request.model
gen_ai.system
agent.name
agent.session_id
For Power BI, also write a compact business reporting event when each LLM call finishes. In Python, the event can be inserted into SQL, sent to Event Hubs, or written to your analytics pipeline.
def record_llm_usage(response, context):
usage_record = {
"usage_id": context.usage_id,
"trace_id": context.trace_id,
"span_id": context.span_id,
"timestamp_utc": context.timestamp_utc,
"environment": context.environment,
"service_name": "agent-service",
"agent_name": context.agent_name,
"agent_version": context.agent_version,
"session_id": context.session_id,
"user_id_hash": context.user_id_hash,
"model_provider": response.provider,
"model_name": response.model,
"operation_name": "chat",
"input_tokens": response.input_tokens,
"output_tokens": response.output_tokens,
"estimated_cost_usd": response.estimated_cost_usd,
"duration_ms": response.duration_ms,
"status": "success",
}
insert_usage_record(usage_record)
Cost Calculation
Keep model pricing in a separate table so costs can be updated without changing historical usage records.
CREATE TABLE model_pricing (
model_provider VARCHAR(100) NOT NULL,
model_name VARCHAR(100) NOT NULL,
effective_from_utc DATETIME2 NOT NULL,
input_cost_per_1k_tokens DECIMAL(18, 8) NOT NULL,
output_cost_per_1k_tokens DECIMAL(18, 8) NOT NULL
);
The estimated cost formula is:
estimated_cost = (input_tokens / 1000 * input_price) + (output_tokens / 1000 * output_price)
You can calculate this in the ingestion pipeline, SQL view, Fabric notebook, or Power BI semantic model. Pipeline or SQL calculation is usually better because all reports use the same cost logic.
Power BI Data Model
Use a simple star schema.
Fact table:
- agent_llm_usage
Dimension tables:
dim_date
dim_agent
dim_model
dim_environment
dim_tool
Relationships:
dim_date[date] -> agent_llm_usage[date]
dim_agent[agent_name] -> agent_llm_usage[agent_name]
dim_model[model_name] -> agent_llm_usage[model_name]
dim_environment[environment] -> agent_llm_usage[environment]
dim_tool[tool_name] -> agent_llm_usage[tool_name]
Core DAX Measures
Create these measures in Power BI:
Total Requests =
COUNTROWS(agent_llm_usage)
Successful Requests =
CALCULATE(
COUNTROWS(agent_llm_usage),
agent_llm_usage[status] = "success"
)
Failed Requests =
CALCULATE(
COUNTROWS(agent_llm_usage),
agent_llm_usage[status] <> "success"
)
Failure Rate =
DIVIDE([Failed Requests], [Total Requests])
Input Tokens =
SUM(agent_llm_usage[input_tokens])
Output Tokens =
SUM(agent_llm_usage[output_tokens])
Total Tokens =
[Input Tokens] + [Output Tokens]
Estimated Cost USD =
SUM(agent_llm_usage[estimated_cost_usd])
Average Duration MS =
AVERAGE(agent_llm_usage[duration_ms])
Average Tokens Per Request =
DIVIDE([Total Tokens], [Total Requests])
P95 Duration MS =
PERCENTILEX.INC(
agent_llm_usage,
agent_llm_usage[duration_ms],
0.95
)
Recommended Power BI Pages
Create these report pages:
Executive Overview
Token Usage
Cost Analysis
Agent Performance
Model Performance
Tool And Retrieval Usage
Errors And Reliability
Recommended visuals for the Executive Overview page:
Card: total requests
Card: total tokens
Card: estimated cost
Card: failure rate
Line chart: requests by day
Line chart: tokens by day
Bar chart: cost by agent
Bar chart: tokens by model
Table: top agents by cost, tokens, and failures
Recommended visuals for the Token Usage page:
Line chart: input tokens and output tokens over time
Stacked column chart: tokens by model
Matrix: agent name by model name with total tokens
Slicer: date range
Recommended visuals for the Cost Analysis page:
Slicer: environment
Slicer: agent name
Slicer: model name
Line chart: estimated cost by day
Bar chart: estimated cost by model
Bar chart: estimated cost by agent
Table: session or hashed user groups by cost
KPI: cost per request
Recommended visuals for the Errors And Reliability page:
Card: failed requests
Card: failure rate
Line chart: failures by day
Bar chart: errors by model
Bar chart: errors by tool
Table: error type, agent name, model name, and count
Refresh And Governance
Recommended refresh setup:
Development: manual refresh
Small production workload: scheduled refresh every 1 to 4 hours
High-volume production workload: Direct Lake, DirectQuery, or incremental refresh
Governance recommendations:
Use row-level security if teams should only see their own agents.
Store only hashed user IDs.
Keep prompt and completion content out of the reporting model.
Certify the semantic model once measures are validated.
Document model pricing assumptions and update them when provider pricing changes.
18. Troubleshooting
No Spans In Collector
Check:
Collector is running.
App exports to http://localhost:4318/v1/traces for OTLP HTTP.
Port 4318 is reachable.
The SDK is initialized before the agent runs.
The app shuts down cleanly so batch spans are flushed.
Root Span Exists But Child Spans Are Missing
Check:
Child spans are created inside the active context.
Async operations preserve context.
Spans are ended after work completes.
Exceptions do not skip span.end() in TypeScript.
Too Much Sensitive Data
Check:
No full prompts are added as attributes.
No raw tool outputs are added as attributes.
Logs do not contain secrets.
Collector processors or backend rules redact known sensitive fields.
High Trace Volume
Check:
Add sampling.
Reduce span count for noisy internal steps.
Keep attributes low-cardinality.
Avoid unique values such as full user IDs, emails, or raw queries in indexed attributes.
19. Final Recommended Baseline
A practical first production baseline is:
Root span:
agent.run
Child spans:
agent.plan
llm.chat
tool.call
retrieval.query
memory.read
memory.write
agent.finalize
Required attributes:
service.name
deployment.environment
agent.name
agent.version
gen_ai.system
gen_ai.request.model
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
tool.name
tool.success
This gives enough visibility to answer the most important operational questions: what happened, where time was spent, which model and tools were used, whether the run failed, and how much token usage it consumed.
