OpenTelemetry in .NET Microservices: A Practical Guide to Observable Distributed Systems

You've deployed your fifth microservice. Requests flow through an API gateway, hit an order service, fan out to inventory and payment, and eventually land in a notification pipeline. Everything works — until it doesn't. A customer reports that checkout takes 12 seconds. Your logs show nothing unusual. Your metrics say average latency is fine. But somewhere in that chain of services, something is dragging, and you have no idea where.

This is the observability gap that kills microservice architectures. Monoliths gave us stack traces. Microservices gave us distributed amnesia.

OpenTelemetry exists to close that gap — and in the .NET ecosystem, it's matured from an experimental curiosity into the de facto standard for instrumentation. This post covers everything you need to go from zero to production-grade observability in your .NET microservices, with concrete C# examples, real debugging scenarios, and deployment patterns for Azure.

Why Observability Is Hard in Microservices

If you've operated distributed systems, you already know the pain points:

• Distributed tracing is non-trivial. A single user request might touch 8 services, 3 databases, and 2 message queues. Correlating that flow requires propagating context across HTTP calls, gRPC channels, and async messaging — consistently.
• Logs are scattered. Each service writes its own logs to its own sink. Reconstructing a timeline means grepping across dozens of streams and mentally stitching timestamps together.
• Metrics lack context. Knowing that P99 latency spiked on the payment service tells you where to look, but not why. Was it a slow database query? A downstream timeout? A garbage collection pause?
• Vendor lock-in is real. You instrumented everything for Datadog, and now leadership wants to move to Azure Monitor. Ripping out and replacing telemetry code across 20 services is not how anyone wants to spend a quarter.

OpenTelemetry addresses all of these problems with a single, vendor-neutral instrumentation framework.

What Is OpenTelemetry?

OpenTelemetry (OTel) is a CNCF project that provides a unified set of APIs, SDKs, and conventions for generating, collecting, and exporting telemetry data — specifically traces, metrics, and logs.

The key value proposition is decoupling instrumentation from backends. You instrument your code once with OTel APIs, and then choose where to send the data: Azure Monitor, Jaeger, Grafana, Coralogix, Datadog, Honeycomb, or any OTLP-compatible backend. Swap backends without touching application code.

OpenTelemetry in the .NET Ecosystem

The .NET OTel story is particularly strong because it builds on primitives that already exist in the runtime:

• System.Diagnostics.Activity is the native .NET distributed tracing API, and OTel's .NET SDK wraps it directly — no parallel abstraction layer.
• System.Diagnostics.Metrics (introduced in .NET 6) provides the metrics API that OTel hooks into.
• ILogger<T> integration means your existing structured logging flows naturally into OTel's log pipeline.

This isn't a foreign framework bolted onto .NET. It's a thin orchestration layer over APIs you're probably already using.

Core Concepts

Traces, Spans, and Context Propagation

A trace represents the entire lifecycle of a request as it flows through your system. Each trace is composed of spans — individual units of work. A span records an operation's name, start/end timestamps, status, and key-value attributes.

Spans form a tree: a root span (e.g., the incoming HTTP request) has child spans (e.g., a database query, an outbound HTTP call), which may themselves have children. The parent-child relationship is established via context propagation — passing a trace ID and span ID across service boundaries.

In .NET, context propagation over HTTP is handled automatically when you use HttpClient with the OTel SDK configured. The SDK injects W3C traceparent headers on outbound calls and extracts them on inbound requests. For messaging scenarios (RabbitMQ, Azure Service Bus), you'll need to propagate context manually through message headers.

[API Gateway] ──► [Order Service] ──► [Payment Service]
  span: gateway       span: order         span: payment
  traceId: abc123     traceId: abc123     traceId: abc123
  spanId: 001         spanId: 002         spanId: 003
                      parentId: 001       parentId: 002

Metrics and Logs

Metrics capture aggregated, numerical measurements over time: request counts, histogram of response times, gauge of active connections. Unlike traces (which capture individual requests), metrics are cheap at scale because they're pre-aggregated.

Logs provide the richest contextual detail — exception stack traces, business-level events, debug output — but are the most expensive signal to store and query.

The three signals complement each other:

• Metrics alert you that something is wrong (latency spike).
• Traces show you where in the call chain the problem is.
• Logs tell you why (the specific error, the slow query text, the unexpected null).

OpenTelemetry correlates all three by attaching trace and span IDs to log records and metric exemplars, so you can jump from a metric anomaly to the specific traces that caused it.

Instrumentation: Automatic vs. Manual

Automatic instrumentation (also called library instrumentation) is provided by community packages that hook into common libraries. For .NET, this includes:

• OpenTelemetry.Instrumentation.AspNetCore — traces for incoming HTTP requests
• OpenTelemetry.Instrumentation.HttpClient — traces for outgoing HTTP calls
• OpenTelemetry.Instrumentation.SqlClient — traces for SQL queries
• OpenTelemetry.Instrumentation.EntityFrameworkCore — traces for EF Core operations
• OpenTelemetry.Instrumentation.GrpcNetClient — traces for gRPC calls

These packages generate spans automatically with no code changes beyond registration.

Manual instrumentation is what you add for your own business logic — things automatic instrumentation can't know about. Processing a payment? Validating inventory? Calling a third-party API with retry logic? You create custom spans and record custom metrics for these.

Exporters and Backends

An exporter is the component that sends telemetry data to a backend. Common exporters in .NET:

• OTLP (OpenTelemetry Protocol): The standard wire format. Works with the OpenTelemetry Collector, Grafana Tempo, Jaeger, and most modern backends.
• Azure Monitor: Sends data to Application Insights / Azure Monitor via the Azure.Monitor.OpenTelemetry.Exporter package.
• Console: Writes telemetry to stdout — useful for local development and debugging.
• Zipkin / Jaeger (direct): Native exporters for these backends, though OTLP is generally preferred now.

Hands-On: Setting Up OpenTelemetry in ASP.NET Core

Step 1: Install the Packages

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.HttpClient
dotnet add package OpenTelemetry.Instrumentation.SqlClient
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Exporter.Console
dotnet add package Azure.Monitor.OpenTelemetry.Exporter

Step 2: Configure the SDK in Program.cs

using OpenTelemetry;
using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;
using OpenTelemetry.Logs;
using OpenTelemetry.Resources;

var builder = WebApplication.CreateBuilder(args);

// Define a shared resource describing this service
var serviceName = "OrderService";
var serviceVersion = "1.0.0";

var resourceBuilder = ResourceBuilder.CreateDefault()
    .AddService(serviceName: serviceName, serviceVersion: serviceVersion)
    .AddAttributes(new Dictionary<string, object>
    {
        ["deployment.environment"] = builder.Environment.EnvironmentName,
        ["host.name"] = Environment.MachineName
    });

// Configure Tracing
builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService(serviceName, serviceVersion: serviceVersion))
    .WithTracing(tracing =>
    {
        tracing
            .SetResourceBuilder(resourceBuilder)
            .AddAspNetCoreInstrumentation(opts =>
            {
                // Filter out health check noise
                opts.Filter = ctx => !ctx.Request.Path.StartsWithSegments("/health");
            })
            .AddHttpClientInstrumentation()
            .AddSqlClientInstrumentation(opts =>
            {
                opts.SetDbStatementForText = true; // Capture SQL text (careful in prod)
                opts.RecordException = true;
            })
            .AddSource("OrderService.Activities") // Custom ActivitySource
            .AddOtlpExporter(opts =>
            {
                opts.Endpoint = new Uri("http://otel-collector:4317");
            });

        // Add Azure Monitor in deployed environments
        if (!builder.Environment.IsDevelopment())
        {
            tracing.AddAzureMonitorTraceExporter(opts =>
            {
                opts.ConnectionString = builder.Configuration
                    .GetValue<string>("AzureMonitor:ConnectionString");
            });
        }
        else
        {
            tracing.AddConsoleExporter();
        }
    })
    .WithMetrics(metrics =>
    {
        metrics
            .SetResourceBuilder(resourceBuilder)
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddMeter("OrderService.Metrics") // Custom Meter
            .AddOtlpExporter()
            .AddPrometheusExporter(); // /metrics endpoint for scraping
    });

// Configure Logging
builder.Logging.AddOpenTelemetry(logging =>
{
    logging
        .SetResourceBuilder(resourceBuilder)
        .AddOtlpExporter()
        .IncludeFormattedMessage = true;
});

var app = builder.Build();
app.MapPrometheusScrapingEndpoint(); // Expose /metrics
// ... rest of pipeline
app.Run();

Step 3: Add Custom Instrumentation

using System.Diagnostics;
using System.Diagnostics.Metrics;

public class OrderProcessingService
{
    // ActivitySource is the .NET equivalent of a Tracer
    private static readonly ActivitySource ActivitySource =
        new("OrderService.Activities", "1.0.0");

    // Custom metrics
    private static readonly Meter Meter = new("OrderService.Metrics", "1.0.0");
    private static readonly Counter<long> OrdersProcessed =
        Meter.CreateCounter<long>("orders.processed", "orders", "Total orders processed");
    private static readonly Histogram<double> OrderProcessingDuration =
        Meter.CreateHistogram<double>("orders.processing_duration", "ms",
            "Time to process an order");

    private readonly ILogger<OrderProcessingService> _logger;
    private readonly PaymentClient _paymentClient;
    private readonly InventoryClient _inventoryClient;

    public OrderProcessingService(
        ILogger<OrderProcessingService> logger,
        PaymentClient paymentClient,
        InventoryClient inventoryClient)
    {
        _logger = logger;
        _paymentClient = paymentClient;
        _inventoryClient = inventoryClient;
    }

    public async Task<OrderResult> ProcessOrderAsync(Order order)
    {
        // Start a custom span
        using var activity = ActivitySource.StartActivity(
            "ProcessOrder",
            ActivityKind.Internal);

        // Add structured attributes to the span
        activity?.SetTag("order.id", order.Id);
        activity?.SetTag("order.item_count", order.Items.Count);
        activity?.SetTag("order.customer_id", order.CustomerId);

        var stopwatch = Stopwatch.StartNew();

        try
        {
            // Validate inventory — this creates a child span automatically
            // because HttpClient instrumentation is active
            var inventoryResult = await _inventoryClient
                .CheckAvailabilityAsync(order.Items);

            if (!inventoryResult.AllAvailable)
            {
                activity?.SetStatus(ActivityStatusCode.Error, "Insufficient inventory");
                activity?.AddEvent(new ActivityEvent("inventory.insufficient",
                    tags: new ActivityTagsCollection
                    {
                        { "unavailable_skus", string.Join(",",
                            inventoryResult.UnavailableSkus) }
                    }));

                return OrderResult.InsufficientInventory(inventoryResult.UnavailableSkus);
            }

            // Process payment
            var paymentResult = await _paymentClient
                .ChargeAsync(order.CustomerId, order.Total);

            activity?.SetTag("payment.transaction_id", paymentResult.TransactionId);

            _logger.LogInformation(
                "Order {OrderId} processed successfully. Transaction: {TransactionId}",
                order.Id, paymentResult.TransactionId);

            activity?.SetStatus(ActivityStatusCode.Ok);
            OrdersProcessed.Add(1,
                new KeyValuePair<string, object?>("order.status", "completed"));

            return OrderResult.Success(paymentResult.TransactionId);
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            OrdersProcessed.Add(1,
                new KeyValuePair<string, object?>("order.status", "failed"));

            _logger.LogError(ex, "Failed to process order {OrderId}", order.Id);
            throw;
        }
        finally
        {
            stopwatch.Stop();
            OrderProcessingDuration.Record(stopwatch.Elapsed.TotalMilliseconds,
                new KeyValuePair<string, object?>("order.status",
                    activity?.Status == ActivityStatusCode.Ok ? "completed" : "failed"));
        }
    }
}

Step 4: Correlating Requests Across Services

For HTTP-based communication, correlation is automatic. The HttpClient instrumentation injects traceparent headers, and the receiving service's AspNetCore instrumentation picks them up.

For message-based communication (e.g., Azure Service Bus, RabbitMQ), you need to propagate context manually:

// Producer: inject trace context into message headers
public async Task PublishOrderEventAsync(OrderEvent orderEvent)
{
    using var activity = ActivitySource.StartActivity(
        "PublishOrderEvent", ActivityKind.Producer);

    activity?.SetTag("messaging.system", "servicebus");
    activity?.SetTag("messaging.destination", "order-events");

    var message = new ServiceBusMessage(JsonSerializer.Serialize(orderEvent));

    // Inject current context into message properties
    if (activity?.Context is ActivityContext context)
    {
        message.ApplicationProperties["traceparent"] =
            $"00-{context.TraceId}-{context.SpanId}-{(context.TraceFlags.HasFlag(ActivityTraceFlags.Recorded) ? "01" : "00")}";
    }

    await _sender.SendMessageAsync(message);
}

// Consumer: extract trace context from message headers
public async Task HandleMessageAsync(ServiceBusReceivedMessage message)
{
    ActivityContext parentContext = default;

    if (message.ApplicationProperties.TryGetValue("traceparent", out var traceparent)
        && traceparent is string tp)
    {
        // Parse W3C traceparent header
        var parts = tp.Split('-');
        if (parts.Length == 4)
        {
            var traceId = ActivityTraceId.CreateFromString(parts[1]);
            var spanId = ActivitySpanId.CreateFromString(parts[2]);
            var flags = parts[3] == "01"
                ? ActivityTraceFlags.Recorded : ActivityTraceFlags.None;
            parentContext = new ActivityContext(traceId, spanId, flags, isRemote: true);
        }
    }

    using var activity = ActivitySource.StartActivity(
        "ProcessOrderEvent",
        ActivityKind.Consumer,
        parentContext);

    activity?.SetTag("messaging.system", "servicebus");
    activity?.SetTag("messaging.operation", "process");

    // Process the message...
}

Real-World Scenarios

Scenario 1: Debugging Latency Across Services

A customer complains that placing an order takes too long. With OpenTelemetry configured, you query your tracing backend for traces where order.processing_duration > 5000ms.

You pull up a trace and see:

[OrderController.Post]          0ms ─────────────────────── 6200ms
  [ProcessOrder]                5ms ──────────────────────  6195ms
    [HTTP GET /inventory]       10ms ─── 120ms
    [HTTP POST /payments]       125ms ────────────────────  6100ms
      [SQL: INSERT payment]     130ms ── 180ms
      [HTTP POST /fraud-check]  200ms ────────────────────  6050ms  ← BOTTLENECK

The fraud-check service is taking nearly 6 seconds. Without distributed tracing, you'd be staring at order service logs wondering why ProcessOrderAsync is slow, with no visibility into what the payment service is doing downstream.

Scenario 2: Tracking End-to-End Request Flow

A webhook delivery to a partner is failing intermittently. You search for traces by webhook.partner_id = "acme" and find that successful deliveries take 200ms, but failures show a pattern: the notification service's outbound HTTP call to the partner endpoint returns 503 after exactly 30 seconds — a timeout.

The trace reveals the full chain: API → Order → Notification → Partner webhook. The partner's endpoint is intermittently overloaded. You add a retry policy with exponential backoff and add a span event to track each retry attempt.

Scenario 3: Identifying Production Bottlenecks

Your orders.processing_duration histogram shows P95 degradation over two weeks. You correlate with exemplars — specific trace IDs attached to the slow metric data points — and find they all share a common pattern: the SQL span for inventory reservation is growing linearly.

Digging into the span attributes, you see db.statement contains a query that's missing an index on the sku column. As the product catalog grew, that query went from 5ms to 500ms. The trace data told you exactly which query, in which service, against which table.

Best Practices

Naming Conventions

Use semantic conventions from the OpenTelemetry specification. This ensures consistency across services and teams:

• Span names should be low-cardinality: ProcessOrder, HTTP GET /api/orders/{id} (not /api/orders/12345). Include the verb and resource, not dynamic IDs.
• Attribute keys follow dot.notation: order.id, payment.method, customer.tier. Avoid inventing your own when a semantic convention exists (http.method, db.system, rpc.service).
• Meter and instrument names use reverse-domain style: orderservice.orders.processed, orderservice.processing.duration.

Reducing Telemetry Noise and Cost

Not every span is worth sending to your backend:

• Filter health checks and readiness probes — as shown in the setup code, exclude /health and /ready endpoints from trace instrumentation.
• Use the Filter callback on instrumentation options to drop low-value spans programmatically.
• Limit db.statement capture in production. Full SQL text is invaluable for debugging but expensive to store at scale. Consider enabling it only in non-production environments or for sampled traces.
• Control attribute cardinality. Adding user.email to every span creates millions of unique attribute values and explodes storage costs. Prefer IDs over free-text values.

Sampling Strategies

Sampling controls which traces get recorded and exported:

• AlwaysOn: Record every trace. Fine for low-traffic services or development.
• AlwaysOff: Record nothing. Useful for suppressing telemetry in batch jobs that would generate excessive noise.
• TraceIdRatioBased: Record a fixed percentage of traces (e.g., 10%). Simple and predictable, but you might miss the rare-but-critical slow request.
• ParentBased: Respect the sampling decision of the upstream service. If the parent was sampled, sample the child too. This is critical for keeping traces complete.

tracing.SetSampler(new ParentBasedSampler(
    new TraceIdRatioBasedSampler(0.1))); // 10% base rate, but always honor parent

In practice, tail-based sampling (deciding after the trace is complete, based on whether it was slow or errored) is far more useful than head-based sampling. This requires the OpenTelemetry Collector with a tail-sampling processor — the SDK doesn't support it natively.

Securing Telemetry Data

Telemetry data can contain sensitive information: query parameters, user IDs, database statements, error messages with PII.

• Scrub sensitive attributes before export. Use the SDK's Enrich and Filter callbacks to redact or remove PII.
• Use TLS for OTLP exports. The OTLP exporter supports gRPC and HTTP/protobuf — ensure endpoints use https:// or encrypted gRPC channels.
• Restrict access to your telemetry backend. Traces contain internal architecture details. Treat your observability platform with the same access control rigor as production databases.
• Never log secrets. This sounds obvious, but automatic instrumentation on HttpClient can capture request headers. Explicitly exclude Authorization and other sensitive headers.

.AddHttpClientInstrumentation(opts =>
{
    opts.FilterHttpRequestMessage = req =>
    {
        // Don't trace calls to the token endpoint
        return !req.RequestUri!.AbsolutePath.Contains("/oauth/token");
    };
    opts.EnrichWithHttpRequestMessage = (activity, request) =>
    {
        // Remove sensitive query parameters
        activity?.SetTag("http.url",
            request.RequestUri?.GetLeftPart(UriPartial.Path));
    };
})

Deploying and Monitoring in Azure

Integration with Application Insights / Azure Monitor

The Azure.Monitor.OpenTelemetry.Exporter package sends traces, metrics, and logs directly to Application Insights without requiring the OpenTelemetry Collector as an intermediary. For many teams on Azure, this is the fastest path to production observability.

Alternatively, the newer Azure Monitor OpenTelemetry Distro (Azure.Monitor.OpenTelemetry.AspNetCore) bundles common instrumentation libraries with the exporter in a single package:

builder.Services.AddOpenTelemetry()
    .UseAzureMonitor(opts =>
    {
        opts.ConnectionString = builder.Configuration["AzureMonitor:ConnectionString"];
    });

This auto-configures ASP.NET Core, HttpClient, and SQL Client instrumentation with the Azure Monitor exporter. It's opinionated but gets you to 80% coverage with minimal code.

In Application Insights, you get:

• Transaction search: Find traces by operation ID, custom attributes, or time range.
• Application Map: Visualize service dependencies and identify failure hotspots.
• Performance blade: Drill from P95 latency to individual slow traces.
• Failures blade: Aggregate exceptions by type and trace them back to the originating request.

Observability in Containerized Environments (Kubernetes)

When running .NET microservices on AKS or any Kubernetes cluster, the recommended architecture is:

1. Deploy the OpenTelemetry Collector as a DaemonSet or sidecar. Services export telemetry via OTLP to the Collector on localhost or the node's IP — fast, low-latency, no cross-network traffic.
2. The Collector handles routing, batching, and export. It can fan out to multiple backends (Azure Monitor + Grafana Tempo), apply tail-based sampling, enrich spans with Kubernetes metadata (k8s.pod.name, k8s.namespace.name), and buffer during backend outages.
3. Use the Kubernetes Attributes Processor in the Collector to automatically attach pod, node, and deployment metadata to every span — invaluable for correlating application behavior with infrastructure events.

A minimal Collector config for Azure:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  k8sattributes:
    extract:
      metadata:
        - k8s.pod.name
        - k8s.namespace.name
        - k8s.deployment.name
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces
        type: latency
        latency: { threshold_ms: 2000 }
      - name: probabilistic
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  azuremonitor:
    connection_string: ${APPLICATIONINSIGHTS_CONNECTION_STRING}
  otlp/tempo:
    endpoint: tempo.monitoring.svc.cluster.local:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [k8sattributes, tail_sampling, batch]
      exporters: [azuremonitor, otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [k8sattributes, batch]
      exporters: [azuremonitor]
    logs:
      receivers: [otlp]
      processors: [k8sattributes, batch]
      exporters: [azuremonitor]

This gives you tail-based sampling (always capture errors and slow requests, sample 10% of everything else), Kubernetes metadata enrichment, and dual export to both Azure Monitor and Grafana Tempo.

Key Takeaways

• OpenTelemetry is the industry standard for vendor-neutral observability. In .NET, it builds on native APIs (Activity, Meter, ILogger), making adoption low-friction.
• The three signals — traces, metrics, and logs — are complementary. Metrics detect anomalies, traces locate the problem, logs explain the cause. Correlation via trace IDs ties them together.
• Automatic instrumentation covers the infrastructure layer (HTTP, SQL, gRPC). Manual instrumentation is where you capture business-domain semantics (order processing, payment flow, inventory checks).
• Sampling and filtering are essential at scale. Tail-based sampling in the OTel Collector is the most effective strategy for capturing interesting traces without drowning in data.
• The OTel Collector is your telemetry control plane. Use it to decouple services from backends, apply processing rules, and route data to multiple destinations.

Implementation Checklist

• [ ] Install OTel SDK and instrumentation packages for ASP.NET Core, HttpClient, and your data access layer.
• [ ] Configure ResourceBuilder with service name, version, and environment attributes.
• [ ] Register automatic instrumentation for all infrastructure libraries in use.
• [ ] Create an ActivitySource and Meter for each service's custom business telemetry.
• [ ] Add meaningful span attributes and events to manual instrumentation — follow semantic conventions.
• [ ] Implement context propagation for asynchronous messaging (Service Bus, RabbitMQ, Kafka).
• [ ] Configure exporters: OTLP to a Collector for production, Console for local development.
• [ ] Set up sampling: ParentBasedSampler with TraceIdRatioBasedSampler as a starting point; evaluate tail-based sampling in the Collector.
• [ ] Filter out noise: health checks, readiness probes, internal polling.
• [ ] Scrub PII from span attributes and suppress sensitive headers.
• [ ] Deploy the OpenTelemetry Collector in your Kubernetes cluster with batching, K8s metadata enrichment, and your chosen exporters.
• [ ] Validate end-to-end trace correlation across at least two services before going to production.

Next Steps

Once you have baseline observability in place, consider exploring:

• Distributed context baggage for passing business-context values (tenant ID, feature flags) across service boundaries without adding them to every call explicitly.
• Exemplars — linking specific metric data points to the traces that produced them — for bridging from metric alerts to root-cause traces.
• Custom span processors for advanced enrichment: attaching git commit SHAs, feature flag states, or canary deployment markers to every span.
• SLO-driven alerting built on OTel metrics: define error budgets and burn rates using the histogram data you're already collecting.
• OpenTelemetry instrumentation for Azure SDK calls (Azure.Core tracing) to get visibility into blob storage, Key Vault, and other Azure service interactions.
• Performance tuning the telemetry pipeline itself: batch sizes, export intervals, memory limits on the Collector — telemetry infrastructure at scale needs its own observability.

Observability isn't a feature you ship once. It's a practice you refine as your system grows. OpenTelemetry gives you the foundation — a single, portable instrumentation layer that evolves with your architecture. Start with automatic instrumentation, add manual spans for the operations that matter most to your business, and iterate from there.