Open Source Project #159: OpenTelemetry Demo (Astronomy Shop) — The Official Distributed Observability Sandbox

Introduction

"Observability isn't a tool — it's a capability. To really learn how to diagnose distributed systems with Traces, Metrics, and Logs, you need a real system to work with, not documentation."

This is article #159 in the "One Open Source Project a Day" series. Today's project is OpenTelemetry Demo (Astronomy Shop) — the official OpenTelemetry demo system, positioned as: a real environment where you deliberately break things, then use observability data to find out why.

Not a Hello World. Not a toy. 17 microservices, 12 programming languages, Kafka message queue, PostgreSQL database, gRPC + HTTP mixed communication, and a full visualization stack: Grafana + Jaeger + Prometheus.

3,300 Stars, 7,000+ Forks. Apache-2.0 license. Over 50 cloud vendors — Datadog, AWS, Google Cloud, New Relic, Dynatrace — have forked it to demonstrate their own integrations.

What You'll Learn

• The three distinct purposes OpenTelemetry Demo serves
• Architecture overview of the 17 services and the logic behind technology choices
• How Traces/Metrics/Logs are implemented across 12 languages
• 13 fault switches: how to manufacture problems for root cause analysis practice
• The full visualization pipeline: Grafana + Jaeger + Prometheus
• Why 50+ vendors chose this as their integration demo foundation

Prerequisites

• Basic understanding of microservice architecture
• Familiarity with Traces, Metrics, and Logs as concepts
• Basic Docker knowledge

Background: The Core Problem With Learning Observability

Learning observability has a fundamental problem: without a real system, there's no real data. Without real data, you can't learn real diagnosis.

Practice distributed tracing on a single-service app and you learn "how to add a span to an HTTP handler." But production problems look like this: a request leaves the browser, passes through the API gateway, calls three microservices, one of which publishes to Kafka, an async consumer picks it up 30 seconds later, and triggers a database write failure — where do you see the error? Which service is responsible? Which hop is the root cause?

These questions only become real inside an actual distributed system.

OpenTelemetry Demo's answer: build a realistic system, equip it with 13 controllable fault switches, let you trigger problems at will, then find them with observability data.

Three Purposes the Project Serves

OpenTelemetry Demo serves three different audiences simultaneously, and the architecture design reflects this.

1. Engineers Learning Observability

A ready-made distributed system covering:

• Real SDK usage in 12 languages (auto-instrumentation vs. manual instrumentation trade-offs)
• Context propagation across gRPC and HTTP service calls
• Async trace correlation across Kafka messages
• Complete three-signal implementation: Traces, Metrics, Logs

Open Jaeger and you see a single request traveling from the browser through Frontend → Checkout → Payment → Email. Every hop's latency visible. Every error visible.

2. Vendors and Tooling Authors

Datadog, Elastic, AWS OpenSearch, Grafana Labs, New Relic, Dynatrace, Splunk, Google Cloud — over 50 companies have forked this repo, connected their own backends, and use it to show "here's what OTel data looks like in our platform."

The result: everything you learn about OTel in this project applies directly in any of those 50+ vendors' products. OTel signals are a neutral standard. No lock-in.

3. OTel Contributors and SDK Authors

New OTel SDK releases need real-world multi-language validation before shipping. This project runs 12 language SDKs simultaneously — natural integration test environment.

Architecture: 17 Services, 12 Languages

Service Inventory

Infrastructure

• Cache: Valkey (Redis-compatible)
• Database: PostgreSQL
• Message Queue: Kafka
• OTel Collector: centralized telemetry collection

Communication Protocols

• Service-to-service: primarily gRPC, some HTTP
• Telemetry export: OTLP/gRPC (port 4317) and OTLP/HTTP (port 4318)
• Async: Kafka TCP

Why These Languages

Each language choice maps to a specific OTel SDK validation need:

• C++ (Currency): highest QPS service — validates C++ SDK performance overhead
• Rust (Shipping): emerging backend language — demonstrates Rust SDK integration
• PHP (Quote): traditional web tech — shows PHP SDK setup
• Elixir (Flagd UI): functional/BEAM platform — OTel support for the Erlang ecosystem

Telemetry Pipeline: Services to Grafana

All services send telemetry to the OTel Collector, which fans out to visualization backends:

Services (12 languages)
    ↓ OTLP/gRPC or OTLP/HTTP
OTel Collector
    ├──→ Prometheus (metrics storage)    → localhost:9090
    ├──→ Jaeger (trace storage)          → localhost:16686
    └──→ OpenSearch (log storage)        → localhost:9200
         ↑ ↑ ↑
        Grafana (unified visualization)  → localhost:3000

The Collector also uses an OpAMP extension to report its own health, version, and effective configuration to the OpAMP server — a demonstration of OTel's remote management capability.

13 Fault Switches: Manufacture Problems on Demand

This is the most distinctive design decision in the project.

All fault switches are managed by Flagd (an OpenFeature-standard Feature Flag service), togglable at http://localhost:8080/feature with no service restarts.

Complete Fault Switch List

Practice Scenarios

Scenario 1: Memory Leak Diagnosis

Enable recommendationServiceCacheFailure:

• Watch Recommendation service memory metrics grow exponentially in Grafana
• See cache-related span anomalies in Jaeger traces for affected requests
• Practice: metric anomaly → identify service → trace investigation → confirm root cause

Scenario 2: Payment Cascade Failure

Enable paymentServiceUnreachable:

• Checkout service starts seeing payment failures
• In Jaeger, the Checkout trace shows Payment child spans timing out
• Practice: user reports "can't complete payment" → Checkout trace → Payment span → service unreachable

Scenario 3: Kafka Backlog Analysis

Enable kafkaQueueProblems:

• Kafka consumers (Accounting, Fraud Detection) start accumulating lag
• Consumer delay metrics in Prometheus climb continuously
• Practice: order processing delays → Kafka metrics → consumer lag → queue overload

Scenario 4: Frontend Latency Analysis

Enable imageSlowLoad (Envoy fault injection):

• Image load times in Frontend spike
• Jaeger Frontend traces show image request spans with unexpected latency
• Practice: browser Web Vitals degrade → Frontend trace → identify specific request

Getting Started

Docker Compose

git clone https://github.com/open-telemetry/opentelemetry-demo.git
cd opentelemetry-demo

# Start all services (first run pulls images — takes a few minutes)
docker compose up --no-build

# Check service readiness (some services take 1-2 minutes to initialize)
docker compose ps

Access points once services are ready:

Kubernetes via Helm

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

helm install my-otel-demo open-telemetry/opentelemetry-demo \
  --namespace otel-demo \
  --create-namespace

The Helm chart is published on Artifact Hub with per-service configuration override support.

Instrumentation Approaches: Auto vs. Manual

OTel provides two instrumentation paths. The project demonstrates both.

Auto-Instrumentation (Zero-Code)

No code changes — a framework-level agent captures telemetry automatically:

• Java (Ad Service): JVM agent via -javaagent:opentelemetry-javaagent.jar
• Python (Recommendation): opentelemetry-instrument python app.py
• .NET (Cart, Accounting): OTEL_DOTNET_AUTO_* environment variables

Best for: adding observability without touching existing code.

Manual Instrumentation

Using the SDK directly in business logic to create spans, add attributes, and record events:

// Go manual instrumentation (Checkout Service)
tracer := otel.Tracer("checkout")
ctx, span := tracer.Start(ctx, "placeOrder")
defer span.End()

span.SetAttributes(
    attribute.String("order.id", orderID),
    attribute.Int("order.items", len(items)),
)

if err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
}

// Rust manual instrumentation (Shipping Service)
let tracer = global::tracer("shipping");
let mut span = tracer.start("shipOrder");
span.set_attribute(KeyValue::new("shipping.method", method));

Best for: fine-grained control over business logic signals, custom attributes and events.

Why 50+ Vendors Chose This Over Building Their Own

Aligning on a shared demo base instead of building separate demos has practical advantages:

1. Shared audience context: engineers learning OTel may already know this project — vendor demos land on familiar ground
2. Fair comparison: all vendors use the same workload source — customers can compare platform experiences on equal terms
3. Maintenance leverage: the official team maintains the service architecture; vendors only maintain their "OTel Collector → their backend" integration
4. Version tracking: as OTel releases new versions, the official project updates, and vendor forks can rebase

Project Links

• GitHub: open-telemetry/opentelemetry-demo
• Documentation: opentelemetry.io/docs/demo
• Helm Chart: Artifact Hub - opentelemetry-demo
• OpenTelemetry: opentelemetry.io

Summary

OpenTelemetry Demo removes the core obstacle in learning observability: the absence of a real system to safely experiment with.

Twelve languages let you see OTel SDK usage across different tech stacks — not abstract documentation, but actual code. Thirteen fault switches let you practice root cause analysis in a controlled environment, walking the full diagnostic chain from metric anomaly to trace to log.

The larger value is what it represents industrially. When Datadog shows you their trace UI, they use this project. When Grafana Cloud demos distributed tracing, they use this project. When AWS demonstrates X-Ray integration, they use this project. Learn to trace a Kafka consumer lag back through this system's metrics, and that diagnostic approach transfers directly to any of those platforms — OTel signals are neutral, nothing locks you in.

3,300 Stars next to 7,000+ Forks is an unusual ratio: forks are twice the stars. The reason is those 50+ vendors, each maintaining their own fork. That fork-to-star ratio marks something the industry treats as infrastructure-level reference material, not just a learning resource.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.