Langfuse vs Arize Phoenix Self-Hosted: What Our Traces, Evals, and Containers Exposed

Why We Brought This Tool Into Our Lab

We did not need another dashboard counting tokens, the chunks of text a model processes. We needed to reconstruct failed agent runs and compare prompt revisions against fixed datasets. Prompts are inputs sent to a model; fixed datasets let us test each revision on the same examples. We also needed to attach human or automated scores and keep sensitive traces inside infrastructure we controlled.

Self-Hosted Observability Trade-Off

Because Langfuse required multiple stateful services while Phoenix scaled from a single container to PostgreSQL, choose Langfuse for product-facing prompt, feedback, and trace-review workflows, and Phoenix for OpenTelemetry-first Python evaluation workflows.

Our immediate production problem was scattered debugging information. A trace records the steps an application takes to handle a request. We collected application activity through OpenTelemetry, a standard set of tools for recording and sharing monitoring data. We kept model details in provider logs, test results in analysis notebooks, and prompt versions in deployment settings. To investigate a poor response, engineers had to match records across four systems manually. That worked for a prototype but wasted time once several engineers were changing the application at once.

We brought both Langfuse and Arize Phoenix into our lab because they overlap without being interchangeable.

Langfuse combined tools for operating applications built with large language models, or LLMs. It organized traces into spans, which record individual steps such as a model call. We could track model responses, group interactions into sessions, and connect them to users and prompt versions. We could also manage test datasets, experiment runs, scores, and annotation workflows in which people label or review results. This worked well for engineering and product teams investigating a customer interaction.

Phoenix helped us inspect application behavior and evaluate results using OpenTelemetry and OpenInference. OpenInference supplies shared names for trace details, such as the model used and the type of operation. We could reuse the code already collecting our traces. We could then inspect how the application fetched information and called tools, build datasets, and run Python experiments without adopting a product-specific trace format.

That distinction drove most of our final decision:

• We preferred Langfuse when our priorities were managing prompts over time, collecting application feedback, grouping interactions into sessions, and reviewing traces as a team.
• We preferred Phoenix when working with other OpenTelemetry tools, analyzing results in code notebooks, checking the quality of retrieved information, and defining experiments in code mattered more.
• We rejected the idea that either deployment was “just one container” in a production sense. Phoenix can start that way, but durable concurrent use pushed us toward PostgreSQL. From the beginning, Langfuse required several supporting services that retain data between restarts.

We also examined the licensing boundary before testing integrations. We found Langfuse’s core repository under the MIT license, while separately packaged enterprise capabilities required their own commercial terms. Phoenix used Elastic License 2.0 rather than a permissive license approved by the Open Source Initiative. Elastic License 2.0 allowed us to deploy and modify Phoenix internally. Its restrictions mattered if we planned to offer Phoenix itself as a managed service. We considered these licensing terms when making architecture decisions and conducting legal reviews, rather than treating them as a minor detail.

For adjacent infrastructure evaluations, we maintain the same deployment-first approach in our AI tools collection. Teams that need a workload-specific observability design can also review our AI infrastructure services.

Hands-On Walkthrough: Setup, Execution & Output

For a reproducible comparison, we would record the host's processor allocation, memory, and storage configuration before testing. We ran Docker Engine with Compose v2 to manage the containers together. We disabled unrelated workloads and recorded the unique fingerprints identifying each container image. We used the same program to generate test traces for both systems.

We followed the official Langfuse self-hosting path and started from its repository Compose definition:

git clone --depth 1 https://github.com/langfuse/langfuse.git
cd langfuse

# We replaced every placeholder before exposing the stack.
export NEXTAUTH_SECRET="$(openssl rand -hex 32)"
export SALT="$(openssl rand -hex 32)"
export ENCRYPTION_KEY="$(openssl rand -hex 32)"
export CLICKHOUSE_PASSWORD="$(openssl rand -hex 24)"
export MINIO_ROOT_PASSWORD="$(openssl rand -hex 24)"

docker compose pull
docker compose up -d
docker compose ps
docker compose config --images
git rev-parse HEAD

Our Compose deployment included the Langfuse web application, its background worker, PostgreSQL, ClickHouse, and Redis. It also included object storage compatible with the Amazon Simple Storage Service interface. We kept those services on a private Docker network. We exposed only the web interface through a reverse proxy, which forwarded incoming requests to it.

After opening http://localhost:3000, we created an account, project, public key, and secret key. For automation, we stored the project credentials outside the Compose file and passed them to the trace producer at runtime.

For Phoenix, we used the official self-hosting instructions and the Compose material from the Phoenix repository:

git clone --depth 1 https://github.com/Arize-ai/phoenix.git
cd phoenix

docker compose pull
docker compose up -d
docker compose ps
git rev-parse HEAD

For a disposable evaluation, we also verified that we could run Phoenix in a single container:

docker run --rm \
  -p 6006:6006 \
  -p 4317:4317 \
  -p 4318:4318 \
  arizephoenix/phoenix:latest

We used that single container only to check that the application started and worked. We switched to PostgreSQL to store data before sending traces and running experiments at the same time.

Both products accepted OpenTelemetry traces, so we could reuse the same trace-collection code. The following test script sends one trace covering information retrieval and answer generation. The OpenTelemetry Protocol, or OTLP, defines how that trace data travels to a receiver. For Langfuse, we supplied the receiver address and a Basic authorization header derived from the project’s public and secret keys. For Phoenix, we sent the same script’s output to its OTLP receiver using the Hypertext Transfer Protocol, or HTTP.

python -m venv .venv
. .venv/bin/activate
pip install \
  opentelemetry-api \
  opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-http

cat > emit_trace.py <<'PY'
import json
import os
import time

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

endpoint = os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"]
headers = {}

authorization = os.getenv("OTEL_AUTHORIZATION")
if authorization:
    headers["Authorization"] = authorization

provider = TracerProvider(
    resource=Resource.create(
        {
            "service.name": "effloow-observability-lab",
            "deployment.environment": "synthetic-test",
        }
    )
)
exporter = OTLPSpanExporter(endpoint=endpoint, headers=headers)
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("effloow.synthetic-rag")

started = time.perf_counter()

with tracer.start_as_current_span("support-agent") as root:
    root.set_attribute("session.id", "session-2026-09-23-001")
    root.set_attribute("user.id", "synthetic-user-17")

    with tracer.start_as_current_span("retrieve-policy") as retrieval:
        retrieval.set_attribute("openinference.span.kind", "RETRIEVER")
        retrieval.set_attribute(
            "input.value",
            "Can an annual subscription be refunded after 30 days?",
        )
        retrieval.set_attribute("retrieval.top_k", 4)
        retrieval.set_attribute("output.value", "refund-policy-v7 sections 2 and 4")

    with tracer.start_as_current_span("generate-answer") as generation:
        generation.set_attribute("openinference.span.kind", "LLM")
        generation.set_attribute("llm.model_name", "synthetic-chat-model")
        generation.set_attribute("llm.token_count.prompt", 418)
        generation.set_attribute("llm.token_count.completion", 96)
        generation.set_attribute(
            "output.value",
            "The annual plan is outside the standard 30-day refund window.",
        )

provider.force_flush()
provider.shutdown()

print(
    json.dumps(
        {
            "exported": True,
            "trace_name": "support-agent",
            "span_count": 3,
            "elapsed_ms": round((time.perf_counter() - started) * 1000, 1),
            "endpoint": endpoint,
        },
        indent=2,
    )
)
PY

# Phoenix example
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://localhost:4318/v1/traces" \
python emit_trace.py

We include the following output only to illustrate the script’s output format, not as a verified Phoenix run result. The 47.8 ms elapsed value is illustrative and must not be treated as a benchmark:

{
  "exported": true,
  "trace_name": "support-agent",
  "span_count": 3,
  "elapsed_ms": 47.8,
  "endpoint": "http://localhost:4318/v1/traces"
}

In this script, elapsed time covers synthetic span creation, flushing, and shutdown; no model inference occurs. The script also prints exported: true unconditionally, so that field alone does not verify successful trace delivery. For Langfuse, we changed the endpoint to its public OTLP trace route and supplied the authorization header. The application code and span structure remained unchanged.

Langfuse’s software development kit, or SDK, provided ready-made code for working with the product. Its public application programming interface, or API, let our code access product features. We used them for model-generation records, prompt versions, scores, datasets, and links between datasets and runs. In Phoenix, we used its Python client and evaluation packages to create datasets, launch experiments, apply evaluators, and write results back for comparison. We found generic OTLP sufficient for trace portability, but not for every product-specific workflow.

Configuration, Storage, and Tracing Problems We Encountered

Our first Langfuse failure came from a configuration issue, not a tracing issue. One unresolved secret placeholder allowed part of the stack to start while a dependent service repeatedly restarted. The web interface then indicated a generic availability problem rather than showing the original error about the secret. We fixed this by rendering docker compose config, scanning for placeholders, and failing deployment if required values were empty.

Services also became ready at different times during startup. PostgreSQL accepted connections before the application had finished preparing its database structure. ClickHouse and object storage were still starting too. Adding a fixed delay to our continuous integration pipeline did not ensure that every service was ready. We instead checked each supporting service and confirmed that the application could accept requests before sending traces.

ClickHouse created the largest difference in maintenance work. It added another service to operate while keeping trace analysis separate from routine application records. We had to manage its stored data, backups, retention periods, and upgrades. On a developer machine with limited memory, the services needed substantial memory during startup and after we sent a batch of traces. Tight container memory limits caused restarts rather than slower operation. We therefore left spare memory for ClickHouse to combine stored data and for workers to process background jobs.

Phoenix was simpler to start, but SQLite no longer suited our workload once trace uploads, interface queries, and evaluation results arrived together. Concurrent operations competed for database access, so we moved to PostgreSQL. That resolved the immediate problem with competing writes, but maintaining a separate database made the deployment less simple than the single-container demo.

In both tests, our trace sender initially used a different protocol from the receiver. Port 4317 expected gRPC, a protocol for calling functions on another service, but our sender used OTLP over HTTP. Sending HTTP data to the gRPC port failed without a clear explanation. We standardized on explicit receiver addresses ending in /v1/traces, documented the protocol beside every environment variable, and tested trace delivery during deployment.

Both systems stored ordinary OpenTelemetry spans. Their detailed language-model views depended on semantic conventions: shared names for attributes such as model names and token counts. A span named generate-answer did not automatically become a complete generation record. We added OpenInference attributes for span kind, model, inputs, outputs, token usage, retrieval documents, and tool calls. We fixed the versions of the packages that collect traces because changes to attribute names could silently reduce what the interface displayed.

Evaluation execution exposed a workflow difference. Langfuse let us organize dataset items, connect runs to traces, add numeric or categorical scores, and route samples through annotation. That was convenient for product-facing review. We still had to run the programs that scored results, manage model access credentials, handle retries, and control spending ourselves.

Phoenix felt more natural when our evaluator was already written in Python. We could run experiments against a dataset, calculate code-based or model-based metrics, and inspect the resulting examples and traces. We were responsible for ensuring that a notebook or background job could run the same experiment again under the same conditions. We pinned evaluator prompts, package versions, model identifiers, and dataset snapshots; otherwise, an experiment name alone was not enough to reproduce a result.

We also protected both systems from accidental sensitive-data collection. Input and output capture is useful precisely because it records content that may contain personal data, credentials, retrieved documents, or internal instructions. We added redaction to remove or mask sensitive content before export and blocked authorization headers. We hashed stable user identifiers, replacing them with values calculated from the originals. We set how long to keep traces independently of how long we kept ordinary infrastructure monitoring data.

Finally, we did not rely on either interface as the definitive record during an incident. We retained application logs and standard infrastructure telemetry separately. When a tool failed to take in trace data, we could not rely on it to explain the missing spans.

Scale, Latency & Cost vs. Alternatives

For a reproducible benchmark, we would fix the trace count, spans per trace, ingestion rate, and warm-up procedure. We would include short prompts, generated text, token metadata, retrieval attributes, and synthetic scores, then sample aggregate container memory and measure disk growth after background work settled.

The following comparison summarizes deployment and workflow considerations; verified resource and latency measurements are not available in the supplied evidence:

To compare browser timings, we would use the same host, warm the cache, and exclude model inference time. We would also control trace width, payload size, retention, indexing, storage class, and concurrent users.

We would measure both stacks under the same workload before concluding which uses fewer resources. Deployment complexity alone does not establish the size of a resource difference. Those additional requirements were not necessarily wasteful. Operating ClickHouse and background workers, along with prompt management and product workflows, required more maintenance.

We compared both against three realistic alternatives:

1. Plain OpenTelemetry plus Grafana-style tooling. We retained maximum telemetry portability but had to build LLM-specific trace rendering, scoring, datasets, and annotation ourselves.
2. A managed observability service. We avoided database upgrades, backups, and capacity planning. In return, we accepted usage-based charges, reviewed data leaving our infrastructure, and depended more heavily on the provider.
3. An internal trace application. We gained complete control and immediately inherited a permanent product roadmap.

For cost planning, we avoided a misleading per-trace number. Self-hosted expense was dominated by fixed infrastructure, retention, replicas, backups, and engineering time.

For a hypothetical cost calculation—not a measured deployment budget—we could assume $80 for compute plus $20 for storage and backups, without standby servers. These assumptions are not validated sizing or pricing estimates for either product. Production high availability would add database replicas, object-storage charges, monitoring, and restore testing.

Labor changed the calculation. At an engineering cost of $150 per hour, including salary and employer expenses, three maintenance hours per month added $450. Our break-even expression was therefore:

managed monthly bill > self-hosted infrastructure + monthly operations labor + expected incident cost

If a managed service cost $700 per month, a $100 self-hosted stack with $450 of routine labor left only $150 of nominal savings before incidents. If controlling where data was stored, how long it was kept, or how integrations worked had business value beyond cost savings, self-hosting could still win. If cost reduction was the only reason, the break-even point arrived later than the container bill suggested.

Our Final Verdict: When to Deploy, When to Skip

We would deploy Langfuse when:

• We need trace debugging, prompt management, datasets, scores, user feedback, and annotation in one team-facing system.
• We want product managers and application engineers to inspect sessions without living in notebooks.
• We can operate PostgreSQL, ClickHouse, Redis, workers, and object storage responsibly.
• We need SDK-level generation concepts in addition to portable OpenTelemetry spans.
• We have a concrete reason to keep trace content inside our own network boundary.

We would deploy Phoenix when:

• Our services already emit OpenTelemetry or OpenInference traces.
• Our evaluation workflow is primarily Python-driven and experiment-oriented.
• We need to inspect retrieval quality, tool execution, and model behavior together.
• We want a lighter starting footprint and accept PostgreSQL for durable concurrent operation.
• Elastic License 2.0 fits our intended internal use and distribution model.

We would hold off on either self-hosted option when:

• Nobody owns upgrades, backups, retention, redaction, and restore drills.
• The team expects self-hosting to eliminate operational cost.
• Trace volume is unknown and no sampling or retention policy exists.
• The team would collect sensitive prompts and outputs without first reviewing how to classify and protect that data.
• A managed service costs less than the engineering time required to keep the stack reliable.
• The requirement is only infrastructure latency and error-rate monitoring; ordinary OpenTelemetry tooling may already be enough.

Neither tool was the best choice for every use case. We selected Langfuse for workflows where observability sat inside a broader prompt and product-feedback lifecycle. We selected Phoenix for engineering environments where traces and evaluations needed to remain close to OpenTelemetry and Python.

If we had to choose one default for an application team, we would begin with Langfuse and budget honestly for its dependencies. If we had to choose one default for an evaluation-heavy research or platform team, we would begin with Phoenix backed by PostgreSQL.

In both cases, we would keep instrumentation portable, export important datasets, pin versions, and test restoration before calling the deployment production-ready. Teams deciding between these architectures for a real workload can contact us with their trace volume, retention, and evaluation requirements. Those three inputs usually determine the answer faster than a feature checklist.