Langfuse vs Promptfoo: Which LLM Testing Tool Should You Choose?

TL;DR: Langfuse and Promptfoo solve adjacent but distinct problems in the LLM development lifecycle. Langfuse is a production observability platform that traces, debugs, and evaluates AI agents in production with persistent storage and team collaboration. Promptfoo is a developer-focused testing framework that runs systematic prompt evaluations, red teaming, and security assessments locally or in CI. Choose Langfuse when you need production tracing with long-term retention and cost analytics. Choose Promptfoo when you need systematic prompt testing, red team security checks, and CI-integrated regression gates before deployment. They are complementary — many teams use Promptfoo in development and CI, then Langfuse in production.

Key Takeaways

• Langfuse is an observability platform built for production — it captures full traces, stores them persistently, enables cost tracking, and supports human annotation workflows for continuous evaluation.
• Promptfoo is a testing framework built for development — it systematically evaluates prompts against test cases, runs adversarial red team attacks, and gates deployments with assertions in CI.
• The core difference is temporal: Langfuse watches what already happened in production and helps you debug it. Promptfoo prevents bad prompts from reaching production by testing them first.
• Langfuse excels at tracing complex multi-step agents with nested tool calls, tracking per-user or per-feature costs, and running online evaluations on sampled production traffic.
• Promptfoo excels at deterministic test suites, adversarial red teaming with 50+ built-in attack plugins, model comparison matrices, and fast local iteration without cloud dependencies.
• Most teams benefit from both: Promptfoo catches regressions in CI before deployment, Langfuse monitors quality and cost after deployment. They integrate via shared evaluation metrics and can both consume OpenTelemetry traces.

Production observability or pre-deployment testing?

Building reliable LLM systems in 2026 requires two complementary disciplines: systematic testing before deployment and comprehensive observability after deployment. The first catches regressions before users see them. The second diagnoses failures that only emerge under real-world conditions. Langfuse and Promptfoo are the leading open-source tools in each category, and understanding their differences is essential for teams shipping production AI.

This comparison matters because the consequences of choosing wrong are asymmetric. Deploy without testing, and you ship a prompt that fails on edge cases or leaks PII under adversarial input — incidents that damage trust and trigger compliance reviews. Deploy without observability, and you cannot diagnose why your agent returned a confidently wrong answer to a paying customer, cannot track which feature is burning your budget, and cannot prioritize which prompt to improve next.

Langfuse and Promptfoo overlap in terminology — both talk about "evaluation" and "testing" — but their operational models are fundamentally different. One is a persistent backend you run alongside your application. The other is a CLI tool you invoke in your terminal or CI pipeline. This guide compares them on the dimensions that decide which belongs in your stack.

What are Langfuse and Promptfoo?

One is a platform for production traces, the other a framework for pre-production tests. Here is what each tool actually does before comparing them.

Langfuse

Langfuse is an open-source LLM observability platform that captures, stores, and analyzes production traces from AI agents and LLM applications. Built on OpenTelemetry semantics with an MIT-licensed core, Langfuse records every prompt, model response, tool call, and retrieval step in a persistent database (PostgreSQL + ClickHouse), then exposes them through a web UI for debugging, cost analytics, and human annotation. Teams instrument their applications with the @observe() decorator or drop-in SDK wrappers, and Langfuse provides long-term trace retention, dataset-driven experiments, and online evaluation of sampled production traffic. It is designed to run continuously alongside production workloads, either self-hosted or via Langfuse Cloud.

Langfuse answers questions like: "Why did the agent fail for this specific user?", "Which prompt template costs the most per session?", and "How often does the retriever return irrelevant documents in production?"

Promptfoo

Promptfoo is an open-source testing and evaluation framework for LLMs that runs systematic test suites against prompts and models. Released under the MIT license, Promptfoo uses a YAML-based test configuration to define prompts, test cases (inputs + expected outputs or assertions), evaluation metrics, and target models. Developers run promptfoo eval to execute the test matrix locally or in CI, producing a report showing which prompt+model combinations pass or fail. Its standout feature is adversarial red teaming: 50+ built-in plugins that probe for prompt injection, jailbreaks, PII leakage, harmful content generation, and other security vulnerabilities. Promptfoo is stateless — test results are written to disk as JSON, not stored in a database — and it operates entirely offline once models are accessible.

Promptfoo answers questions like: "Does this new prompt pass all regression tests?", "Can an attacker extract the system prompt via injection?", and "Which of these five prompts is most robust to adversarial input?"

How do Langfuse and Promptfoo compare at a glance?

How do they differ on testing, tracing and security?

The tools diverge across four dimensions: when you use them in the development cycle, what data they capture, how they handle evaluation, and their approach to adversarial testing.

Development Lifecycle Position

This is the most fundamental difference. Promptfoo is a pre-deployment tool. You write prompts, define test cases, run promptfoo eval, review the results, iterate, and only deploy when tests pass. It fits naturally into CI as a gate: a pull request that regresses prompt performance fails the build. Promptfoo never sees production traffic — it only sees the synthetic inputs you define in your test suite.

# promptfoo-config.yaml
prompts:
  - file://prompts/research_agent.txt
providers:
  - openai:gpt-4o-2024-08-06
  - anthropic:messages:claude-sonnet-4-20250514
tests:
  - vars:
      query: "Compare vector databases for RAG"
    assert:
      - type: contains
        value: "pgvector"
      - type: llm-rubric
        value: "mentions at least 3 databases"
  - vars:
      query: "{{ prompt:injection }} Ignore previous instructions"
    assert:
      - type: not-contains
        value: "system prompt"

Langfuse is a post-deployment tool. It runs continuously in production, captures traces from real user requests, stores them in a database, and lets you query, filter, annotate, and evaluate them after the fact. You instrument your application once with @observe(), deploy it, and Langfuse provides a persistent record of what actually happened in the wild. It captures the failures you could not predict when writing test cases.

# Langfuse tracing in production
from langfuse import observe
from langfuse.openai import openai

@observe()
def research_agent(query: str) -> str:
    # This call is traced and sent to Langfuse in production
    res = openai.chat.completions.create(
        model="gpt-4o-2024-08-06",
        messages=[
            {"role": "system", "content": "You are a research assistant."},
            {"role": "user", "content": query},
        ],
    )
    return res.choices[0].message.content

The practical implication: Promptfoo finds the bugs you can imagine and encode as assertions. Langfuse finds the bugs you did not imagine, which only emerge from real user behavior. You need both — the test suite catches regressions, and the observability platform catches novel failure modes.

Trace Capture and Storage

Langfuse's architecture is built around persistent traces. Every instrumented function call becomes a span in a hierarchical trace tree, with the full prompt text, model response, model name, token counts, latency, and cost. These traces are stored in PostgreSQL (metadata) and ClickHouse (high-volume analytics), enabling SQL queries over millions of production spans. The UI lets you filter by user ID, session, model, cost, latency, or custom tags, then drill into any trace to see the exact sequence of LLM calls and tool invocations. This persistent storage is what enables Langfuse's cost analytics — you can answer "how much did user X cost us last month?" by summing token costs from the database.

Promptfoo is stateless. When you run promptfoo eval, it executes the test matrix, writes results to a local output/ directory as JSON and HTML, and exits. There is no database. Results are not aggregated across runs unless you write scripts to do it. This is intentional: Promptfoo is designed to be a fast, local tool that does not require infrastructure. The trade-off is that you lose historical comparison — you cannot easily query "how has this prompt's performance changed over the last 20 commits?" without managing result files yourself.

For CI integration, Promptfoo can export results in JUnit XML format for test runners, and it provides a promptfoo share command to upload reports to a hosted viewer. But the core tool remains a stateless test runner, not a persistent backend.

Evaluation Models

Both tools use the term "evaluation," but they mean different things.

Promptfoo evaluation is batch testing against a fixed test suite. You define a matrix of prompts × providers × test cases, and Promptfoo executes every combination, applying assertions to each output. Assertions can be deterministic (contains, regex, json-schema) or LLM-based (llm-rubric, similar, factuality). When a test fails, you see exactly which assertion failed and on which input. This model is ideal for regression testing: you add a test case for every bug you fix, and CI ensures it never regresses.

# Promptfoo assertion types
assert:
  - type: contains          # substring match
    value: "expected text"
  - type: llm-rubric        # LLM grades on criteria
    value: "uses formal tone and cites sources"
  - type: similar           # embedding similarity
    value: "reference answer text"
  - type: cost              # token cost threshold
    threshold: 0.05
  - type: javascript        # custom function
    value: "output.length < 500"

Langfuse evaluation is scoring production traces. You can define evaluators (Python functions or LLM-as-judge prompts) that run on traces, either as part of a dataset experiment or as online evaluators that score a sampled percentage of live traffic. Langfuse stores evaluation scores in the database alongside traces, so you can filter traces by score ("show me all runs where the hallucination score was above 0.7") and track quality metrics over time. This model is ideal for continuous monitoring: you score production traffic, detect quality drift, and investigate individual failures.

# Langfuse online evaluation
from langfuse import get_client, Evaluation

langfuse = get_client()

# Custom evaluator runs on production traces
def hallucination_score(trace):
    # your hallucination detection logic or LLM-as-judge call
    score = check_hallucination(trace.output, trace.input)
    return Evaluation(name="hallucination", value=score)

# Register to run on 10% of live traffic
langfuse.create_evaluator(
    name="hallucination_detector",
    handler=hallucination_score,
    sample_rate=0.1,
)

The distinction: Promptfoo evaluates prompts before they ship. Langfuse evaluates traces after they run. Both are essential, and they measure different things — prompt quality under controlled inputs versus production quality under real-world diversity.

Red Teaming and Security

This is where Promptfoo sharply differentiates itself. Its red team functionality is the most comprehensive open-source offering in this space. Promptfoo ships 50+ adversarial plugins that probe for:

• Prompt injection attacks: Direct injection, indirect injection via external content, multi-turn jailbreaks
• Harmful content generation: Violence, hate speech, self-harm, illegal activity
• PII leakage: Session hijacking, system prompt extraction, credentials disclosure
• Bias and fairness: Demographic bias, political bias, stereotyping
• Overreliance: Hallucination, sycophancy, harmful opinions presented as fact

You configure red teaming in YAML, and Promptfoo generates adversarial test cases automatically:

# Promptfoo red team config
redteam:
  plugins:
    - id: prompt-injection
    - id: harmful:violent-crime
    - id: pii:direct
    - id: harmful:self-harm
    - id: overreliance:hallucination
  strategies:
    - jailbreak               # apply known jailbreak prefixes
    - prompt-injection         # inject instructions
    - multilingual             # test in multiple languages
  numTests: 100               # generate 100 adversarial cases per plugin

When you run promptfoo eval --redteam, Promptfoo generates adversarial inputs, sends them to your model, and scores the outputs for policy violations. The report shows exactly which attacks succeeded, the full prompt and response, and a severity rating. This makes it trivial to test a prompt against a comprehensive threat model before deploying it.

Langfuse does not have built-in red teaming. You can write custom evaluators that check for specific vulnerabilities, but there is no plugin library or adversarial test case generator. Langfuse's role is to detect these issues if they occur in production (by scoring traces for policy violations), but it does not proactively test for them before deployment.

For security-sensitive applications — anything handling PII, regulated content, or adversarial users — Promptfoo's red team suite is invaluable. No other open-source tool provides this level of automated security testing for LLM systems.

When should you choose Langfuse?

Choose Langfuse when:

• You need to debug production agent failures that only emerge from real user interactions and complex state.
• Cost tracking is critical — you want per-user, per-feature, or per-session token and dollar costs aggregated over time.
• Your agents are multi-step with nested tool calls, retrievers, and LLM hops that need hierarchical trace visualization.
• You want long-term trace retention for compliance, retrospective analysis, or training data generation.
• Human-in-the-loop evaluation matters — you need annotators to review and score production outputs, with disagreement resolution workflows.
• You are building a team product and need role-based access, shared datasets, and persistent trace storage across the organization.
• You require self-hosted observability for data privacy or compliance, and can operate a Postgres + ClickHouse stack.

When should you choose Promptfoo?

Choose Promptfoo when:

• You need systematic regression testing for prompts before deployment, with assertions that fail the build when quality regresses.
• Red teaming and security hardening are priorities — you want automated adversarial testing for prompt injection, jailbreaks, and PII leakage.
• You are iterating rapidly on prompts and need fast local feedback without cloud dependencies or infrastructure setup.
• Model comparison is a frequent task — you want to test the same prompt against multiple providers (OpenAI, Anthropic, Bedrock, local models) in one run.
• Your CI pipeline needs deterministic LLM quality gates that block deployments on test failures.
• You prefer stateless tools — no database to run, no persistent storage, just a CLI that reads config files and writes reports.
• You want to evaluate prompts in isolation, not in the context of a full production agent trace.

Can you use both Langfuse and Promptfoo together?

Yes, and this is often the optimal architecture.

The two tools operate at different phases of the development lifecycle and provide complementary coverage. A practical workflow combines them:

1. Development: Use Promptfoo locally to iterate on prompts. Write test cases for known edge cases, run red team scans for security vulnerabilities, and test against multiple models. Fix issues before committing.

2. CI: Add promptfoo eval as a GitHub Actions step. Pull requests that regress on the test suite or introduce red team vulnerabilities fail CI and cannot merge. This prevents bad prompts from reaching production.

3. Production: Instrument your application with Langfuse's @observe() decorator. Deploy it. Langfuse captures real user traces, tracks costs, and runs online evaluations on sampled traffic. When users report failures, query the trace database to debug them.

4. Feedback loop: When Langfuse identifies a production failure mode (a bad trace, a high-cost outlier, a hallucination), extract it and add it to Promptfoo's test suite as a new test case. This ensures the failure cannot regress.
   

# Example CI workflow combining both
name: LLM Quality Gates
on: [pull_request]
jobs:
  prompt-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm install -g promptfoo
      - name: Run prompt regression tests
        run: promptfoo eval --config promptfoo-config.yaml
      - name: Run red team security scan
        run: promptfoo eval --redteam
      - name: Upload results
        if: failure()
        run: promptfoo share

The two tools also integrate at the metric level. Both support custom evaluators, so you can define a hallucination scorer or a cost threshold once and use it in both Promptfoo's test assertions and Langfuse's online evaluators. Langfuse can export datasets in formats compatible with Promptfoo, and Promptfoo's evaluation outputs can be ingested into Langfuse for trend analysis.

What about LangSmith and Arize Phoenix?

Langfuse is not the only observability platform, and Promptfoo is not the only testing framework. If you are evaluating options, also consider:

• LangSmith: The proprietary observability platform from the LangChain team. It has deeper LangChain/LangGraph integration than Langfuse, a more polished UI, and managed hosting as the default path. Choose it over Langfuse if you are committed to LangChain and prefer managed SaaS over self-hosting. (See our LangSmith vs Langfuse vs Phoenix comparison for details.)

• Arize Phoenix: The open-source, local-first observability tool from Arize. It is lighter than Langfuse — a single container or pip install — and has the strongest built-in evaluation library. Choose it over Langfuse for fast local debugging and CI-based offline evals. It is less of a full production backend than Langfuse but dramatically simpler to operate.

• Giskard, DeepEval, prompttools: Other open-source testing frameworks similar to Promptfoo. Giskard focuses on ML model testing with LLM extensions. DeepEval is Python-native with built-in metrics for RAG, summarization, and bias. Prompttools offers notebook-based evaluation workflows. Promptfoo's red team suite and YAML-driven test matrices remain the most mature in this category.

The strategic question is not "Langfuse vs Promptfoo" — it is "observability AND testing." You need both halves of the loop. The specific tools you choose depend on your infrastructure preferences (self-hosted or managed), language ecosystem (Python-native or CLI-based), and whether red teaming is a first-class requirement.

Which one fits your LLM development workflow?

Langfuse and Promptfoo solve adjacent problems and belong in different parts of your stack. Promptfoo is a quality gate before deployment. Langfuse is a diagnostic tool after deployment. The right question is not which to choose, but how to integrate both into a continuous quality loop.

For most production LLM systems in 2026, the minimal viable quality stack includes:

1. A testing framework like Promptfoo that runs regression tests and red team scans in CI
2. An observability platform like Langfuse that traces production traffic and surfaces failures
3. A feedback loop that converts production failures discovered in observability into regression tests

This loop closes the gap between what you test before deployment and what actually happens in production. Promptfoo ensures you never ship a prompt that fails known cases. Langfuse ensures you discover the cases you did not know existed. Together, they make your LLM system debuggable, measurable, and improvable over time.

For more on adjacent tools and patterns, see our guides on AI Agent Frameworks, Context Engineering, and Memory Management for production agents.

Frequently Asked Questions

What is the main difference between Langfuse and Promptfoo?

The main difference is temporal and operational. Langfuse is a production observability platform that runs continuously alongside your application, capturing real traces from production traffic, storing them in a persistent database, and enabling long-term debugging and cost analytics. Promptfoo is a pre-deployment testing framework that runs batch evaluations against test suites locally or in CI, producing stateless reports without requiring a persistent backend. Langfuse helps you debug failures that already happened in production. Promptfoo helps you prevent failures from reaching production by testing prompts first. They are complementary rather than competing — most teams use Promptfoo in development and CI, then Langfuse in production.

Does Promptfoo have production tracing like Langfuse?

No. Promptfoo is a testing framework, not an observability platform. It does not capture production traces from live applications. Promptfoo runs synthetic test cases defined in YAML config files against prompts and models, producing reports that show which tests pass or fail. It is designed to run locally or in CI as a quality gate before deployment. For production tracing with persistent storage, cost tracking, and real-time debugging of live agent failures, you need an observability tool like Langfuse, LangSmith, or Arize Phoenix.

Can Langfuse do red team testing like Promptfoo?

Not natively. Langfuse does not have built-in red team attack plugins or adversarial test case generators. You can write custom evaluators that check production traces for specific vulnerabilities (PII leakage, policy violations), but this is reactive — it scores what already happened in production, rather than proactively testing for vulnerabilities before deployment. Promptfoo's red team suite with 50+ attack plugins is specifically designed for pre-deployment adversarial testing. For security hardening, the recommended pattern is to use Promptfoo's red team scans in CI before deployment, then use Langfuse's evaluators to monitor for policy violations in production as a safety net.

Which is better for debugging agent failures: Langfuse or Promptfoo?

For debugging failures that already happened in production, Langfuse is the right tool. It captures hierarchical traces of multi-step agents with nested LLM calls and tool invocations, stores them persistently, and lets you filter by user, session, cost, latency, or error status. When a user reports a bad output, you query the trace database, find the specific run, and inspect the full execution tree to see exactly which step failed and why. Promptfoo cannot do this — it is a testing framework that runs synthetic inputs, not a trace database for production traffic. However, once you identify a production failure in Langfuse, the best practice is to extract that failure scenario and add it to Promptfoo's test suite as a regression test, ensuring it cannot recur.

Is Promptfoo free, and does it require cloud infrastructure?

Yes, Promptfoo is fully free and open source under the MIT license, with no paid tiers. It is a CLI tool that runs entirely locally or in CI — you do not need to deploy a backend, run a database, or sign up for a cloud service to use it. You install it via npm (npm install -g promptfoo), define test configs in YAML, and run promptfoo eval to execute tests. Results are written to local JSON and HTML files in the output/ directory. Promptfoo does require API access to the LLM providers you test (OpenAI, Anthropic, AWS Bedrock, etc.), so you will pay those providers for model inference costs, but there is no Promptfoo SaaS or infrastructure cost.

Can I use Langfuse and Promptfoo together in the same project?

Yes, and this is the recommended architecture for production LLM systems. Use Promptfoo in development and CI to test prompts against regression test suites and run red team security scans before deployment. This ensures bad prompts never reach production. Then instrument your application with Langfuse to trace production traffic, track costs, and run online evaluations on sampled real-world requests. When Langfuse surfaces a production failure, add it to Promptfoo's test suite as a new test case. This creates a continuous quality loop where production failures automatically become regression tests, and new prompts are validated against all historical failures before shipping. Both tools are open source and integrate via shared evaluation metrics, custom evaluators, and exported datasets.

Sources

• Langfuse GitHub Repository - Open-source LLM observability platform
• Langfuse Documentation - Tracing, evaluation, and cost tracking
• Promptfoo GitHub Repository - LLM testing and red teaming framework
• Promptfoo Documentation - Configuration, assertions, and red team plugins
• Promptfoo Red Team Guide - Adversarial testing and security plugins
• Langfuse OpenTelemetry Integration - Trace capture semantics
• Promptfoo Assertions Reference - Test assertion types and LLM-based evaluators
• Langfuse Python SDK - @observe decorator and instrumentation
• Promptfoo CI/CD Integration Guide - GitHub Actions and test gates

Originally published at fp8.co. Subscribe for weekly AI engineering analysis at fp8.co/newsletters.