Vizra Evals and Pest's Evals Plugin: When You Want Which

If you are testing AI agents in Laravel, there are now two packages with "evals" in the description, and the obvious question is whether you need both.

Short answer: probably not, and which one depends on a single question. Do you need to know whether your agent is good now, or whether it is worse than it was last month?

This is a genuine comparison rather than a sales pitch. I wrote one of them, and there is a section below telling you when to use the other one on its own.

Start with what they share

Vizra Evals is built on Pest. Not alongside it, not as an alternative to it. Your evals are Pest tests, they live in your test suite, and they run through the Pest binary.

Both packages also use the same activation contract:

./vendor/bin/pest             # evals skipped
./vendor/bin/pest --evals     # evals run against the real model

The --evals flag and the PEST_EVALS environment variable mean the same thing to both. That is deliberate. You can have both installed in one suite and neither will fight the other for the flag.

So this is not a migration decision. Nothing has to be torn out.

What Pest's evals plugin does

pestphp/pest-plugin-evals scores agent output inside an expectation. You point it at an agent, give it a prompt, and assert on the response:

expect(CapitalCityAgent::class)
    ->prompt('What is the capital of Japan?')
    ->toBeCorrect(expected: 'Tokyo');

It has a genuinely broad set of expectations. Deterministic ones that cost nothing: toContain(), toMatch(), toBe(), toBeJson(), toHaveToolCalls() and toFollowTrajectory(), which checks tools were called in a given order. Then scored ones that call a model: toBeCorrect(), toBeRelevant(), toBeSafe(), toBeSimilar() for semantic similarity via embeddings, toSatisfy() for plain-English criteria, and toPassScorer() for your own.

Scorers return 0.0 to 1.0 against a configurable threshold. toBeCorrect() is more nuanced than a boolean: an exact match scores 1.0, approximately equal 0.9, a superset of the expected answer 0.8, a subset 0.6, and a contradiction 0.0.

It handles nondeterminism too. repeat() runs the same prompt several times and asserts every expectation against all of the samples, so a test only passes when the agent is consistent. That is the right instinct and a lot of eval tooling misses it.

prompt() also takes a plain closure, not just an agent class, so you can score anything that returns a string.

This is a well-built plugin. If you have read this far expecting me to find fault with it, that is not where this is going.

What Vizra Evals adds

One thing, and everything else follows from it: it keeps the runs.

Every sample, every score, every judge's reasoning, every tool call, every token cost, written to your own database. That turns a passing test into a data point rather than an event.

Once runs are persisted, three things become possible that are not possible from a single assertion.

A baseline. Your first passing run becomes the reference. Everything after it is measured against that rather than against a fixed threshold you guessed at.

Row-level regression detection. Rows are joined across runs by a hash of their content, so the framework knows that "What is your refund policy?" in today's run is the same row as in last month's. When that specific row drops, the build fails and names it:

Eval [pest: answers support questions from documented policy] score 61.7%,
pass rate 33.3% across 6 samples.
Gate failed: 2 rows regressed against the reference run (allowed: 0).
  ↓ regressed: "What is your refund policy?" 96.7% → 51.7%
  ↓ regressed: "Can I return it?" 93.3% → 55.0%

Not "the suite scored 61.7%". Which two inputs got worse, and by how much. That difference matters most in a pull request, where the useful question is never "is this good" but "is this worse than what we had".

History you can look at. A second package, vizra/evals-ui, mounts a dashboard route inside your own app: score trends per suite, any sample's assertions and judge reasoning one click deep, two runs side by side. It reads the same database your evals already write to.

The eval itself looks like this:

it('answers support questions from documented policy', function () {
    expect(SupportBot::class)->toPassEval(fn ($eval) => $eval
        ->dataset(base_path('evals/support.jsonl'))
        ->samples(3)
        ->assert(fn ($a, $row) => $a
            ->notEmpty()->gate()
            ->contains($row->expected())
            ->costBelow(0.02))
        ->judge('Answers using only documented store policy.', min: 7)
        ->gate(minScore: 0.8, maxRegressions: 0)
    );
});

Note maxRegressions: 0. That parameter cannot exist without stored history, and it is the whole argument for this package in one keyword.

When you want Pest's plugin on its own

Plenty of the time. Genuinely.

You have a handful of agents and a handful of checks. If what you need is "this classifier returns valid JSON" or "this agent never recommends a competitor", that is an expectation, and adding a database and a dashboard to it buys you nothing.

You want zero extra infrastructure. Vizra needs a migration. Pest's plugin needs nothing. If you are evaluating in a package rather than an application, or in something with no database at all, that is close to decisive.

You are exploring rather than shipping. Working out whether the prompt is any good in the first place is a different activity from defending it over time. Expectations are the faster loop.

Your agent's quality genuinely does not drift. Some don't. A structured extraction task with a pinned model and a stable prompt can sit still for a year. If nothing changes, there is nothing to compare.

You don't have a baseline worth defending yet. Baselines are only useful once you have a version you are happy with. On week one you are still finding it.

In all of those, the honest recommendation is to use Pest's plugin and stop reading.

When the baseline starts to matter

The switch tends to flip at a specific moment, and it is usually a question someone asks you.

"Did the answers get worse after we changed the model?"

With expectations alone, the answer is a shrug, or a threshold that either failed or didn't. You cannot compare against a number you never wrote down.

Concretely, the baseline earns its keep when:

• More than one person changes the prompts. You want the reviewer to see which rows moved, not to take someone's word that it seems fine.
• You are changing retrieval. Swapping an embedding model, changing chunk size, adding a reranker. These shift quality in ways that look fine on the three examples you happen to test and quietly get worse on the fiftieth.
• You need to explain a regression weeks later. "It got worse sometime in the last month" is a very different investigation from "it got worse on the 14th, on these two rows, and here is what the judge said about each".
• Cost is creeping. Per-sample cost is recorded, so you can see it climb rather than discover it on an invoice.
• Something is at stake. Regulated work, customer-facing answers, anything where "we think it's fine" is not an acceptable answer to an auditor.

Running both

There is no conflict. A reasonable setup uses Pest's expectations for quick assertions during development, and Vizra evals for the suites you want to defend over time. Both respond to --evals, so one command runs everything.

The docs cover running them side by side if you want the detail.

The summary

If you are starting today, start with Pest's plugin. It is less to install and it answers the first question you will have.

Come back to this when someone asks whether the answers got worse, and you realise you have no way to know.

If you want to see what kept history looks like before installing anything, there is a live demo with six weeks of runs behind it.