How to Test AI Agents in Laravel (Beyond Fakes)

Your test suite is green. All twelve AI tests pass, none of them touch a provider, and the whole run takes 200ms. Then someone tightens a system prompt on Tuesday, and three weeks later a customer forwards you a support reply that confidently invented a refund policy you do not have.

Every test still passed. They were never testing for that.

This post is about the gap: what fakes are genuinely good at, what they structurally cannot tell you, and how to test the part they leave out.

Fakes are good, and you should use them

The Laravel AI SDK ships a faking layer that works like every other test double in the framework. You define responses up front and assert the right prompts went out:

use Laravel\Ai\Facades\Ai;

Ai::fakeAgent(SupportBot::class, ['We offer a 30 day return window.']);

$this->post('/chat', ['message' => 'Can I return this?'])
    ->assertOk();

Ai::assertAgentWasPrompted(SupportBot::class, fn ($prompt) =>
    str_contains($prompt->text(), 'return')
);

There is a good Laravel News tutorial on this, and it is worth your time. preventStrayPrompts() in particular is the sort of thing you want on from day one. It fails the test if a code path calls a model you did not expect, which is how you find the accidental LLM call inside a loop before your bill does.

What fakes give you is fast, free, deterministic tests of your code. Does the route authenticate. Does the tool get called with the right arguments. Does the job get queued. Does the response get persisted. Every one of those is your logic, and every one of those should be tested with fakes, in CI, on every commit.

What a fake cannot tell you

A fake returns the answer you wrote. That is the entire point of it, and it is also the boundary.

When you assert that SupportBot returned "We offer a 30 day return window", you are asserting that you typed that string into the test. The model was never consulted. So the test passes identically whether the real agent is excellent, mediocre, or has been quietly degraded by a prompt change three commits ago.

Two different questions, and it is worth being blunt about which is which:

• Does my code work? Fakes. Fast, free, deterministic, run on every commit.
• Is the answer any good? Fakes cannot help. Nothing in a normal test suite can.

Most teams test the first and assume the second. It holds right up until it doesn't, and the way you find out is a customer.

Why you cannot just write a normal test for it

The obvious next thought is to skip the fake and call the real model. That breaks immediately, for a reason that is easy to state and easy to underestimate.

The same input produces a different answer every time.

Ask an agent the same question five times and you get five phrasings, sometimes five different levels of correctness. So assertEquals is useless, and assertStringContainsString is a coin flip you have dressed up as a verdict. It passes or fails depending on which roll you happened to run in CI.

Worse, a single run tells you nothing about the distribution. An agent that gets it right 95% of the time and one that gets it right 55% of the time both look identical if you sample them once and get lucky. The thing you actually care about is the shape of the answers, not one of them.

That is what an eval is: run each input several times, score every response, and look at the distribution rather than a single sample.

Writing one

Vizra Evals is an MIT-licensed package that does this as Pest tests. Install it as a dev dependency:

composer require vizra/evals --dev
php artisan migrate

An eval looks like a test, because it is one:

use App\Agents\SupportBot;

it('answers support questions from documented policy', function () {
    expect(SupportBot::class)->toPassEval(fn ($eval) => $eval
        ->dataset(base_path('evals/support.jsonl'))
        ->samples(3)
        ->assert(fn ($a, $row) => $a
            ->notEmpty()->gate()
            ->contains($row->expected())
            ->costBelow(0.02))
        ->judge('Answers using only documented store policy.', min: 7)
        ->gate(minScore: 0.8, maxRegressions: 0)
    );
});

The dataset is one JSON object per line:

{"input": "What is your refund policy?", "expected": "30 days"}
{"input": "Can I return a sale item?", "expected": "not eligible"}

And crucially, this does not run in your normal suite:

./vendor/bin/pest             # evals skipped, zero tokens, zero cost
./vendor/bin/pest --evals     # evals run against the real model

That separation matters more than it looks. Evals cost real money and take real time. A test suite that quietly spends $4 every time a junior runs pest is a test suite people stop running.

Three details worth understanding

samples(3). Every row runs three times. This is the whole premise. One sample proves nothing about a nondeterministic system, and three is the smallest number that gives you a mean and a sense of spread. Push it higher for rows you care about most.

->gate(). If notEmpty() fails, everything after it is skipped for that sample, including the judge. A response that came back empty is already broken; sending it to another model to be scored is spending tokens to confirm what you know. Order your cheap deterministic checks first and gate them.

judge(...) is for the things you cannot assert with string matching. "Did it stay within documented policy" is not a substring check. A judge is a second model scoring the first, returning a structured {score, reasoning} rather than prose you have to parse.

The judge needs its own scepticism

LLM-as-judge is the part people adopt fastest and check least. Three things to know before you trust a number it produced.

It grades its own family leniently. Point the judge at a different model family than the agent under test. It is the cheapest correctness win available and it takes one argument.

It rewards length. Longer answers score higher whether or not they are better. If concision matters to you, say so in the criteria, or you are quietly training your agent toward waffle.

An uncalibrated judge is an opinion with a number attached. Label fifty examples by hand, run the judge over them, and measure agreement:

php artisan evals:calibrate storage/labelled.jsonl --criteria="Correctness"

If it disagrees with you 30% of the time, its scores are not data. You will make decisions on those numbers for months, so it is worth an afternoon to find out whether they mean anything.

The part that actually catches regressions

Everything above scores today's run. On its own that is a report card, and a report card with no previous term is close to useless. You have no idea whether 71% is good.

The point is comparison. Your first passing run becomes the suite's baseline. Rows are joined across runs by a hash of their content, so the framework knows that "What is your refund policy?" in today's run is the same row as in last month's. When a specific row drops, the build fails and names it:

Eval [pest: answers support questions from documented policy] score 61.7%,
pass rate 33.3% across 6 samples.
Gate failed: 2 rows regressed against the reference run (allowed: 0).
  ↓ regressed: "What is your refund policy?" 96.7% → 51.7%
  ↓ regressed: "Can I return it?" 93.3% → 55.0%

Not "a test failed". Which inputs got worse, and by how much. That is the output you want in a pull request comment when someone has rewritten a prompt.

Where Pest's own evals plugin fits

Pest ships an evals plugin, and it does the scoring well: toBeCorrect(), toBeRelevant(), toBeSafe(), toSatisfy(), toHaveToolCalls(), toFollowTrajectory(), and repeat() to sample the same prompt several times. If what you need is to assert an agent is behaving right now, that is genuinely all you need, and you should use it.

Vizra Evals sits on top of that rather than against it. The difference is what happens after the run finishes. Every sample is persisted, rows are joined across runs by content hash, and a baseline lets the build fail when one specific row drops. Pest scores a run. This keeps them and compares them.

The two share the same --evals flag and PEST_EVALS environment variable, so both can live in one suite without fighting over the flag. The docs cover running them together.

So which do you use

Both, for different jobs, on different schedules.

Fakes belong in your normal suite and should run constantly. Evals belong on a slower loop: a nightly job, a pre-release gate, or a CI workflow triggered when anything under app/Agents or your prompt files changes.

If you only take one thing from this: the tests you have prove your plumbing works. They were never designed to tell you whether the thing coming out of the pipe is any good, and no amount of adding more of them will change that.

Where to start

Pick your most important agent. Write down ten inputs where you know what a good answer looks like. Real ones, from support tickets or logs, not invented ones. That file is your first dataset, and it is genuinely the hardest part of the whole exercise. Everything after it is mechanical.

If you want to see what the output looks like before installing anything, there is a live demo with real history behind it. The quickstart takes about five minutes, and the source is on GitHub.