About 1,800 tests passed. Running the real app found 14 bugs

My last post about Filament Studio was about v1.2.0 and multilingual content, back in April.

Since then I have shipped two larger things: an MCP server, so AI agents can manage collections and records, and Flows, an automation engine inside the plugin. Flows has a visual designer, four trigger types (manual, webhook, collection event, cron), a set of operations (create/update records, HTTP requests, email, conditions, calling another flow), draft/publish versioning, and a step-through debugger.

By the end of August the test suite was at about 1,800 tests. I run mutation testing on top of that, with an 80% MSI target per module. By the numbers I had, Flows was in good shape.

In September I stopped trusting those numbers and did something I should have done earlier. I installed the package into a separate Laravel app and used it the way a user would:

• real HTTPS webhook deliveries
• a real queue worker
• MySQL instead of in-memory SQLite
• the admin panel, clicked through in a browser

That found fourteen defects in one release (v1.8.0), plus one more the day before (v1.7.1). The existing suite covered none of them.

This post goes through the interesting ones. What I took from it: almost every bug had the same shape. The system reported success while doing nothing, or doing the wrong thing.

1. A run that completed in 0 ms

The first bug showed up before I had even started on webhooks.

I opened a seeded "Welcome Email" flow, triggered it, and the run page said Completed. Duration: 0 ms. Steps: none.

Nothing had run, and the engine said everything was fine.

The engine walks the flow as a graph. After each node it asks for the successors on the success branch (or failure). The lookup matched the edge's sourceHandle exactly:

// Before
$matches = $matches->where('sourceHandle', $branch);

The visual designer always sets sourceHandle, because its nodes have named success and failure handles. But a graph written by hand, by a seeder, or through the REST API usually looks like this:

{ "source": "trigger", "target": "send_email" }

No handle. So the walk stopped at the trigger, found nothing to do, and finished. "Finished with nothing to do" was recorded as success.

// After
$matches = $matches->filter(
    fn (array $edge): bool => ($edge['sourceHandle'] ?? 'success') === $branch,
);

Every graph in my tests had the shape the designer produces. The engine had only ever been tested on input from its own UI.

2. A webhook that looked signed but accepted anyone

This is the one I am least proud of.

Flows can be triggered by a webhook, and the default auth mode is HMAC: the sender signs "{timestamp}.{body}" with a shared secret, and the endpoint checks the signature.

A few things combined here:

1. The flow table has a webhook_auth_mode column that defaults to hmac.
2. The secret was generated only when the trigger node's config contained auth_mode: hmac.
3. The trigger's config schema never declared an auth_mode field, so the designer never showed it and nobody could set it.
4. So on the normal path, webhook_secret stayed NULL.
5. The verifier took the secret as a string. (string) null is ''.

The endpoint said it required a signature, but anyone who could compute hash_hmac('sha256', "{ts}.{body}", '') could pass the check. That is anyone.

The fix has two parts. First, the verifier refuses to verify against an empty secret:

if ($secret === '') {
    // Without this guard an unconfigured flow verifies against hash_hmac(..., ''),
    // which any caller can compute — an unauthenticated endpoint that looks signed.
    throw new InvalidWebhookSignatureException('Webhook secret is not configured for this flow.');
}

Second, the trigger now reads the webhook_auth_mode column instead of node config. The webhook trigger node has no config at all anymore. Auth mode, secret, API-key allowlist and redaction paths all live on the flow record, so there is one source of truth.

The real bug was having two places to set one security setting. Each had its own tests, and nothing checked that they agreed.

If you run Flows with a reachable webhook: check for flows where webhook_auth_mode = 'hmac' and webhook_secret IS NULL. Those endpoints were effectively public. After upgrading, publish the flow again and a secret will be generated. Until then it returns 401.

In the same area, webhook_redact_paths redacted the parsed body but not the raw body stored next to it, so a value scrubbed from body still sat in trigger_payload.raw. The raw body is now re-encoded from the sanitized one.

3. Cache::forever is not forever

Collection-event triggers ("run this flow when a record in orders is created") need a fast lookup: given a collection and an event, which flows care?

I kept that index in the cache:

Cache::forever('studio.flows.collection_event_subscriptions', $subscriptions);

When a flow was published, it subscribed. When unpublished, it unsubscribed. The tests passed.

Then I ran php artisan optimize:clear, which plenty of deploy scripts run. Every collection-event flow stopped firing. Nothing was logged and nothing showed in the UI. The only way to recover was to publish every flow again.

"Forever" in a cache means "until something clears it": a deploy, an eviction, a Redis restart. The cache was the only record of the subscriptions, so losing it lost the data.

The fix makes the cache a read-through index over what is actually published:

public function all(): array
{
    $cached = Cache::get(self::CACHE_KEY);

    // An explicitly empty map is a real answer — every flow unsubscribed.
    // Only a missing key means the index was lost and must be rebuilt.
    if ($cached !== null) {
        return $cached;
    }

    $rebuilt = $this->rebuildFromPublishedVersions();
    Cache::forever(self::CACHE_KEY, $rebuilt);

    return $rebuilt;
}

The comment covers a detail I almost got wrong: [] (nobody subscribed) and null (the index is gone) have to be treated differently. Otherwise an install with no subscriptions would query the database on every record save.

Since then, whenever I put something in the cache, I ask: if this key disappears right now, what breaks, and would anyone notice?

4. Config schemas that disagreed with the code reading them

Each trigger declares a config schema. The designer renders a form from it and the API validates against it. Separately, the trigger's runtime code reads config keys.

In three of the four triggers, these didn't match:

A schedule flow configured exactly as its own schema described could not be published.

There were tests for the schemas and tests for the runtime. None of them built a config from the schema and gave it to the runtime.

5. Secrets that were never injected

Flows support per-flow encrypted secrets, used in configs as {{ $secrets.API_TOKEN }}.

The table existed. Encryption at rest worked. FlowContext had a $secrets slot, and the template engine knew how to resolve $secrets.*.

Nothing ever filled the slot. Every {{ $secrets.X }} rendered as an empty string, so any flow using one was sending an empty credential.

Wiring it up was the easy part. The harder part was what the secrets must not leak into:

• Step logs. Values are already masked by key name (/token/i, /password/i, …), but a secret interpolated into a URL or into a field called note gets past key-name matching. Resolved secret values are now scrubbed from every string before a step is saved.
• The step-through cache. A paused debugging session stores its context in the cache store. Secrets are left out of that blob and loaded from the database again when the session resumes.
• Sub-flows. A flow that triggers another flow doesn't pass its secrets along. The child loads its own.

6. The quick ones

Each of these deserves a paragraph, but the pattern is clear by now:

• Form-encoded webhooks wrote blank records, and the run showed as successful. The body was parsed only when $request->isJson(), so {{ $trigger.body.email }} rendered as ''.
• The run timeline listed steps alphabetically. Sorting used started_at, which has second precision. In a fast run every step ties, and MySQL fell back to index order, which was alphabetical by operation key. It now tie-breaks on the UUIDv7 primary key, which increases monotonically.
• Every fast step showed 0ms for the same reason. There is now a duration_ms column measured by the engine.
• A synchronous "trigger another flow" step reported success when the child flow failed. It now takes the failure branch.
• The Studio Dashboard page returned 403 for everyone using spatie/laravel-permission. It checked a permission name that the package never registers.
• The audit log recorded updated for every lifecycle change: draft save, publish, rollback. Now they are draft_saved, published, rolled_back.
• Which API answered depended on service-provider boot order. Collection routes are a catch-all, api/studio/{collection_slug}, so GET /api/studio/flows went to the collection controller when it was registered first.

The route fix gave me a small lesson of its own. My first version excluded reserved segments with a lookahead ending in $:

'(?!(?:flows|webhooks)$)[^/]+'

The regression test failed. Laravel splices the parameter pattern into the regex for the whole path, so $ doesn't mean the end of the segment. /api/studio/webhooks/my-flow still matched. The working version ends the lookahead at a segment boundary:

'(?!(?:flows|webhooks)(?:/|$))[^/]+'

I only caught that because I wrote the test first. That was the rule for this release: every one of the fourteen fixes started with a test that failed.

Two older bugs from the same family

Looking back, I had seen this pattern before and hadn't named it.

Search that returned nothing. In v1.5.0 I fixed record search, which matched nothing for any term. Filament's default searchable() generated where "column" like ? against the records table. But in an EAV model, the values live in another table, and on the record query they exist only as subquery aliases, which a WHERE clause can't reference. On MySQL that throws Unknown column. On SQLite, which my tests used, an unresolved double-quoted identifier quietly turns into a string literal. So the query compared the literal string 'title' to the search term and returned zero rows, and the tests passed.

A dependency I never used. The v1.4.0 changelog said Flows was "built on durable-workflow/workflow". It wasn't. Nothing in the package referenced it. The engine had always walked its own graph and used plain Laravel queued jobs. I removed it in v1.7.0 (three fewer packages and about thirty fewer migrations on every install) and corrected the changelog. It wasn't a runtime bug, but it was the same problem in documentation: a claim nobody had checked against reality.

The pattern

Grouped together, the fifteen or so bugs fall into three categories.

1. Silent success. The system's default outcome was "success", so doing nothing looked the same as doing the job. A 0 ms run, a blank record, a swallowed child failure, an empty credential, a subscription index that disappeared, a search with no results. None of them threw an exception.

2. Two things that should agree, tested separately. Schema vs. runtime. Flow column vs. node config. Designer-shaped graphs vs. API-shaped graphs. Collection routes vs. flow routes. A policy's permission name vs. the permissions that exist. Each side had tests, and the gap between them had none.

3. A test environment gentler than production. SQLite that forgives bad SQL. A cache that never gets flushed mid-test. Requests that are always JSON.

A green suite showed that the code did what the tests said. It said much less about whether the product worked.

What I changed

• An end-to-end pass in a real host app before releases: real queue, real database engine, real HTTP. It's slow and manual, and it found more in a week than the suite had in months.
• A failing test first for every bug, so each bug from this list is now permanently covered.
• Checking where "success" comes from. For each code path that reports success, I ask what it reports when nothing happened.
• Asking what happens if the cache key disappears, every time I write one.

v1.8.0 is out with all of this. Upgrading is php artisan migrate (for duration_ms). There are two behavior changes: a collection with the slug flows or webhooks is no longer reachable through the collection API, and HMAC webhooks without a secret now return 401. The changelog has the details.

What I would like to hear

• Do you run an end-to-end pass against a real host app for your Laravel packages? How much of it is automated?
• Has Cache::forever bitten you, or have you been using the cache as an index from the start?
• For "silent success": do you have a pattern for making "nothing happened" look different from "it worked"?

Repo: https://github.com/serhii-f8/filament-studio
Packagist: https://packagist.org/packages/serhii-f8/filament-studio

If you have a good way to catch "two things that should agree" bugs before a user does, I would like to hear it.