vibeaudit: 11 static checks for the security holes AI-built apps ship with

The last few years of app building went like this: you describe what you want, a model writes the code, the code runs, you ship it. The tooling got dramatically faster; the verification step did not happen at all, and nobody - including the model that wrote the code - was checking.

The numbers are not theoretical. A June 2026 study on DEV.to scanned 62 Lovable apps: 63% had critical or high severity vulnerabilities, 10 findings per app on average. A separate study of 400 public Lovable, Bolt and v0 repositories, published by VibeSafe, found 59% had at least one critical issue and 18.5% had committed a .env file to git.

These are not random bugs. The same handful of mistakes keep showing up, in the same order of severity. That makes them automatable, and it is why I built vibeaudit.

What vibeaudit actually does

vibeaudit is a command line tool. Point it at a project folder and it reads the code, the SQL migrations, the git history and the build output, then produces a ship-readiness report.

Three properties mattered more to me than the detection itself:

• Zero runtime dependencies. Node 20+ and the standard library only. No npm install, no build step, no package tree to trust on a client's machine. The only outbound call is npm audit.
• Every finding carries a fix, not just a label. File, line, CWE reference, the offending code with the secret redacted, the exact replacement, and a paste-ready prompt.
• It reports what it did not find. Passed checks and skipped checks are both listed, with the reason. "Skipped" is never reported as "passed".

The 11 checks, grouped by what they protect

Three checks cover secrets because a key can leak three ways: into a committed file, into source, into the JavaScript every visitor downloads. Deleting the file is not enough, which is why secrets-source also reads the git history - a secret that was committed and later removed is still recoverable, and still valid. Detection is 16 credential patterns, plus JWT handling that decodes the payload to tell a harmless anon key from a deadly service_role key. The database pair is the highest-value part of the tool and the part a linter does not do: with Row Level Security off, the Supabase REST endpoint returns every row to anyone holding the public anon key - which ships inside your app by design.

The scoring model

The score starts at 100 and every finding deducts points:

score = max(0, 100 - 25*critical - 12*high - 4*medium - 1*low)

The weights are chosen so that four criticals, or one critical plus a few highs, can sink a project on their own. A straight subtraction is trivially explainable to a client ("each critical finding costs 25 points") and deterministic, so two audits produce comparable numbers.

One rule sits on top of that: a single critical finding caps the verdict at "Do not ship yet", whatever the score. A score must never be able to hide a problem that puts customer data at risk.

What it actually prints

Run against the bundled demo project - a small, deliberately broken app - here is the real output:

Ship readiness: 0/100 - Unsafe to ship

| Critical | High | Medium | Low |
|---|---|---|---|
| 12 | 20 | 7 | 2 |

And the plain-English paragraph that follows it, verbatim:

We checked 16 files in your project for the security mistakes that AI-built apps make most often - leaked passwords and keys, database tables that anyone on the internet can read, and admin pages that are not protected. We found 41 issues - 12 serious problems, 20 important problems, 7 medium problems and 2 small problems. That gives a readiness score of 0 out of 100, which means the app currently has serious security holes and should not be exposed to users or real data until they are fixed. The most urgent one is: A powerful secret key is sitting in your project files. Anyone who gets hold of it can act as you, followed by stripe live secret key found in .env. All of these can be fixed without rebuilding the app. We only looked at the files you gave us - we did not test your live site, so the online parts of your app still need a separate check.

The evidence keeps a short prefix so you can tell which key it is; the rest never appears:

SUPABASE_SERVICE_ROLE_KEY=eyJhbGciOiJI************************[REDACTED len=203] role=service_role

That is the report's design: the reader is usually a founder who does not read code, so every finding carries a plain-English line, the file and line, the fix, and a prompt to paste back into your assistant.

The safety design

The live checks touch the network, so they are fenced off.

• --url alone does nothing. The live checks need both --url <url> and --i-have-authorization. Without the flag they are reported as SKIPPED with the reason, never as passed.
• That is not a promise in the README, it is a test. The suite runs a local HTTP server that counts requests: without --i-have-authorization the tool sends NO request at all and with --i-have-authorization the tool sends exactly ONE read request, asserting the method is GET and the path is /. 69 tests, all passing.
• It never enumerates tables. The single Supabase probe uses one table name that already appears in the project's own SQL files and sends exactly one request: GET {project-url}/rest/v1/{table}?select=*&limit=1 with the public anon key. No other table is tried.
• It never attempts to authenticate. No password guessing, no token replay, no directory listing. The only other request is a single GET to the root of the URL you supplied, to read the response headers.
• It reads at most 64 KB of the response body and gives up after 12 seconds.
• Secrets are never printed. Every secret is replaced with a short identifying prefix and [REDACTED len=...]. The report gets emailed and pasted into chat windows; a tool that leaks the key it found makes the problem worse.

Everything else is offline and static.

Two engineering decisions that are not obvious

1. The demo fixture's secrets are generated at runtime, not committed. GitHub push protection refuses any push containing a string shaped like a live credential - even a fabricated one - and it scans git history as well as the tip. A demo fixture full of realistic keys would make the repository impossible to publish, defeating the purpose of shipping the tool. So fixtures/build-fixture.mjs writes the vulnerable files at runtime, assembling each credential from parts so no contiguous secret pattern appears in the published source tree, and it creates the fixture's own git repository so the .env is genuinely committed rather than merely present. The demo still exercises every detection path it is supposed to.

2. Scanning a subdirectory of a repo still finds committed secrets. A client sends you one sub-project of a monorepo, and the file that leaks their service_role key is committed in the parent repository. A naive implementation checks for a .git directory in the target folder, finds none, and reports "no git history" - losing the finding that matters most. The fix was isGitRepo, which falls back to git rev-parse --show-toplevel and walks up to the repository root. git ls-files and git log still run from the scanned directory, so the paths they return stay relative to what you asked for and no unrelated project is pulled in.

Install and run

There is nothing to install. Node 20+ is the only requirement.

# 1. See what it finds, against a bundled, deliberately broken demo app.
node cli.js --demo

# 2. Scan a real project.
node cli.js --target ./some-project

# 3. Write the report to a file you can send to someone.
node cli.js --target ./some-project --out report.md

# 4. Add the live checks. Only if you own the site or have written permission.
node cli.js --target ./app --url https://app.example.com --i-have-authorization

Also useful: --json for machine-readable output, --no-history to skip the git history scan, --fail-on-critical for CI, --quiet for just the headline.

What this does not do

It is not a penetration test. It does not break in, exploit anything, attack your live app, or take a copy of your data, and it certifies nothing. It is not a compliance audit, and it is not a guarantee that your app is secure. It is a focused code and configuration review of the files you gave it - the subset of a real pentest that can be automated. If you need a signed penetration test report for a compliance file, that needs a certified tester, and this is not it. It does not cover your hosting, DNS, firewalls or third-party services beyond the keys it found.

If you would rather have a person read the results than run the tool yourself, that is the same review sold at a fixed price: $199 for the report, $599 with the fixes implemented, $899 for a pre-launch ship check that adds deployment hardening and a written checklist.

Try it: https://github.com/rafikulislammollaq-dev/vibeaudit

MIT licensed. Run node --test test/ and you should see 69 passing tests.

Try it

Command line (Node 20+, zero dependencies):

git clone https://github.com/rafikulislammollaq-dev/vibeaudit
cd vibeaudit
node cli.js --demo

In your browser, nothing uploaded: rafikulislammollaq-dev.github.io/vibeaudit/scanner

Source: github.com/rafikulislammollaq-dev/vibeaudit - MIT licensed.

If you would rather have a person read the findings and explain them, that is the $199 review: rafikulislammollaq-dev.github.io/vibeaudit