I built a vulnerability scanner that refuses to lie to me

Static analysis tools have a trust problem. They read your source, pattern-match something that looks injectable, and hand you a warning — plus, increasingly, an AI-generated "fix." You have no idea if the warning is real or if the fix works. You just get more things to check.

I wanted the opposite: a tool that doesn't guess, doesn't trust the AI's word, and doesn't call anything fixed until it's proven fixed. So I built secfix.

The idea: proof by execution

Given a finding from Semgrep, secfix doesn't re-read the code. It runs it. It builds a test harness that calls the flagged function with a unique tainted marker, executes it inside a locked-down Docker sandbox, and records exactly where that marker ends up. If the marker lands unescaped inside a SQL string, or in a shell command, or in a file path that escapes its directory — that's a confirmed vulnerability, demonstrated by execution, not inferred from syntax.

Then it patches the function, runs the same harness again on a fresh trace, and only calls the fix validated if the marker now lands somewhere safe. Crucially: it never trusts the model's own description of what it changed. The AI's account of the fix is irrelevant — only a fresh execution trace counts. If re-verification doesn't come back clean, the patch is reported as unvalidated, full stop.

The whole thing is built to fail closed. When it can't prove something, it says uncertain — it never rounds up to "safe."

Where it broke — and why that's the interesting part

The tool passed all its tests. But passing your own fixtures proves very little. So I pointed it at real vulnerable Django apps.

It hit a wall immediately. Framework code isn't like a plain function — you can't just import a Django view and call it. The moment you touch it, Django wants settings, an app registry, a database, a request object. My harness imported the module and crashed before it reached a single line of the actual vulnerability.

I could have documented that as a limitation and moved on. Instead I went after it, one wall at a time: detecting the framework and its required Python version, matching the sandbox base image to it, calling django.setup(), baking a migrated database into the container, and constructing a real HTTP request with Django's own RequestFactory.

And it worked — I got a real Django view SQL-injection all the way to a confirmed verdict, the tainted input landing unparameterized in executed SQL, proven by running it.

The honest conclusion

But reaching that verdict took something I'm not going to pretend away: I had to hand-feed the tool knowledge it couldn't derive on its own — the specific database row the view expected, and a framework-internal file-size threshold buried in Django's upload handling. Some walls generalized cheaply. Others didn't — and the ones that didn't are exactly the ones that separate a clever solo project from the multi-year integration problem that funded security companies work on with whole teams.

I wrote all of it down — which walls fell easily, which didn't, and precisely where "automatic" stops. That map is the real deliverable. Not a demo where everything works, but an honest boundary of what execution-based vulnerability reproduction can and can't do on its own today.

The code, the safety design, and the full investigation are here: https://www.github.com/balbaks/secfix — with the Django deep-dive on the v1-django-bootstrap-spike branch.