I built a tool (mri) that runs untrusted code snippets inside a sandbox, checked against an execution allowlist derived from a dependency graph. Before I trusted it for anything, I wrote a demo: three attacks, all blocked. Then I stopped testing.
That was a mistake, and I only found out because I went back and wrote a second benchmark whose entire purpose was to disprove the first one — 19 cases of ordinary code that should just run, and 14 attack cases that weren't in my original three.
The legitimate-code suite failed first, and badly. 5 of 19 normal patterns got blocked, including a plain function declaration with parameters and a try/catch block. The static scanner registered local bindings for arrow-function params but never for function_declaration params or catch-clause bindings — so those names looked like unresolved external references to the scanner, and unresolved means blocked, by design. A tool that blocks ordinary code isn't safer than one that doesn't. It's the kind of tool that gets a comment added to disable it.
The attack suite found something worse than a missed attack. One case: a getter that doesn't touch process.env until the return value gets read on the way out of the sandbox. That access triggered the runtime's real violation check — the detection fired correctly. Then the code responsible for serializing that return value back to the host caught the resulting error, converted it to a string, and handed back a clean "executed" result. Nothing in the breach log. The check worked and the report lied about it. I'd rather have a tool that misses something and says so than one that catches something and hides it.
Also found: the sandbox wasn't a real boundary in the first place. I was using node:vm, and console.log.constructor.constructor (or the same trick through any host-provided reference) reaches the host process's actual Function constructor. This is documented — Node's own docs say vm doesn't provide security isolation — but "documented" and "tested against my own code" are different things, and I hadn't done the second one.
Fixed the sandbox by moving to isolated-vm, a real separate V8 isolate rather than a shared-realm context. Fixed the marshaling bug by making sure any violation, wherever it's triggered from, produces a blocked verdict before a return value ever gets serialized. Re-ran the full 33-case benchmark after both fixes: zero regressions, both cases now correctly blocked.
Still true after the fixes: no OS-level sandboxing underneath the isolate, no taint tracking between two resources that are each individually granted, and it only evaluates one snippet at a time — no whole-program analysis.
Repo, including the benchmark harness and raw per-case results: https://github.com/zaydmulani09/mri
