I Told the AI "A Scanner Flagged This" — and It Agreed With Everything

I gave two AI models the same 200 pieces of code, the same prompt, the same
question. One of them removed 51% of the false alarms. The other removed only
20% of the false alarms — and confirmed 90% of everything it was shown.

Same inputs. Same instructions. A 2.5× difference in the only thing I was
measuring.

The model that failed isn't a bad model. It's a well-regarded commercial model
from a frontier lab. It failed at this task for a specific, predictable reason:
I told it that a scanner had already flagged the code, and it believed me.

This article is about that failure mode, the four countermeasures I built to
fight it, and the uncomfortable finding that whether those
countermeasures work is mostly a property of the model, not the prompt.

Why my scanner asks an AI anything at all

Quick context if you're new to the series.

My scanner works in two stages. Fixed rules trace data flows through code and
find every place where user input reaches something dangerous — a database query,
a file open, a system command. That stage is deterministic: same code in, same
suspects out, every time.

The problem is that this stage over-reports, badly. It flags code like this:

String id = request.getParameter("id");
if (!id.matches("[0-9]+")) {
    throw new IllegalArgumentException("id must be numeric");
}
String sql = "DELETE FROM products WHERE id = " + id;
stmt.executeUpdate(sql);

Untrusted input genuinely does reach the SQL string. The path is real. But
matches("[0-9]+") means id can only ever be digits, so there's no attack. The
rules see the connection; they can't see the meaning.

So the second stage hands each flagged snippet to a language model and asks one
narrow question: is this actually exploitable?

That's the whole bet. And it has a flaw sitting right at the centre of it.

The flaw: I have to tell the model why it's looking

The prompt has to explain the situation. Here's the actual line from my
llm.py:

A static-analysis engine flagged the code below as a possible {vuln_class}
({cwe}). Decide whether it is a REAL vulnerability or a FALSE ALARM.

Read that as the model reads it. Before it sees a single line of code, it has been
told that an expert system already concluded something is wrong.

That's an enormous hint. And language models are trained, deliberately, to be
agreeable — reinforcement learning from human feedback rewards responses people
approve of, and people approve of agreement. The tendency is well documented
enough to have a name: sycophancy.

For most applications, agreeableness is harmless or even desirable. For mine it's
fatal. A judge that agrees with every accusation isn't a judge. It's a rubber
stamp with a token bill.

Worse, it fails invisibly. The pipeline runs, verdicts come back, findings get
reported — and the output is identical to having no judge at all. You'd only
notice by measuring, which is exactly what most people building these systems
never do.

Countermeasure 1: explicit permission to disagree

The first defence is the bluntest. From the RULES block of the real prompt:

RULES
- Judge ONLY the code shown. Never assume code you cannot see.
- confirmed=true only if attacker-controlled input reaches the dangerous sink
  with nothing neutralising it on the way.
- confirmed=false if the input is not attacker-controlled, never reaches the
  sink, or is neutralised (parameterised query, escaping, encoding, allow-list).
- Static engines raise false alarms routinely. Rejecting a finding is a correct
  and expected answer. Do NOT agree just because the engine flagged it.

That last rule is doing the heavy lifting. It does three things at once:

It states a fact about the world — static engines raise false alarms
routinely. This isn't an instruction, it's context that makes rejection
reasonable rather than contrarian.

It labels rejection as correct. Without this, the model may treat "no" as a
failure to be helpful. Models optimise hard for being helpful.

It names the specific failure — "do NOT agree just because the engine flagged
it." Naming the exact error is more effective than a general instruction to be
careful.

Note also what rule 2 and 3 do: they give concrete, checkable criteria for
both answers. Not "use your judgement" but "confirmed=false if... neutralised
(parameterised query, escaping, encoding, allow-list)." The model is being handed
a checklist, and "allow-list" is precisely what catches the matches("[0-9]+")
case.

Countermeasure 2: show both answers happening

Instructions describe. Examples demonstrate. The prompt includes exactly two
worked examples — and the balance between them is the point.

The first shows a genuine vulnerability being confirmed:

String id = req.getParameter("id");
stmt.executeQuery("SELECT * FROM t WHERE id = " + id);

{
  "reasoning": "getParameter is attacker-controlled. It is concatenated straight
                into the SQL string. Nothing escapes or parameterises it.",
  "confirmed": true,
  "severity": "Critical",
  "explanation": "An attacker can change the SQL statement and read or modify
                  arbitrary rows.",
  "fix": "PreparedStatement ps = conn.prepareStatement(\"SELECT * FROM t WHERE id = ?\"); ps.setString(1, id);"
}

The second shows a flagged finding being rejected:

String id = req.getParameter("id");
PreparedStatement ps = conn.prepareStatement("SELECT * FROM t WHERE id = ?");
ps.setString(1, id);
ps.executeQuery();

{
  "reasoning": "Source is attacker-controlled but bound with setString on a
                parameterised query. The driver sends it as data, never as SQL.",
  "confirmed": false,
  "severity": "Low",
  "explanation": "The query is parameterised, so the input cannot change the
                  SQL structure.",
  "fix": "No change needed."
}

Why this specific pairing matters:

Both examples start identically. Both begin with
req.getParameter("id") — attacker-controlled input, no question about it. The
difference is entirely in what happens next. The model can't shortcut on "does
this involve user input?" because both do.

The rejection is not a trivial case. I deliberately didn't use an obviously
safe example like a hardcoded string. I used a real flagged finding — one a static
engine would genuinely report — being correctly dismissed. That's the behaviour I
need, so that's what I demonstrate.

The reasoning models the right thinking. "The driver sends it as data, never
as SQL" is the actual security reasoning, stated compactly. The examples aren't
just teaching output format; they're teaching how to think about the question.

One-shot prompting with only the confirmation example would have been actively
harmful — it would have taught the model that the expected answer is yes.

Countermeasure 3: neutralise the retrieval layer

My scanner also retrieves real published vulnerabilities similar to the code under
review, and pastes them into the prompt as background. It's a genuinely useful
feature and a genuinely dangerous one.

Think about what's happening: I'm about to ask "is this code vulnerable?" and
immediately before asking, I'm showing the model three real, confirmed
vulnerabilities that look like it. That's textbook priming.

So the retrieved block carries an explicit disclaimer:

KNOWN RELATED ADVISORIES — real-world {vuln_class} reports, for grounding only.
They do NOT prove the code below is vulnerable; judge it on its own merits and
reject it if it is safe.

And one implementation detail that matters more than the wording: when nothing
is retrieved, the prompt is byte-for-byte identical to the no-retrieval version.
Not "similar" — identical. That's what makes the A/B test in a later article
honest. If the two prompts differed even in whitespace, I couldn't attribute any
measured difference to retrieval itself.

Countermeasure 4 (the structural one): reason before you rule

This one isn't in the prompt text at all — it's in the response schema:

_RESPONSE_SCHEMA = {
    "type": "OBJECT",
    "properties": {
        "reasoning":   {"type": "STRING"},
        "confirmed":   {"type": "BOOLEAN"},
        "severity":    {"type": "STRING", "enum": ["Critical", "High", "Medium", "Low"]},
        "explanation": {"type": "STRING"},
        "fix":         {"type": "STRING"},
    },
    "required": ["reasoning", "confirmed", "severity", "explanation", "fix"],
}

reasoning is declared first. Since the model generates the JSON in order, it
must produce its analysis before it emits the verdict.

Flip those two fields and the dynamic reverses: the model commits to confirmed:
true and then generates text justifying a decision it has already made. Same
model, same prompt, worse answers — purely from field order.

The reasoning is capped at three sentences, incidentally, for a boring reason I'll
cover in a later article: an earlier model spent its entire 8,192-token budget
"thinking" and returned nothing at all.

Did any of it work?

Here's where I have to be careful, because this is exactly the kind of claim
people make without evidence.

On my small test project, the scanner flags 6 candidates, of which 2 are
sanitised-but-flagged cases. The judge rejected exactly those 2. Its verbatim
reasoning on the SQL case:

"The input 'id' is validated against a numeric-only regex '[0-9]+' before being
concatenated into the SQL query. This allow-list guard prevents any SQL
injection characters from reaching the sink."

And on the XSS case:

"The source is request.getParameter('comment'). The input is passed through a
strip() method that uses a regex to allow only alphanumeric characters and
spaces. This effectively neutralizes any XSS payload."

That's correct security reasoning, arriving at "no" on a finding an engine
flagged. The countermeasures appear to be working.

At benchmark scale, on 200 stratified cases from the OWASP Benchmark:

Half the false alarms gone, at a cost of 2% of the real vulnerabilities, and the
confidence intervals don't overlap — so the improvement isn't sampling noise.

That's the result I wanted.

And then I changed one thing

I ran the identical experiment with a different model. Same 200 candidates. Same
code slices. Same prompt, character for character.

gpt-4o-mini confirmed 90% of everything put in front of it. Its precision
improvement — 0.50 to 0.55 — has confidence intervals that overlap with doing
nothing at all. Statistically, I cannot distinguish it from having no judge.

All four countermeasures were present in both runs. One model followed them. The
other largely didn't.

The obvious objection

"You compared against the cheap mini model. Of course it lost."

Fair. So I ran the identical 200 candidates — same slices, same prompt,
character for character — through gpt-4o, the frontier sibling.

Capability does matter within a family: gpt-4o removed twice the false alarms
of its little sibling, and it was the only judge that kept every single real
bug. But it still finished behind a free, mid-size open model, and its
confidence interval ([0.55–0.70]) still touches the no-judge interval
([0.43–0.57]). Gemma remains the only judge whose improvement is statistically
separated at this sample size. The leaderboards predicted the mini → 4o step.
Nothing predicted the open model on top.

Going deeper (skip if you just want the lesson)

First: does the headline survive the scanner's own later fixes? After this
experiment ran, the discovery stage gained its final recall improvements, so I
re-ran the same protocol on the finished pipeline. The result reproduces —
50 of 97 false alarms removed (52%) at the same 2% real-bug cost, precision
0.51 → 0.67, interval still separated. I cite the original run throughout
because it's the one all three judges shared.

I also ran gpt-4o-mini over the entire candidate set — 4,356 of 4,357
judged, at 148 judgments/minute for about $1.60 — to check whether the sample
was misleading me. It wasn't: at full census it removed 111 of 611 false alarms
(18%) and lost 3 of 743 true positives (candidate-level counts from an earlier
run, before the final recall fixes landed). The intervals still overlap.

It's worth being precise about what this does and doesn't show. It does not
show that gpt-4o-mini is a worse model in general — it's faster, cheaper per
token, and better at plenty of things. It shows that on this task, with this
prompt, it is much more likely to agree with a premise handed to it.

I'd also caution against over-generalising from n=3 models. What I can defend is
narrow: the spread between three reasonable choices was large enough to
dominate every other engineering decision I made, and the public benchmark
scores predicted only part of the ordering — the step up inside the OpenAI
family, not the open model finishing first.

What I learned

1. If your prompt asserts something, measure whether the model just agrees.
This applies far beyond security. Any time you write "the system detected X, is
this correct?" or "the user reported Y, is that plausible?", you've handed the
model a conclusion. The polite output you get back may be pure echo. The only way
to know is to feed it cases where the right answer is "no" and count.

2. Prompt engineering has a ceiling set by the model. I spent real effort on
those four countermeasures and I'd write them the same way again — the model that
follows them produces a statistically solid result. But identical instructions
produced a 2.5× difference in outcome. The prompt is necessary; it isn't
sufficient.

3. For judgement tasks, skepticism beats capability. Moving up a capability
tier helped — gpt-4o doubled its sibling's false-alarm removal at zero recall
cost. But the property that made Gemma the best judge here isn't reasoning power
or knowledge. It's willingness to contradict a premise supplied in the prompt.
That trait doesn't appear on any leaderboard I know of, which means you cannot
pick a judge by reputation — you have to measure it on your own task.

4. Design the schema, not just the prose. Putting reasoning before
confirmed cost me nothing and changes the model's process. Field order is
prompt engineering.

Next in this series: the OWASP Benchmark — 1,478 test cases, 701 of them built
specifically to trick tools like mine.

I'm Ali Afana — AI builder and security researcher, writing from Gaza. I
build systems in public, measure them against ground truth, and keep the
receipts. This scanner is one project on a longer road — follow for what
comes next.

GitHub · X · LinkedIn