Where the LLM Stops: Deterministic Scoring in an AI-Assisted VAPT Pipeline

Every VAPT report ends the same way: a handful of numbers. A CVSS score. A severity label. A priority rank. Sometimes an aggregate risk score. Those are the numbers a remediation team actually acts on: what gets patched this sprint, and what waits.

Once a large language model enters that pipeline (writing summaries, explaining findings, drafting remediation steps), a quieter architectural question follows it in: is the model narrating scores that already exist, or is it, somewhere along the way, actually shaping them?

This is a technical write-up of how ONUS, an open-source, self-hosted DAST (dynamic application security testing) platform, answers that question by construction rather than by policy. It's drawn from ONUS's internal project report, its architecture, its test suite, and scan data collected during development and validation, with the test count independently verified against the live repository (more on that below), not from GitHub stars, scan counts, or its listing in OWASP's Vulnerability Scanning Tools directory. None of those are evidence that an architecture is sound, and this article deliberately sets them aside in favor of what was actually built, tested, and observed. Links to the repository, project site, and that listing are at the end, as references, not as proof of anything.

ONUS's own validation confirms the design held in practice: every scored field traces to a single function, so re-running a scan against an unchanged target reproduces identical numbers. When the local language model is slow or unreachable, only the report's prose degrades. The severity, CVSS, and priority numbers never move.

What ONUS Is, Briefly

I originally built and validated ONUS as a supervised project for the IIT Kanpur Computer Centre, under Navpreet Singh, before open-sourcing it.

For readers meeting it for the first time: ONUS is a self-hosted DAST platform. An operator submits an authorized domain; eight scanning modules run against it in parallel; results are deduplicated, passively re-verified, scored against the official CVSS v3.1 formula (the industry-standard 0–10 vulnerability severity scale), described in plain English by a local language model, and delivered as both a downloadable PDF and a live dashboard.

Three commitments run through the whole system:

• Air-gapped by design. No scan data or finding ever leaves the local network, including the AI component, which is Ollama running Qwen 2.5 7B on local hardware, not a hosted API.
• Zero licensing cost. Every scanning tool it orchestrates (nmap, OWASP ZAP, Nikto, testssl.sh, Nuclei, FFUF, WhatWeb, WAFW00F, subfinder, Amass) is open-source.
• Non-destructive by construction. Active tests use read-only, proof-of-concept payloads only.

That architecture isn't really this article's subject, though. What is: the line ONUS draws around what its LLM is allowed to touch, and what happens on both sides of that line when things go wrong.

The Core Design Question: What Is the LLM Allowed to Decide?

ONUS's project report states this as one of its core design objectives, in plain terms: every numeric score in the final report (CVSS score, CVSS vector, severity, priority, aggregate risk score) has to come from a deterministic formula, never from the language model, so that running the same scan twice produces byte-identical numbers.

That's a strong claim. It's worth asking why it's the right one, not just noting that ONUS makes it.

Reproducibility is testable; "usually right" isn't. A pure function (cvss_scorer.py::score_finding()) can be checked against known CVSS vectors in a unit test and asserted to never drift. ONUS's test suite (690 tests, as of this writing) does exactly that: the CVSS formula is validated against known vectors as one specific, ongoing test category. There's no equivalent test you can write for "the model rates this finding as Medium," because a model's output can vary across prompts, model versions, and (for a self-hosted, comparatively small model like the 7B one ONUS runs to stay air-gapped) whether it's even reachable that day.

Availability and correctness shouldn't share a failure mode. If the LLM produced the severity number, LLM downtime would force a choice: block the report entirely, or quietly fall back to some second, less-tested scoring path: exactly where a less-tested path is most dangerous. ONUS's own discussion of results confirms this decoupling held in practice: a slow or unreachable Ollama instance degrades the report's prose quality, but never its severity, CVSS, or priority numbers. When Ollama can't be reached after retries, a rule-based fallback description is substituted and the report is explicitly flagged (ai_unavailable), never silently left blank, and never routed through a different scoring path.

The model never sees evidence before it's scored. This is structural, not just policy: aggregation, passive re-verification, and CVSS scoring all happen, and all complete, before the LLM is invoked at all. By the time Qwen 2.5 7B sees a finding, it's already deduplicated, already tiered by confidence, and already carries its final CVSS score and priority. Its only job from there is to write about it.

That last point has a second, more general benefit worth naming, even though it isn't a claim the report itself makes or set out to test: a scanner's raw evidence often includes text pulled directly from the target: page titles, error strings, response bodies. That's attacker-adjacent content, and occasionally attacker-controlled. A model that reads raw target content and assigns the severity number is a model whose score a sufficiently motivated target could, in principle, try to influence. A model that only writes prose about a number it has no ability to change has a much smaller blast radius if something in that evidence turns out to be adversarial. Scoring first and describing second closes that door structurally, whether or not it was the original motivation for the design.

System Architecture

ONUS is a six-layer pipeline. Each layer has one job and talks to its neighbors.

That Layer 5 label is a simplification: the architecture diagram it's drawn from only has room for one box. Layer 5 is actually four sequential stages, and only the last one touches Ollama at all. That's the subject of "Inside the Analysis Pipeline," below.

A scan starts when an operator submits a domain with authorization confirmed. FastAPI validates the request, rejecting private IP ranges (RFC 1918) and localhost outright, checks for duplicate or concurrent scans against the same domain, creates a Scan row, and pushes a job to Redis. Celery dispatches a group of eight parallel scanning subtasks; a chord callback (a Celery primitive that fires once every task in a group has completed) triggers once all eight report back, and only then does aggregation, verification, scoring, description, and PDF rendering begin.

Two schema-level choices are worth calling out, because they show the layering is enforced in the database, not just the diagram. The generated PDF is stored in a separate reports table (as BYTEA), deliberately kept apart from the scans row, so that polling a scan's status never has to read or write binary PDF data. And updates to a scan's per-module status map use a raw, atomic jsonb_set SQL statement rather than a read-modify-write ORM update, specifically to avoid a race condition when multiple parallel Celery workers try to update the same scan's status at once. (WeasyPrint, notably, is also what rendered ONUS's own underlying project report: same renderer, both jobs.)

Eight Modules, One Schema

None of these tools is novel: nmap, ZAP, Nikto, testssl.sh, and Nuclei already do their individual jobs well on their own. What none of them do alone, and what the project report frames as the actual contribution over just running them separately, is operate in one coordinated pipeline against a single target, deduplicate and cross-reference their overlapping findings, apply one consistent formula-derived score across all of them, and produce a single narrative a non-technical reader can act on.

Every module wraps its external tool via subprocess under a controlled timeout, and every module has to normalize its output to one shared finding schema: module, tool, type, title, evidence, target, found_by, and a confidence/verifiable flag. That schema is treated as non-negotiable throughout the codebase, for a specific reason: a module that emits a malformed finding causes silent data loss at the aggregation stage. Hold onto that: it comes back later, once as a design decision and once as a real bug.

Scanning Behind a Login

For targets that sit behind authentication, an operator can supply login credentials on submission. ONUS stores them in Redis, keyed by scan ID, never as a Celery task argument, and never written to the scans table. A scanning module retrieves the credentials, auto-detects whether the login is an HTML form or a JSON API, logs in, and crawls and tests only the authenticated surface. Logout-shaped links are explicitly excluded from the crawl: a rule that exists because of a real bug, covered below. Credentials are deleted from Redis once the scan finalizes.

Inside the Analysis Pipeline

Once all eight modules report back, ONUS runs a fixed four-stage pipeline before anything reaches a human, or an AI model acting as narrator. This pipeline (and the fact that its stages run in a fixed order, each depending on the last one having already finished) is the actual trust boundary this article is about.

flowchart TD
    A["8 module result envelopes"] --> B["Aggregator<br/>dedupe + fingerprint collapse"]
    B --> C["Confidence Verifier<br/>passive re-observation only"]
    C --> D["Deterministic CVSS Scorer<br/>CVSS v3.1 formula"]
    D --> E{"Ollama reachable?"}
    E -->|Yes| F["Ollama (Qwen 2.5 7B)<br/>description + remediation prose"]
    E -->|No, after retries| G["Rule-based fallback<br/>ai_unavailable = true"]
    F --> H["Merge: scores from D, prose from F/G"]
    G --> H
    H --> I["Scored + described findings"]

1. Aggregation

The aggregator deduplicates findings reported by more than one module into a single entry, and collapses large groups of identically-shaped responses into one summarized finding. That second part matters more than it sounds: it's the defense against a wordlist-based directory brute-force returning thousands of near-identical "findings" and drowning out everything else in the report.

2. Confidence Verification: Passive, and Strictly Separate from Scoring

This is the stage most worth slowing down for, because it's the one most easily skipped in a simpler design, and ONUS's report is explicit that it wasn't: confidence verification is a distinct, standalone stage, in its own module (backend/analysis/verifier.py), strictly between aggregation and scoring.

Its rule is absolute: every verifier re-issues the exact same non-destructive request a scanning module already sent, and checks whether the same evidence still reproduces. It is never given a new exploitation technique or payload, even where adding one would be trivial. That constraint is what keeps "verification" from quietly turning into a second, less-authorized exploitation pass. For reflected XSS specifically, that re-observation happens in an actual browser (Playwright-driven headless Chromium) rather than a raw HTTP replay, since confirming a reflected payload actually executes needs a real rendering context, not just a string match in a response body.

Every finding lands in one of three tiers:

A finding that fails to reproduce is never dropped: it's demoted to unverified, with a recorded reason. The report is explicit about why: silently dropping it would reintroduce the exact class of data-loss bug this stage exists to prevent.

The tier isn't just a label; it deterministically shifts two things: a finding's priority (a confirmed finding moves one step more urgent, an unverified one moves one step less) and its contribution to the overall risk score. The final PDF also groups its findings catalogue by tier (Confirmed, then Probable, then Unverified) rather than mixing them, specifically so a reader can immediately tell a re-proven vulnerability apart from one that still needs manual review.

3. Deterministic CVSS Scoring

Every finding (now carrying both a deduplicated identity and a confidence tier) runs through a CVSS v3.1 scorer with an explicit rule for 73 distinct finding types. This is the single function the reproducibility claim at the top of this article rests on.

4. LLM Description and Remediation: A Fallback That Never Touches a Number

Only after scoring is complete does Ollama see the findings, and only to produce descriptive prose and remediation text. If it's unreachable or times out after retries, a rule-based fallback description is substituted and the report is flagged accordingly: never left blank, and critically, never routed back through a different scoring path. A final merge step combines the scores from stage three with the prose from whichever of the two description sources ran.

A Worked Example: One Real Scan

Architecture diagrams are easy to nod along to and hard to actually picture. Here's what the pipeline above produced against testphp.vulnweb.com, an intentionally-vulnerable public test site, as documented in the report's own dashboard screenshots.

Overall: 4/100, Low Risk (0 Critical, 0 High, 3 Medium, 0 Low, 6 Informational).

Layered on top of that table, here's what the LLM contributed for the same scan:

The security scan of testphp.vulnweb.com revealed several issues related to domain security configurations and network accessibility. The site lacks essential email authentication records (SPF, DMARC, DKIM) which can lead to phishing attacks and undetected spam. Additionally, the absence of an HTTPS service on port 443 and no web application firewall suggests potential vulnerabilities in data protection and traffic filtering. Overall, these findings indicate a moderate security posture that needs improvement to protect against common cyber threats.

Two things are worth noticing. First, the three Medium findings all carry the identical CVSS score (4.3): they're the same underlying finding type (a missing email-authentication record), scored by the same deterministic rule, every time. That's the "byte-identical numbers on re-run" claim made concrete: same finding type, same score, no exceptions. Second, the paragraph above is the only part of this output the LLM touched. If Ollama had been unreachable during this run, the table wouldn't change at all: only the paragraph would, replaced by generic fallback text and flagged as such.

Operational Resilience: When Things Fail Mid-Scan

A scan's status is a small state machine, and the interesting design decisions are all about what happens off the happy path.

stateDiagram-v2
    [*] --> queued
    queued --> running
    running --> analysing: all 8 modules succeeded or partial
    running --> awaiting_user_decision: a module failed or timed out
    running --> failed: stuck-scan deadline exceeded
    awaiting_user_decision --> running: operator retries failed modules
    awaiting_user_decision --> analysing: operator continues without them
    awaiting_user_decision --> cancelled: operator cancels
    awaiting_user_decision --> failed: stuck-scan deadline exceeded
    analysing --> complete
    complete --> [*]
    cancelled --> [*]
    failed --> [*]

If any of the eight modules reports failed or timeout, the pipeline doesn't quietly proceed without it: it pauses at awaiting_user_decision and surfaces the failure to the operator, who chooses to retry the failed modules, continue without them, or cancel the scan outright. That pause is a tested, load-bearing state, not just a box on a diagram: the retry/continue/cancel endpoints are part of the 690-test unit suite.

Separately, a stuck-scan reaper independently fails any scan that exceeds a hard deadline with no progress. That guards against a specific, easy-to-miss failure mode: a Celery hard time-limit that kills a task outright, before it ever gets the chance to report back as failed. Without the reaper, that scan would just sit at running forever, with nothing in the pipeline aware anything had gone wrong.

Testing: Three Levels, Kept Deliberately Separate

ONUS's report frames testing as three distinct exercises, kept apart from the implementation work on purpose:

• Unit: 690 tests, pytest, mocked database sessions, runs in seconds. Covers the CVSS scoring formula against known vectors, the aggregator's dedupe and response-fingerprint-collapse logic, the confidence verifier's pass/fail/demote behavior, the decision-flow endpoints (retry/continue/cancel), the stuck-scan reaper, and the Scans-listing endpoint's filter/sort/search/pagination behavior.
• Integration: a live Docker Compose stack (PostgreSQL, Redis, ZAP, backend, worker), not mocked. Exercises the full Celery pipeline: group dispatch, chord callback, aggregation, scoring, the Ollama call, PDF generation. This is deliberately not mocked, specifically to catch issues that only appear across process boundaries. It's how both bugs discussed later in this article were actually found.
• End-to-end and authenticated validation. Real scans through the actual frontend, in a real browser (Playwright-driven headless Chromium), against nine deliberately-vulnerable practice targets and one additional authorized public target. Authenticated scanning was validated separately from unauthenticated scanning, since it exercises a distinct code path.

It's worth being precise about what this establishes. All three levels test that the pipeline behaves correctly: that a known CVSS vector scores the way it should, that a failed module actually pauses the scan, that a login flow actually authenticates before crawling. None of them measure how often the underlying scanners correctly identify real vulnerabilities in an application ONUS has never seen before. That's a different, harder question, and the report doesn't claim to answer it. More on that a few sections down.

What Was Actually Measured

One number in that table is not the report's own figure, and it's worth flagging plainly rather than letting it blend in: the test count. The report states 438 automated tests. While preparing this article I cloned the live repository and ran pytest --collect-only directly against it; it collected 690 tests cleanly, with no errors. That's the number used throughout this piece from here on. Every other figure in the table above is exactly what the report states, unchanged.

Two more things here deserve a beat of interpretation, clearly marked as mine rather than the report's own framing:

• "OWASP Top 10 category" is a partial tag, not a universal one. Only 5 of the 10 official OWASP Top 10:2021 categories are actively mapped by the aggregator. It's visible directly in the worked example above: the three misconfiguration findings got an OWASP tag; the six informational ones didn't. That's expected behavior, not a bug. But "OWASP category" should be read as "mapped where a rule exists," not as a claim of full Top 10 coverage.
• Scan durations aren't averaged, and the report is explicit about why not. Duration varied considerably by target and by how many modules found work to do, so instead of a single figure, the report gives three specific, real durations: a scan against dvwa.local completed in as little as 0.9 minutes and as long as 5.8 minutes across different runs; clinkl.in completed in approximately 4.6 minutes; nodegoat.local took approximately 13.5–14 minutes on its two full runs. A single "average scan time" would have been a punchier, more citable number, and a more misleading one, given that spread. Reporting the range instead is a small methodological choice worth noticing.

Two Bugs That Shaped the Architecture

Real validation surfaced two bug classes that a design review alone wouldn't have caught, and the report calls both out because they generalize past their specific fixes.

The Logout Link That Broke IDOR Detection

The IDOR (insecure direct object reference) detector built for NodeGoat's /allocations/:userId vulnerability initially found nothing when run through the full pipeline, despite working correctly when tested in isolation. The root cause: the crawler was following NodeGoat's own logout link mid-crawl, silently destroying the authenticated session for every remaining test in that run, not just the one that happened to hit logout.

The fix, excluding logout-shaped links from the crawl, improved every OWASP Top 10 test against authenticated targets, not just IDOR detection. The report's own framing of the generalizable lesson is worth keeping intact: a detector that "works in isolation" is not yet validated until it's run through the actual pipeline that will call it.

Tools That Failed Without Ever Saying So

Separately, several external-tool integrations (testssl.sh, WHOIS lookups, WhatWeb) were, at different points, confirmed non-functional in ways that produced no error and no empty-result warning. They simply never contributed a finding, silently, for reasons ranging from missing system packages to an incorrect flag.

This is arguably the scarier bug class of the two, because wrong output is at least visible; no output and no error isn't. Both bug classes motivated the same structural response: a module's execution status, whether it found nothing, failed outright, or succeeded quietly, is now always visible in the report, tied back to the non-negotiable finding schema described earlier. That's a direct, traceable line from two specific production bugs to a specific architectural invariant: nothing about a module's run is allowed to be silent.

Guardrails

A handful of structural guardrails run through every layer of the system:

What This Doesn't Prove Yet

The report is candid about ONUS's functional scope (see Limitations below). It's worth being equally candid about what its evidence does and doesn't establish, since that's easy to blur in any project write-up, this one included.

• No detection-accuracy numbers. There's no reported false-positive rate, false-negative rate, or precision/recall figure for any of the eight modules or the OWASP Top 10 test functions. The 690 unit tests confirm the pipeline's logic behaves correctly against known inputs (the CVSS formula, the confidence transitions, the dedup logic), not how often the underlying scanners correctly flag real vulnerabilities.
• A small, mostly-known validation set. All 79 recorded scans and both bugs above came from nine targets: eight deliberately-vulnerable practice applications built specifically to be found (DVWA, NodeGoat, Mutillidae, Juice Shop, WebGoat, bWAPP, Metasploitable2, and testphp.vulnweb.com) plus one additional authorized public target. That's the right way to safety-test a scanner without touching anything unauthorized. But it's a different exercise from measuring performance against diverse, unfamiliar, real-world applications.
• The verifier's own reliability against adversarial evasion is untested. Passive re-observation of the same non-destructive request is a sound way to avoid false confidence, but the report doesn't describe adversarial testing of the verifier itself: for instance, a target that behaves inconsistently on purpose.
• No evaluation of remediation quality. The LLM stage and its rule-based fallback both produce description and remediation prose, but the report includes no assessment (automated or human) of how accurate, complete, or useful either one actually is.
• The tradeoffs of running a small local model aren't quantified. A 7B model was the deliberate, correct choice for staying air-gapped. But the report doesn't report how often the fallback actually triggered in practice, typical inference latency, or how the prose compares to what a larger model would produce.

None of this is a criticism unique to ONUS: most VAPT write-ups, open-source or commercial, don't publish precision/recall data either. But an article whose whole point is separating what's measured from what's designed should hold itself to the same standard.

Key Takeaways

• Keep every number a remediation decision depends on (CVSS score, vector, severity, priority, aggregate risk) behind a deterministic, unit-testable function. Never behind a model call.
• Give the LLM a narrow, late job: writing prose about findings that have already been fully scored, with a flagged, deterministic fallback for when it's unreachable.
• Passive, non-destructive re-verification (re-issuing the same request, never a new payload) adds confidence tiers without turning a scanner into a second exploitation pass.
• Never silently drop a finding that fails re-verification. Demote it, with a reason, so a false negative can't masquerade as a clean scan.
• Cross-process integration testing against a live stack, not mocks, found bugs unit tests structurally could not have: a logout link killing an authenticated session mid-crawl, tools failing without ever raising an error.
• A component that "works in isolation" hasn't been validated until it's run through the real pipeline that will call it.

Limitations

Directly from the project report (open, acknowledged gaps in current functionality, not evidence against the deterministic/LLM boundary itself):

• Adjacent-ID IDOR detection has a known ceiling. The IDOR detector probes nearby numeric or ObjectId-shaped identifiers; it will not find an IDOR protected only by a non-sequential, random identifier scheme.
• Authenticated scanning covers HTML-form and JSON-API logins only, both auto-detected. Multi-step or OAuth-style login flows are out of scope for the current implementation.
• Ollama availability is a soft dependency, by design. Every deterministic field still gets computed if it's unreachable, and the fallback is clearly flagged rather than silently degraded. But description and remediation quality genuinely varies depending on whether the local model was reachable during a given run.
• The Scans dashboard's "select all" is page-scoped, with no bulk action wired up yet: it's a hook point for a future bulk retry/cancel action, not a shipped feature.
• Sort-by-status on the Scans dashboard is alphabetical on the status string, not ranked by operational urgency.
• OWASP Top 10 category tagging currently covers 5 of the 10 official 2021 categories (see "What Was Actually Measured" above).

Future Research Directions

From the report's own future-scope section:

• A bulk action (retry/cancel) on the Scans dashboard's existing selection-bar hook point.
• Feeding the IDOR detector a second real authenticated user's identifier, rather than only guessing nearby values, to remove its adjacent-ID ceiling.
• Support for additional login flow types in authenticated scanning.
• An operator-facing settings page for subfinder's optional provider API keys (currently a bind-mounted config file), so deeper subdomain enumeration can be configured without editing files on the host.

Beyond what the report proposes, the evaluation gaps above point to a few natural additions to that agenda:

• Publishing precision/recall or false-positive-rate data against a labeled, more diverse target set, not just deliberately-vulnerable practice applications.
• Adversarial testing of the confidence verifier itself: targets designed to behave inconsistently on purpose.
• A structured comparison of LLM-generated versus fallback remediation text, evaluated by someone other than the pipeline that produced it.

Resources

• Source code: github.com/maverickaayush/ONUS
• Project site: tryonus.tech
• OWASP Vulnerability Scanning Tools directory: owasp.org/www-community/Vulnerability_Scanning_Tools