A Private Laboratory for Forensic AI Detection and Academic Rigor

A Private Laboratory for Forensic AI Detection and Academic Rigor

This post serves as my masters degree midterm at the University of Pamulang. The subject is Advanced Computer Vision.
Author: Abdul Saboor Hamedi
Repo

Taking Back Our Data: Why We Built a Private AI Detector (And How It Works)

If you have spent any time in the modern academic landscape, you already know about the climate of fear surrounding AI writing tools. While large language models offer incredible assistance, they have also created a massive problem: students and researchers live in constant fear of being wrongly accused of AI plagiarism.

Have you ever written a highly structured, perfectly grammatical paper, only to have a cloud-based detector slap a 90% AI-generated label on it? You aren't alone. This is known as "Standardization Bias." Highly disciplined writers and non-native English speakers frequently get flagged simply because their writing is "too perfect." Most mainstream detectors rely on a flawed, single-metric approach that struggles to tell the difference between Standardized Academic English and machine-generated text.

Even worse is the data privacy crisis. When you upload your thesis or a novel research paper to a cloud-based AI detector, you are often participating in a "pay with your data" model. You are essentially handing your intellectual property over to a third-party corporation to train future models.

To solve this, a new system was developed called the Neural Lab. It is designed as a localized, private laboratory for forensic AI detection, giving the power of auditing back to the user before their work ever reaches a teacher's desk. Let’s dive into the architecture, the detection methodology, and some of the cool engineering challenges solved along the way.

The Architecture Stack

The Neural Lab was built for industrial production, running entirely on the user's local hardware to guarantee full data sovereignty.

The backend relies on a high-performance FastAPI web server with a PostgreSQL database acting as the persistence layer to safely store research history. On the machine learning side, the engine integrates a PyTorch backend to run the foundational transformer model, which seamlessly auto-detects CUDA availability to load the model in half-precision (float16) on GPU machines, saving VRAM and boosting throughput.

The Four Pillars of Detection (The Secret Sauce)

Instead of relying on a single, easily confused metric, the Neural Lab utilizes a four-metric fusion approach to dramatically reduce false positives.

The baseline model powering this is GPT-2. Why an older model? Because GPT-2 represents the foundational autoregressive logic used by all modern LLMs. When you feed GPT-2 text generated by a more advanced model like GPT-4 or Claude, it recognizes the familiar probability-maximization patterns.

The first core metric is Perplexity, which basically measures how "surprised" the model is by the text. Advanced AI text registers as low perplexity because GPT-2 easily anticipates it, whereas idiosyncratic human text registers as high perplexity.

The second metric is Shannon Entropy, which measures predictability at the token level. AI is highly confident at each step, taking highly predictable token paths (low entropy). Humans, on the other hand, are uncertain, and our text could go in many different directions at any given point (high entropy).

The third metric is my personal favorite: Sentence Burstiness, which acts as the "Human Pulse." Human writers naturally vary their rhythm. We mix short, punchy sentences with long, drawn-out elaborations, creating high statistical variance. AI models are optimized for coherence and fluency, meaning they tend to spit out sentences of highly uniform complexity resulting in low burstiness.

Finally, the system calculates a Lexical Uniqueness Ratio. This catches the AI's tendency to constantly lean on common transitional vocabulary (think words like "furthermore," "moreover," and "essentially").

These four metrics are fused together using a non-linear sigmoid calibration to create a final AI probability score. The developers even added a brilliant "Human Override" pipeline that scans for structural keystroke typos—if a human error is found, it applies a massive static deduction to the AI score.

Forensic Audit Performance Metrics

To demonstrate the system's precision, the following data represents a typical industrial audit performance. The engine uses a fused scoring model to ensure high-fidelity results.

Neural Signature & Coverage

Integrity Audit & Neural Accuracy

Streamlined Ingestion: Drag & Drop PDF Support

A critical feature for researcher productivity is the native support for Drag & Drop PDF ingestion. Instead of traditional file picking, users can simply drag a PDF file directly onto the editor workspace. This triggers an automated pipeline:

1. Extraction: The system uses PyMuPDF to extract text with per-word precision.
2. Coordinates: It maintains spatial bounding boxes (bbox) for every word.
3. Visualization: The frontend renders a visual text layer over the original PDF layout, allowing for spatial forensic highlighting without altering the source document.

Overcoming Engineering Challenges

Building a high-fidelity local detector brought up some fascinating edge cases.

One major issue was the Calibration Consistency Problem. Early versions of the app would sometimes show a global score of 97% AI, but the editor would highlight almost no sentences in red. This happened because the global score and the sentence scores were being calculated by two completely independent algorithms. To fix this UI nightmare, the global score was rewritten to be a character-length-weighted average of the individual sentence probabilities. This means a 50-word analytical paragraph now correctly carries more weight than a 5-word transition sentence, ensuring the UI is exactly 100% mathematically consistent with what the user sees highlighted.

Another massive win was optimizing the Integrity Audit Engine (the plagiarism checker). Scanning a massive document by running exact string matches against a database is a fast track to an expensive $O(N)$ database loop. Scanning a 78,000-character document used to take about 60 seconds and required over 520 individual queries.

The engineering fix? They partitioned the text into fixed 55-character overlapping fingerprints and utilized PostgreSQL's pg_trgm trigram indexing. By passing the extracted fingerprints in a single batched execution using UNNEST and ILIKE, they reduced the database scan time from 60 seconds down to under 2 seconds.

Wrapping Up

The Neural Lab proves that high-fidelity, mathematically rigorous AI detection doesn't require uploading your intellectual property to a corporate cloud. By leveraging local hardware, smart database indexing, and a multi-metric forensic approach, developers can build tools that actually protect students and researchers from false accusations while keeping their data completely safe and sovereign.