A Private Laboratory for Forensic AI Detection and Academic Rigor
This post serves as my masters degree midterm at the University of Pamulang. The subject is Advanced Computer Vision.
Author: Abdul Saboor Hamedi
Repo
Taking Back Our Data: Why We Built a Private AI Detector (And How It Works)
If you have spent any time in the modern academic landscape, you already know about the climate of fear surrounding AI writing tools. While large language models offer incredible assistance, they have also created a massive problem: students and researchers live in constant fear of being wrongly accused of AI plagiarism.
Have you ever written a highly structured, perfectly grammatical paper, only to have a cloud-based detector slap a 90% AI-generated label on it? You aren't alone. This is known as "Standardization Bias." Highly disciplined writers and non-native English speakers frequently get flagged simply because their writing is "too perfect." Most mainstream detectors rely on a flawed, single-metric approach that struggles to tell the difference between Standardized Academic English and machine-generated text.
Even worse is the data privacy crisis. When you upload your thesis or a novel research paper to a cloud-based AI detector, you are often participating in a "pay with your data" model. You are essentially handing your intellectual property over to a third-party corporation to train future models.
To solve this, a new system was developed called the Neural Lab. It is designed as a localized, private laboratory for forensic AI detection, giving the power of auditing back to the user before their work ever reaches a teacher's desk. Let’s dive into the architecture, the detection methodology, and some of the cool engineering challenges solved along the way.
The Architecture Stack
The Neural Lab was built for industrial production, running entirely on the user's local hardware to guarantee full data sovereignty.
The backend relies on a high-performance FastAPI web server with a PostgreSQL database acting as the persistence layer to safely store research history. On the machine learning side, the engine integrates a PyTorch backend to run the foundational transformer model, which seamlessly auto-detects CUDA availability to load the model in half-precision (float16) on GPU machines, saving VRAM and boosting throughput.
The Four Pillars of Detection (The Secret Sauce)
Instead of relying on a single, easily confused metric, the Neural Lab utilizes a four-metric fusion approach to dramatically reduce false positives.
The baseline model powering this is GPT-2. Why an older model? Because GPT-2 represents the foundational autoregressive logic used by all modern LLMs. When you feed GPT-2 text generated by a more advanced model like GPT-4 or Claude, it recognizes the familiar probability-maximization patterns.
The first core metric is Perplexity, which basically measures how "surprised" the model is by the text. Advanced AI text registers as low perplexity because GPT-2 easily anticipates it, whereas idiosyncratic human text registers as high perplexity.
The second metric is Shannon Entropy, which measures predictability at the token level. AI is highly confident at each step, taking highly predictable token paths (low entropy). Humans, on the other hand, are uncertain, and our text could go in many different directions at any given point (high entropy).
The third metric is my personal favorite: Sentence Burstiness, which acts as the "Human Pulse." Human writers naturally vary their rhythm. We mix short, punchy sentences with long, drawn-out elaborations, creating high statistical variance. AI models are optimized for coherence and fluency, meaning they tend to spit out sentences of highly uniform complexity resulting in low burstiness.
Finally, the system calculates a Lexical Uniqueness Ratio. This catches the AI's tendency to constantly lean on common transitional vocabulary (think words like "furthermore," "moreover," and "essentially").
These four metrics are fused together using a non-linear sigmoid calibration to create a final AI probability score. The developers even added a brilliant "Human Override" pipeline that scans for structural keystroke typos—if a human error is found, it applies a massive static deduction to the AI score.
Forensic Audit Performance Metrics
To demonstrate the system's precision, the following data represents a typical industrial audit performance. The engine uses a fused scoring model to ensure high-fidelity results.
Neural Signature & Coverage
| Metric | Value | Explanation |
|---|---|---|
| Global Neural Confidence | 35% | The character-weighted average of AI intensity across the entire document. |
| Human Authorship | 65% | The percentage of the document identified as purely human-written. |
| Exact AI Match | 0% | Territory coverage for sentences with 100% machine signature. |
| Minor AI Changes | 70% | Territory coverage for sentences likely refined or edited by AI. |
| Human Written | 30% | Territory coverage for sentences with no machine signature. |
Integrity Audit & Neural Accuracy
| Metric | Value | Explanation |
|---|---|---|
| Plagiarism Index | 25% | Percentage of this document found in previously archived records. |
| Archive Size | 176 | Total records currently stored in the private forensic archive. |
| Source Traces | 61 | Individual matches detected across the internal database. |
| F1-Score | 95.80% | The balanced accuracy of the neural engine. |
| Audit Confidence | 86.20% | The statistical reliability of the current forensic audit. |
| Precision | 95.20% | The engine's accuracy in identifying true AI-generated content. |
| Recall | 96.10% | The engine's ability to detect all instances of machine text. |
Streamlined Ingestion: Drag & Drop PDF Support
A critical feature for researcher productivity is the native support for Drag & Drop PDF ingestion. Instead of traditional file picking, users can simply drag a PDF file directly onto the editor workspace. This triggers an automated pipeline:
-
Extraction: The system uses
PyMuPDFto extract text with per-word precision. -
Coordinates: It maintains spatial bounding boxes (
bbox) for every word. - Visualization: The frontend renders a visual text layer over the original PDF layout, allowing for spatial forensic highlighting without altering the source document.
Overcoming Engineering Challenges
Building a high-fidelity local detector brought up some fascinating edge cases.
One major issue was the Calibration Consistency Problem. Early versions of the app would sometimes show a global score of 97% AI, but the editor would highlight almost no sentences in red. This happened because the global score and the sentence scores were being calculated by two completely independent algorithms. To fix this UI nightmare, the global score was rewritten to be a character-length-weighted average of the individual sentence probabilities. This means a 50-word analytical paragraph now correctly carries more weight than a 5-word transition sentence, ensuring the UI is exactly 100% mathematically consistent with what the user sees highlighted.
Another massive win was optimizing the Integrity Audit Engine (the plagiarism checker). Scanning a massive document by running exact string matches against a database is a fast track to an expensive $O(N)$ database loop. Scanning a 78,000-character document used to take about 60 seconds and required over 520 individual queries.
The engineering fix? They partitioned the text into fixed 55-character overlapping fingerprints and utilized PostgreSQL's pg_trgm trigram indexing. By passing the extracted fingerprints in a single batched execution using UNNEST and ILIKE, they reduced the database scan time from 60 seconds down to under 2 seconds.
Wrapping Up
The Neural Lab proves that high-fidelity, mathematically rigorous AI detection doesn't require uploading your intellectual property to a corporate cloud. By leveraging local hardware, smart database indexing, and a multi-metric forensic approach, developers can build tools that actually protect students and researchers from false accusations while keeping their data completely safe and sovereign.




