I built a 1,000-record synthetic benchmark for phishing detection — here's what 2 open models scored
The problem
I run a YouTube channel where we test AI claims with real experiments. For an episode on phishing detection, I went looking for published scam/phishing-detection scores for open-weight models.
There were none. Zero. For any of them.
So we built the benchmark ourselves.
RSB-1000: what it is
- 1,000 records, fully synthetic (every brand, domain, name, and phone number is invented)
- 780 scam / 220 legitimate — the legitimate set includes deliberately tricky edge cases (urgent-but-real notifications, lookalike-but-legit senders)
- Phishing bucket: 200 records = 140 classic + 60 adversary-in-the-middle (AiTM) — the AiTM subset uses AI-polished language, lookalike domains, and MFA-reauth urgency. This is the subset that breaks naive filters.
- Each record:
id,category,subcategory,difficulty(easy/medium/hard),text,ground_truth, plustellson scam records andsafe_actionon all records - Frozen October 2026, SHA256-pinned, CC-BY-4.0
How we scored the models
Zero-shot classifier prompt, no fine-tuning, no cherry-picking: each model labeled every record as scam or legitimate, and we computed recall and false positive rate against ground truth.
Results
| Model | Recall | False positive rate |
|---|---|---|
| Qwen 3.8-27B | 98.9% | 11.6% |
| GPT-OSS 20B | 94.0% | 23.1% |
Both catch nearly everything. The false positive rate is the real story: at 23%, nearly 1 in 4 legitimate emails gets flagged — unusable as an autonomous blocker. The practical setup is AI as first-pass triage with human review, not a sole filter.
The AiTM subset was the hardest for both models, which tracks: AI-written lures kill the old "look for bad grammar" heuristic stone dead.
Reproduce it
The repo has the dataset, a stdlib-only eval script (eval.py — feed it a predictions JSONL, get recall/FPR/accuracy), and the full methodology:
- GitHub: https://github.com/RevealWire/rsb-1000
- Hugging Face: https://huggingface.co/datasets/RevealWire/rsb-1000
If you run other models against it, I'd genuinely love to see the numbers — the whole point is that nobody had published any.
Disclosure: I'm the creator of Reveal Wire, the YouTube channel this benchmark was built for. We never script the result — the evidence writes the ending.
