If you build anything that touches user generated text, sooner or later someone asks: can we just detect the AI written stuff and filter it out? I spent a while putting tools like GPTZero, Originality.ai, Copyleaks and Turnitin through their paces. Here is the short version of what I found.
Detection is a probability game, not a yes or no
Every detector outputs a likelihood, not a verdict. Under the hood most of them lean on perplexity (how predictable the next token is) and burstiness (how much sentence length and structure vary). Machine generated text tends to be smooth and low perplexity. Human text tends to be lumpy. That signal is real, but it is statistical, and statistics produce false positives.
The false positive problem is worse than the marketing admits
The failure mode that actually hurts people is flagging genuine human writing as AI. It hits two groups hardest:
- Non native English (or non native any language) writers. Simpler vocabulary and more regular sentence structure read as low perplexity, which is exactly what detectors score as machine like. Stanford researchers documented this bias clearly.
- Technical and formulaic writing. Documentation, legal boilerplate and academic abstracts are repetitive by design, so they trip the same wire.
OpenAI quietly pulled its own classifier in 2023 because the accuracy was not good enough to ship. That should tell you something about how hard the problem is for everyone else.
What this means if you are building with it
A few practical takeaways from the testing:
- Never auto reject on a detector score alone. Treat it as one weak signal in a review queue, not a gate.
- Watch the threshold. Vendors tune for a low false negative rate so they look tough on AI. That trade pushes false positives up. Pick your own threshold for your own risk.
- Longer samples are more reliable. Most tools are close to coin flips under ~150 words.
- Paraphrasers beat detectors. A single pass through a humanizer often drops the AI score to near zero, so a determined user routes around you anyway.
My honest conclusion: detectors are useful as a triage hint and useless as a tribunal. If a decision matters (a grade, a payment, a ban), a human has to look.
I compiled the full hands on comparison, including how each tool handles Dutch and other non English text, in this 2026 AI detector guide. Curious whether others here have found a detector that holds up in production, or whether you have given up on the idea entirely. What is your experience?
