AI Writing Markers: Building an Open Dataset and Checker on Hugging Face

Commercial AI detectors promise a percentage score and a verdict. Turnitin, GPTZero, and dozens of startups sell certainty to educators and employers. The problem is that certainty is fiction. Independent studies report false-positive rates from roughly 5% to over 60% on genuine human writing. Non-native English speakers and formal academic prose get flagged most often — Liang et al. (Patterns, 2023) showed GPT detectors are systematically biased against non-native writers.

I built ai-writing-markers as the opposite of that model: a source-backed catalogue of concrete textual patterns associated with LLM output, plus a zero-dependency checker anyone can run locally. It is a transparency and self-editing aid, not an AI detector. Every marker cites a public source. Every report says so upfront.

The honest thing a tool can do is point at the patterns that make writing read as machine-generated — so a human can decide whether to change them. That is a different product category from "AI probability: 87%."

Why opaque detectors fail

Most AI detectors are black boxes. They output a score without explaining which features drove it, which training data shaped the classifier, or what the error rate is on your specific population. That opacity matters because the stakes are high: academic misconduct accusations, hiring decisions, content moderation.

The underlying metrics — perplexity (how predictable word choice is) and burstiness (how much sentence length varies) — are real stylometric signals. Classical detectors lean on them. But they are weak signals, not proof. A formal essay from a non-native speaker can have low burstiness and elevated perplexity scores for entirely human reasons. A short passage dense with overused LLM vocabulary might be a human imitating chatbot prose.

Wikipedia's WikiProject AI Cleanup produced the best public reference on this problem: the Signs of AI writing field guide. Volunteers documenting real cleanup work catalogued the vocabulary, transitions, structural tells, and formatting habits that make LLM drafts obvious to a trained eye. That guide became the primary source for this dataset.

The honest alternative: show the patterns

Instead of collapsing stylometric complexity into a single score, ai-writing-markers surfaces individual hits grouped by category. A report might show that your draft contains three overused vocabulary terms, two cliche phrases, and a burstiness reading below the human-typical range. You see exactly what triggered the flag. You decide whether each hit is worth fixing.

Markers are weak signals. One instance of "delve" means almost nothing. A cluster of overused vocabulary, formulaic transitions, stock shells, and low burstiness in a short passage is a stronger hint — still not proof, but a useful prompt for a closer human read. The tool's job is to make that cluster visible, not to render a verdict.

Dataset structure: six categories, 87+ markers

The source of truth is markers.json. It defines six categories, each with a detector signal type, remediation guidance, and an array of individual markers. Every marker links back to one or more source IDs defined in a top-level sources array — full citations live in SOURCES.md.

Overused vocabulary entries carry an era tag because LLM habits drift. "Delve" peaked in 2023–early 2024 and faded by 2025. "Underscore" in figurative use persists through 2026 — Grok still overuses it. A marker catalogue frozen at one moment in time would miss that evolution; the schema supports retiring and re-dating terms as models change.

The full schema wraps everything in a versioned document with metadata, licensing, and an explicit disclaimer field that downstream tools can surface programmatically:

{
  "schema_version": "1.0",
  "name": "ai-writing-markers",
  "updated": "2026-07-15",
  "license": "CC0-1.0 (data) / MIT (code)",
  "disclaimer": "Do not use these markers as sole evidence of misconduct.",
  "sources": [
    {
      "id": "wikipedia_aisigns",
      "title": "Wikipedia: Signs of AI writing",
      "url": "https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing"
    }
  ],
  "categories": [ "... six category objects ..." ]
}

Each category object carries detector_signal (what kind of feature this is), description (what the pattern looks like in the wild), and remediation (how to fix it without cargo-cult synonym swaps). That structure makes the dataset usable as training material for explainable classifiers, writing-assistant lint rules, or classroom curriculum — not just as input to a single CLI tool.

Example marker entry

{
  "term": "delve",
  "kind": "word",
  "era": "2023-mid2024",
  "note": "The canonical tell; usage in academic papers spiked ~25x post-ChatGPT.",
  "sources": ["wikipedia_aisigns", "alston"]
}

Structural markers can include regex patterns for mechanical detection — negative parallelisms like "not only X but Y", vague attribution phrases like "studies show", and leftover chatbot markup (contentReference, oaicite, turn0search0). The statistical category documents burstiness thresholds (human prose typically 0.65–0.85; AI-typical below 0.30) with explicit notes that true perplexity requires a reference language model the checker cannot load locally.

Zero-dependency checker: no pip install

check.py is stdlib-only Python 3.8+. No virtual environment, no Hugging Face token, no GPU. Clone the repo and scan:

git clone https://github.com/humzakt/ai-writing-markers.git
cd ai-writing-markers

# Scan a file
python3 check.py essay.md

# Pipe from stdin
cat essay.txt | python3 check.py -

# Machine-readable output for downstream tooling
python3 check.py essay.md --json

The checker loads markers.json, runs lexical matching per category, applies regex patterns for structural and formatting tells, and computes burstiness plus type-token ratio as perplexity proxies. It prints per-category hit counts normalized to per-1,000-words density — because density matters more than raw counts. A handful of hits in a long document is ordinary human writing; a dense cluster in a short passage is the real signal.

Example output on an intentionally AI-flavoured sentence:

$ echo "In today's fast-paced world, we must delve into the rich tapestry of innovation." | python3 check.py -
  Overused vocabulary: 2 hit(s)
      - delve: 1
      - tapestry: 1
  Cliche phrases and stock shells: 2 hit(s)
      - in today's fast-paced world: 1
      - rich tapestry: 1

The report header repeats the disclaimer on every run: this is not an AI detector. Markers are weak signals, not proof. That is not legal boilerplate — it shapes how you read the output.

How the analyzer works

The core logic in check.py separates lexical categories from pattern categories. Vocabulary, transitions, and cliche phrases use literal string matching with optional sentence-start anchoring for words like "moreover" and "furthermore." Structural and punctuation categories iterate regex patterns from the JSON. Rule-of-three detection runs a separate triple-pattern pass because it is too structural for a single static regex.

LEXICAL_CATEGORIES = {"vocabulary", "transitions", "phrases"}

def analyze(text, markers):
    n_words = max(word_count(text), 1)
    sentences = split_sentences(text)
    lengths = [word_count(s) for s in sentences] or [0]

    for cat in markers["categories"]:
        if cat["id"] in LEXICAL_CATEGORIES:
            # literal word/phrase matching
            ...
        elif cat["id"] in ("structural", "punctuation_format"):
            # regex-based pattern matching
            ...

    burstiness = pstdev(lengths) / mean(lengths)
    return results

After category hits, the checker computes burstiness (standard deviation of sentence length divided by mean) and type-token ratio as local proxies. True perplexity needs a reference language model — the checker reports the proxies and says so explicitly rather than faking a score.

Known limitations

• Markdown noise — tables and **bold** labels legitimately trip formatting checks; strip formatting for the cleanest read
• Naive sentence splitting — a table row becomes one long "sentence," skewing burstiness
• Literal matching only — the checker cannot judge tone, hedging quality, or factual grounding
• English LLM output, mid-2026 — the dataset will age as models shift their default voice

Publishing on Hugging Face: dataset + Space

The project ships in two Hugging Face surfaces:

• Dataset — the marker catalogue as a reusable, CC0-licensed dataset with metadata for discovery under tags like ai-detection, stylometry, and writing-style
• Space — a browser-only checker where you paste text and scan; nothing leaves your machine

The repo contains two Space implementations. space/ is a Gradio app — full Python server, but Hugging Face requires a PRO plan to host Gradio Spaces on the free tier. space-static/ is the deployed version: client-side JavaScript that loads the marker data and runs matching in the browser. Same catalogue, no server round-trip, free to host. Publishing steps are documented in HUGGINGFACE.md.

The dataset card on Hugging Face includes YAML front matter — license, language, tags, task categories — so it surfaces in searches for stylometry, academic integrity tooling, and lexicon datasets. Tags like ai-detection, llm, burstiness, and wikipedia connect the catalogue to the communities most likely to need cited, explainable markers rather than opaque scores.

Why markers.jsonl exists alongside markers.json

Hugging Face's dataset viewer expects tabular data — one row per record, flat columns. Nested JSON with category arrays and marker objects does not render cleanly in the viewer. So markers.jsonl is a generated, flattened view: one line per marker with denormalized category labels, source IDs as comma-separated strings, and empty-string placeholders for fields that do not apply.

{"category": "vocabulary", "category_label": "Overused vocabulary", "term": "delve", "kind": "word", "era": "2023-mid2024", "position": "", "regex": "", "note": "The canonical tell; usage in academic papers spiked ~25x post-ChatGPT.", "sources": "wikipedia_aisigns, alston"}

markers.json remains the source of truth — richer structure, nested sources array, category-level remediation text, statistical thresholds. markers.jsonl is a build artifact optimized for HF dataset viewer compatibility and programmatic row-wise consumption. Edit the JSON; regenerate the JSONL. The checker reads only markers.json.

Ethical framing: never sole evidence of misconduct

This project exists in the gap between "ignore AI writing entirely" and "trust a black-box percentage." Both extremes fail. Ignoring the patterns leaves editors and reviewers without vocabulary for what they are already noticing. Trusting opaque scores produces false accusations against the writers least able to defend themselves.

The dataset's top-level disclaimer is explicit:

Do not use these markers, or any tool built on them, as sole evidence of misconduct. Treat every result as a prompt for a closer human read.

Legitimate uses: auditing your own drafts before submission, teaching what "AI voice" looks like in a writing workshop, building explainable tooling that shows its work, feeding a richer classifier with cited features instead of mystery weights. Illegitimate uses: automated rejection, academic integrity verdicts based on marker density alone, hiring filters that penalize formal non-native prose.

Swapping flagged words for synonyms does not work either — synonyms are also high-probability tokens, and synonym swaps do nothing for burstiness. What actually helps:

1. Vary sentence length deliberately — drop a 3–8 word sentence between long ones; allow fragments
2. Replace generic claims with specifics — names, numbers, examples tied to your actual subject
3. Cut stock shells — delete "it is important to note that" and state the point directly
4. Take a position — replace "there are pros and cons" hedging with an actual stance
5. Break parallel structures — the Rule of Three reads as filler; two items or four breaks the rhythm

The per-category remediation fields in markers.json and the human-readable explainers in data/ encode that guidance with citations. The checker points; the explainers teach.

Licensing and attribution

Code ships under MIT. Data (markers.json, data/) is CC0 — public domain dedication. Underlying observations come from public sources credited in SOURCES.md; Wikipedia content is CC BY-SA. That split lets researchers fork the dataset without license friction while keeping the checker embeddable in commercial tooling.

What I would build next

Markers evolve with models. The contributing guide accepts PRs that add, retire, or re-date markers — each change must cite a source. I would like a small CI step that validates JSON schema, regenerates JSONL, and checks that every marker's source ID resolves in the sources array. I would also expose era-filtering in the checker so you can scan against "2024-era ChatGPT" vocabulary separately from current-model tells.

The broader lesson applies beyond writing: when a system's output affects people, transparency beats confidence scores. Show the features. Cite the sources. Let humans decide.