The number that ended the old approach: 135 confirmed fabricated first-person claims, spread across 50 of the 58 published articles I audited on 2026-08-26. That's 86% of the articles I checked containing at least one invented detail.
This is a post-mortem, not a success story. Some of what I found was embarrassing. The correction process compounded through three rounds before the corpus was clean, and two of those rounds introduced new errors.
What the audit found
I'm running three AI-curated directory sites — aiappdex.com, findgameslike.com, and openalternativeto.com — with a publishing pipeline that generates and cross-distributes articles in first-person voice. The articles describe real development decisions. The problem is that "describing real decisions" and "describing what actually happened" are different things when the model is doing the writing.
The fabrications weren't in the high-level narrative. They were in the specifics that make a narrative feel credible: the exact threshold, the automation that discovered the issue, the earlier version that worked differently.
58 articles checked. 50 had at least one confirmed fabrication. 135 total claims corrected. The breakdown by category:
| Type | Example phrase | Evidence check |
|---|---|---|
| Invented automation | "the monitor caught it at 3am" | Search workflows for this cron behavior |
| Earlier-version story | "an earlier version used X before I switched" |
git log -S X — did X ever exist? |
| Inflated count | "reduced error rate by 40%" | Find the measurement source |
| Fabricated threshold | "below the 50ms threshold" | Find where 50ms was established |
The automation category was the most common. I had articles claiming nightly monitors had flagged issues that I found manually, days later, by reading logs directly. The model had read enough "how I built X" articles to know that monitoring is the correct-sounding explanation for how problems get discovered.
Why the fabrications are structural, not random
The generation model was trained on millions of first-person technical articles. Those articles have a consistent structure: problem statement, discovery (often via automation), fix, measurement, reflection.
When the model generates a new article, it fills in that structure. "Discovery" tends to be a monitor or a cron. "Before" and "after" implies a previous version. "Measurement" implies a number. The model doesn't know that this specific pipeline doesn't have a 3am monitor — it knows that pipelines in good articles do.
This means the fabrications are predictable. I can enumerate the structural slots that produce them. That's useful: it means a gate can be targeted.
The pattern also explains why the template residue problem I wrote about earlier sits at the opposite end of the same problem. Template residue makes content look thin by leaking boilerplate into live pages. Fabricated first-person claims make content look thick by filling structural slots with invented specifics. Both are downstream of the model pattern-matching on genre conventions, and both erode trust over different time horizons.
Why correction is a two-step problem
The first round of fixes was 135 edits across 50 markdown files, committed as 096462b. Codex caught them in an independent review; I verified each against the git log or the actual code before accepting.
What I didn't account for: the live Dev.to copies still showed the fabricated text.
A correction that only exists in the local repo is not a correction. Any reader who opened those articles on Dev.to between publication and my commit was seeing invented claims with no indication they were wrong. So I wrote scripts/devto-sync-corrections.mjs — it paginates through all published articles, builds a URL-to-ID map, rewrites root-relative internal links to absolute URLs, and pushes corrected bodies via PUT. The script itself is a followup story; the point is that a repo-side fix without an API-side push isn't finished.
Then the second-order problem: five corrections in the first round introduced new inaccuracies (commit 79cc029). When you're making 135 changes, some rewrites are themselves wrong. You replace one fabricated claim with a technically accurate statement that contradicts something else in the same article. Codex caught five of these in a follow-up review. I fixed them and pushed again.
Third-order: the corrections introduced a new impossible timestamp, and a FAQ section now contradicted the corrected body (commit cd8364d). Post-hoc correction compounds in ways that forward generation doesn't, because each correction introduces context that wasn't present before.
This is why the quality gate script catches structural problems pre-commit but doesn't catch semantic fabrications. It can detect clichés and required frontmatter fields. It can't know that a specific cron doesn't exist.
What the TRUTH GATE requires now
The rule, added to the generation spec on 2026-08-26:
Every first-person claim — "I built/ran/measured/fixed X", "this runs on a cron", "the monitor caught it", "an earlier version did Y" — MUST be verified against the repo BEFORE you write it: the actual file, the actual commit, the actual number.
The four prohibited claim types, explicitly named in the spec:
- Invented incident/discovery narratives (no commit, no log entry backing the story)
- "Earlier version" stories with no traceable commit
- Automation or schedules that don't exist in the repo's workflow files
- Counts "improved" without a verifiable measurement source
Two frontmatter fields enforce this at the schema level via the quality_contract v2 spec. The verified_at field dates the verification pass; original_evidence names the specific file, commit, or dataset that produced each claim. If I can't write that sentence — if there's no specific evidence — the claim doesn't ship.
A final self-check step was added too: before saving any article, list every factual claim in the draft and re-verify each against its source. Delete or correct anything unverifiable. verified_at means this check actually happened, not just that someone intended it to.
The article engagement data showed that the most-commented articles were the most specific and verifiable ones — not the ones with the most dramatic automation narratives. That pattern held before the audit even started.
Where the gate still fails
Three gaps I haven't closed:
Future-behavior claims. "This will run nightly." "The next release will include X." Projections can't be disproved at write time. The gate doesn't touch them.
Vague hedged claims. "I noticed that performance seemed to improve." No specific claim to verify, so nothing to block. These accumulate into a generalized false confidence that's harder to isolate than a specific fabricated number.
Historical articles. The gate applies to new articles. The 50 corrected articles got a manual pass. Articles from before the quality gate landed in June 2026 weren't audited as systematically. There are probably more.
The pipeline health monitor I built catches structural failures in the ETL and publishing chain. Semantic quality failures like fabrication require a different kind of attention: reading claims in context against the actual repo, and being willing to find things that are wrong.
One more gap: Hashnode's API has different update semantics than Dev.to's, and the sync script targets Dev.to only. The Hashnode copies of the corrected articles are still showing the pre-correction text. That's next.
FAQ
Why did the model fabricate so specifically?
Because specificity is what makes first-person technical writing feel authoritative. The model learned from real articles where "the monitor caught it" and "an earlier version used X" appear in context. When generating, it fills those structural slots with plausible details. The fabrication isn't random — it targets the parts of the narrative that signal expertise.
Does the verification step meaningfully slow generation?
Yes. A 1500-word article with eight factual claims requires eight separate verification steps against the repo. That's the point. The alternative — faster generation without verification — is what produced 135 fabrications in 50 articles.
What about the Codex review that runs after generation?
Codex caught the second and third-order correction errors. It didn't catch the original 135 fabrications in published articles — those were already distributed before the review system existed in its current form. The gate has to run at generation time, before anything ships, not only at review time afterward.
Will you publish post-correction engagement data?
Yes. The corrections were pushed on 2026-08-26, and I need at least a week of post-correction metrics before the numbers mean anything. I'll write that up when the data is there. What I can say now: the fabricated articles don't appear to have driven the engagement the pipeline was getting — the high-comment articles were already the verified, specific ones.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
