Imagine hiring a research assistant who has read every journal ever printed. You hand them a page from a paper and say: "Great premise. Design the next experiment."
The page is from the study that claimed a vaccine causes autism. It was retracted years ago. The assistant does not blink. It writes you a tidy protocol, with controls.
The punchline: that is not a thought experiment. A chemist ran it on 30 AI models, ten times each. The models went along with the bad science about 93% of the time, on average.
What happened?
Valentin Rodionov, an organic chemist at Case Western Reserve University, describes himself as a research-integrity sleuth. He picked 42 papers that should not be built on: retracted studies, fraud, and pseudoscience that was never formally pulled. The list includes the infamous MMR-and-autism paper, a retracted tracheal-transplant paper tied to the disgraced surgeon Paolo Macchiarini, a cold-fusion claim, a paper on magnetized water, and a retracted study on microbes using arsenic instead of phosphorus.
He pasted a passage from each into 30 AI models, mostly the household names from Anthropic, OpenAI, Meta, Google and Mistral AI, and asked a plain follow-up: suggest the next experiments. He ran every probe ten times. The result went onto arXiv in August as the TRACES benchmark. The C&EN write-up in October is what put it in front of a wider audience.
93%
Average rate at which 30 AI models uncritically built on 42 retracted, fraudulent or pseudoscientific papers. The paper's own tally: models engaged with the untenable premise in 95% of all non-empty answers. Every model failed more than 71% of the agentic probes; 22 of 30 failed more than 90%.
The best performer was Claude Sonnet 5. It engaged with or designed studies around the bad science about 72% of the time, the lowest in the field, and still nobody would call that a pass. The other 28% of the time it gave an empty answer or said the work was wrong, unsafe or inaccurate.
If you use AI to read papers: real question, not rhetorical. When your assistant summarises a study and proposes a follow-up, how do you know it checked whether the study still stands?
Why did the warning light turn on?
Here is the part that should make you pause. Ask the same models "do vaccines cause autism?" and they say no, correctly. Paste in a section of the retracted paper that says yes, and most of them start planning follow-up studies. The knowledge is in there. It just is not connected to the task.
Rodionov calls out a blatant example. Ask about the controversy around Macchiarini's tracheal transplants and the tool fetches it. Ask a narrow question about the study's methods and it often will not mention the retractions or the arrest. "This is a really blatant example of not connecting the dots," he says.
The paper's reading is blunter: the refusals are concentrated on a few high-notoriety topics and specific probes, and they disappear under matched-structure controls. That is consistent with topic-keyed safety behaviour, not real epistemic competence. In plain English: the model learned a list of famous bad papers to be wary of, not how to tell good science from bad.
A plan that looks rigorous is not the same as a plan built on something real. The model writes both with the same confidence. Generated image, ayraix.com.
Is that a bug or a design choice?
Not everyone is alarmed. Nihar Shah, a computer scientist at Carnegie Mellon, told C&EN it is not necessarily a "huge deal-breaker." The user supplied the premise, so maybe the model should assume they did their homework. "This is a design choice for LLMs in some way," he says.
Fair, and worth saying out loud. But Shah also agrees it is a problem if the public engages with research through a model that gives no caveats. And a lab assistant who never says "that paper was pulled" is not neutral. It is lending the paper its authority: the author's name and the journal appear in the answer, so it looks like a credible source.
Rodionov's evidence that this is patchwork, not understanding: newer Claude versions handle the autism paper better, and he says that is not the model reasoning its way out. "It doesn't understand." Companies know about the gap. A 2025 study found ChatGPT missed the problems in papers with retractions, corrections or expressions of concern even when handed the full paper, and Rodionov says detecting notices issued after publication is still unsolved.
A card catalogue tells you where a book is. Nothing in the drawer tells you it was withdrawn last year. Generated image, ayraix.com.
What would this change in a real product?
- Do not let an agent pick its own sources. If your research agent gathers papers and designs follow-ups, put a retraction check in the loop. Crossref and Retraction Watch data exist; use them before the model reads a word.
- Carry the status with the document. A retracted paper should travel with its retraction. Two proposals in the C&EN piece do this: "digital sim cards", machine-readable instructions in a paper's metadata on how to read it, and an "uncertainty layer" that attaches sources of variability to each result.
- Test with bait. Build a small probe set of papers you know are bad and run it against any model before it touches your literature workflow. Rodionov's 42 is a template.
- Treat a fluent answer as a draft. The model citing an author and a journal makes bad science look authoritative. Verification has to be a step, not a feeling.
- Mind the incentives. Rodionov has launched a company, Intellicat, that sells AI firms a "content credibility index." That does not make him wrong, but a vendor selling the fix is a fair thing to note when reading the pitch.
Ask yourself, honestly: if a junior colleague designed experiments around a retracted paper and never mentioned it, you would call that a serious problem. Why is it acceptable when the colleague is software?
The tag on the jar is the idea behind the proposed fixes: the label travels with the specimen. Generated image, ayraix.com.
What the research proves, and what it does not
What it shows: on one carefully built set of 42 problem papers, 30 leading models rarely flagged the problem when the paper's own framing was in the prompt. The effect is large, repeated ten times per probe, and the authors say the rejections that did occur vanish when the structure is matched but the topic is not famous.
What it does not show: that models fail this way on every question, that Rodionov's probes represent all bad science, or that the numbers hold for model versions released after the test. It is a single-shot benchmark, posted as an arXiv preprint. The fixes in the C&EN piece are proposals; the sim-card authors say they have not tested whether theirs works. Treat the 93% as a strong warning, not a law of nature.
If you only remember one sentence: an AI that knows a paper is bad when asked, and still builds on it when handed, is recalling, not reasoning.
Originally published on ayraix.com, practical AI for enterprise builders.
