The Loop Was Full of Humans. The Ship Was Almost Boarded Anyway.

Military aircraft were already airborne. Armed personnel were staged to board a Chinese vessel in the Middle East. Then somebody read the intelligence report a second time and found that a chatbot wrote it.

CNN broke that story on September 18, and TechCrunch and Ars Technica both picked it up within hours. Every version of the story reaches for the same word: hallucination. That word is doing a lot of hiding. The model got the cargo wrong, yes. But a wrong model output is a Tuesday. What turned a wrong answer into planes in the air was the step nobody is writing about, and it is a step I see in almost every agent pipeline that reaches production.

CNN's exclusive is the only first hand account. Every other outlet that day, including the two I quote below, is reporting on this report.

What actually happened with the Chinese ship?

An intelligence report circulated across the US military this spring, during the war with Iran, claiming a Chinese ship in the Middle East was carrying components of a nuclear weapons program. The military moved to intercept. Four sources described the episode to CNN. Two of them said armed personnel were preparing to board. One of those two and a further source said military planes were in the air. Officials dug into the report just before the operation and found it had been generated with the help of AI.

The specifics matter more than the headline. A Special Operations Command analyst queried a chatbot about intelligence reporting on the ship's manifest that originated with US Special Operations Command Pacific in Hawaii. The bot fused open source intelligence with secret signals intelligence held in government systems and reached a conclusion about the cargo. It was wrong. CNN was not able to learn what the cargo actually was, and it is not clear from the reporting whether the chatbot was a commercial product or a government one. One source called the report "entirely false" and said it also "almost started a war."

Special Operations Command Pacific and the Pentagon did not respond to CNN's request for comment.

Why was the second prompt the real failure?

Because the second prompt is where the output stopped looking like a model output. CNN's sentence is the one to read twice. The analyst, it reports, "used AI again to package the findings into a standard intelligence report," a format CNN describes as "the kind that is trusted by military officials." Two calls, not one. The first produced a claim. The second produced a document.

That second call did something the first could not. It took a probabilistic assertion with no provenance and poured it into a container whose entire function is to signal provenance. The intelligence report format is the trust claim. Its structure tells every downstream reader that a named analyst assessed named collection against a known confidence convention. Once the model's guess is inside that container, nothing about the artifact distinguishes an analyst concluded this from a chatbot concluded this and an analyst forwarded it.

TechCrunch described that same step as formatting findings "into an official-looking summary, which was circulated across command channels." I think that phrasing undersells it. Formatting is not cosmetic here. Formatting is the laundering operation. Everyone downstream did their job correctly, and their job was to trust the format.

TechCrunch put the second AI call in print and then framed the story around speed. The second call is the more interesting half.

I keep seeing the identical shape in commercial systems. A retrieval step returns three passages with scores and document IDs. A synthesis step turns them into a paragraph. A formatting step turns the paragraph into a PDF with a logo, a date and a signature block. Scores gone. Document IDs gone. The reader gets a document that carries more authority than the evidence underneath it, and the authority was manufactured by a template.

Does human in the loop AI actually catch fabricated output?

Not on its own, and this incident is the proof. A human ran the first prompt. A human ran the second. A human released the report. Humans read it up the chain and acted on it. The loop was full of humans at every step and it still put aircraft in the air. Human in the loop AI is a control over authority, not a control over truth, and the two get conflated constantly.

A reviewer can only catch what the artifact shows them. If the artifact has been stripped of its own uncertainty, the reviewer is reading a confident document and their honest response to a confident document is to believe it. One of CNN's sources put the problem in one sentence: "AI in targeting is definitely something that is ramping up and there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide."

CNN reports this is not an isolated incident. Hallucinations of this kind have shown up repeatedly across the intelligence community since these tools started proliferating, according to one of its sources. And the pressure runs one direction. AI pushes analysts to produce and disseminate faster, which is exactly the condition under which a reviewer skims. Another source gave the whole thing its epitaph: "AI allows you to get to a bad idea faster."

If you want the longer version of why a review step fails when it cannot see the evidence, I wrote about the same dynamic in OpenAI's Hugging Face incident report and in Anthropic's research on multi agent failure modes. The cost side of the argument sits in OpenAI's 20% monitoring overhead, which is roughly what real oversight prices at.

Where do the three reports disagree about the cause?

They agree on the facts and split on the diagnosis, which is the most useful thing about reading all three. CNN blames structure. TechCrunch blames speed. Ars Technica blames the technology itself. Each diagnosis implies a different fix, and only one of them is something you can build.

CNN's reporters wrote the sentence that should have been the headline: "There's no one set of standards for how the US verifies the information generated by these tools." That is a plumbing problem, not a philosophy problem. Ars, by contrast, files the episode in a long catalogue of hallucination stories running from judges to doctors to call centres, and links research suggesting hallucination may be impossible to eliminate entirely. That framing is accurate and also a dead end, because the acceleration is not up for debate.

It is genuinely not up for debate. In January, Defense Secretary Pete Hegseth released an "Artificial Intelligence Acceleration Strategy" whose announcing memo describes "democratizing AI experimentation and transformation across the Department by putting America's world-leading AI models directly in the hands of our three million civilian and military personnel, at all classification levels." Ars reports that as of June, a Pentagon representative told Congress 1.5 million active Defense Department personnel had already used the military's generative AI tools. Against the 3 million the memo targets, that is 50.0% penetration before anyone agreed on how to verify what the tools produce. You do not put the brakes on that with a blog post about hallucination rates.

GenAI.mil went live in December 2025 with Gemini for Government. Grok for Government was added this year. The platform question was settled long before the verification question was asked.

What did every outlet miss in the 2023 declaration?

All three reports gesture at the State Department's 2023 Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy as the norm that got ignored. None of them read its numbered measures against this incident. I did, and two of the ten describe this failure with uncomfortable precision.

Measure six: "States should ensure that military AI capabilities are developed with methodologies, data sources, design procedures, and documentation that are transparent to and auditable by their relevant defense personnel." Data sources, auditable, by the personnel using it. That is the provenance requirement, written down and endorsed, three years before an analyst pasted model output into a report format that erased exactly those things.

Measure seven names the human factor by its technical name: "States should ensure that personnel who use or approve the use of military AI capabilities are trained so they sufficiently understand the capabilities and limitations of those systems in order to make appropriate context-informed judgments on the use of those systems and to mitigate the risk of automation bias." Automation bias. Not hallucination. The declaration's authors understood in 2023 that the model erring was the ordinary case and the human deferring was the dangerous one.

One correction worth making, because it bears directly on the thesis. Ars wrote that the declaration "urged that 'accountable' use of AI systems must always involve 'a human in the loop, a responsible human chain of command and control.'" I pulled the declaration text to check the quote. The phrase "a human in the loop" does not appear anywhere in that document. What it actually says is that military use of AI "needs to be accountable, including through such use during military operations within a responsible human chain of command and control." Close in spirit, and not the same claim.

I am not scoring points on a reporter. I am pointing at the mechanism, because it happened to me in the middle of writing this. A paraphrase hardened into a quotation, the quotation went into an article with a dateline and a byline, and I would have repeated it as the declaration's own words if I had trusted the article's format instead of fetching the primary source. That is the ship story in miniature, running on a news pipeline instead of a command channel. Provenance dies at the reformatting step. It does not matter whether the reformatter is a model or a person on deadline. The same mechanism shows up in Apple's reference image work, where a credential proves one thing and gets read as proving another.

The declaration's opening paragraph. Read measures six and seven further down the page and the 2026 incident looks less like a surprise and more like a prediction.

How do you keep provenance attached when a model reformats its own output?

Four rules. I have deployed all of them, they are cheap, and none of them requires the hallucination rate to improve.

Never let the system that generates a claim also render it into the trusted artifact. Generation and presentation are separate privileges. In the ship case one analyst held both, with the same tool, minutes apart. Split them and the rendering step becomes a place where you can enforce something. Keep them together and rendering is just generation wearing better clothes.

Make provenance a field, not a convention. Every claim your system emits should carry a structured origin: which retrieval returned it, which document ID, what score, which model, which prompt version. Not in a comment. Not in a log. In the object, travelling with the claim, so that dropping it is an explicit act somebody has to write code to perform.

Make the rendering step fail closed on missing provenance. This is the one people skip. If a claim arrives at the template without an origin, the template should refuse to render rather than emit a clean paragraph. I run the same rule on budget checks in my own systems, where a spend lookup that cannot be reached refuses the call instead of assuming zero, and it is the same instinct. An empty field is not an absence of risk, it is an absence of information.

Show confidence on the surface a human reads, in the format that human trusts. A confidence score that lives in your trace viewer is decoration. The reviewer is looking at the report. If the report cannot say "this paragraph came from a model, from these two sources, at this score," then your reviewer is not reviewing, they are ratifying. I tell clients this in plainer terms: if the reviewer cannot tell which sentences the machine wrote, you do not have a review step, you have a signature.

What should you change in the agent you are shipping this quarter?

Go find every place your pipeline turns model output into something that looks authored. Report builders, PDF generators, email drafters, summary cards, Slack digests, ticket descriptions, anything that takes a generation and gives it a container with your logo on it. Those are the laundering points. Then ask one question at each: can the person who reads this tell what a model asserted versus what a system verified?

Two checks make this concrete, and I would run them this week.

The first is a deterministic gate that fires on shape and not on content. It is necessary and it is nowhere near sufficient, and I got a live reminder while assembling this post. My screenshot pipeline validates dimensions, file size and pixel variance before an image is allowed into a draft. It passed a capture cleanly: 2,880 by 1,800 pixels, dimensions within 2.0% of the target, pixel variance 69 against a floor of 50. The capture was a bot challenge page reading "Let's confirm you are human." Every number was green and the content was worthless. Only looking at the image caught it. A validator that checks the envelope will approve an empty envelope every time, and 69.0 is a perfectly respectable variance for a mostly blank page with one orange button on it.

The second is an independent reviewer that did not produce the artifact and that gets the underlying evidence, not the rendered output. Every post on this site passes through a second model with fresh context and access to the raw source files, precisely so it can check claims against sources rather than against my prose. It catches things I cannot see, including defects introduced by its own previous round of edits. That last part is not a footnote. The edit application step is where new errors enter, which is why one review round is a coin flip and two is a process.

The uncomfortable conclusion is that the military's problem is not exotic. Nothing about this story required a classified network or a targeting system. It required a model, a template and an org chart, and most companies running agents have all three. The incident is worth your attention precisely because it is boring underneath. An analyst asked twice, the second answer looked official, and the format carried it the rest of the way. It is the containment problem from Gemini's sandbox escape inverted: there the model got out of its box, here the model's output got into a box it had no business being in.

If you want to find out where your own pipeline strips its evidence, the AI readiness assessment walks through the same questions on your systems rather than the Pentagon's. And if you are earlier than that, what an AI agent actually is is the place to start.

Frequently asked questions

Did the US actually board the Chinese ship?

No. CNN reports the operation was stopped just before it happened. Two of its four sources said armed personnel were preparing to board, and one of those two plus a further source said military planes were in the air, when officials examined the underlying report and found a chatbot had produced its central claim.

Which AI model produced the false intelligence?

Not publicly known. CNN reports it was not clear whether the analyst used a commercially available chatbot or a US government product. A former senior official familiar with the systems told CNN that "the internal tools are mostly just copies of the commercial stuff wearing lipstick," which suggests the distinction may matter less than it sounds.

Is human in the loop AI enough to prevent this kind of failure?

Not by itself. Humans were present at every step of this incident and it still escalated. Human review catches errors only when the reviewed artifact preserves enough provenance for a reviewer to spot that a claim is unverified. Strip the provenance during formatting and the reviewer becomes a rubber stamp with a security clearance.

What is automation bias and why does it matter here?

Automation bias is the tendency of people to defer to a machine's output over their own judgement, especially under time pressure. The State Department's 2023 Political Declaration names it directly in measure seven and asks states to train personnel to mitigate it. It describes this incident better than the word hallucination does.

How do I add provenance tracking to an existing agent pipeline?

Start at the render step rather than the model. Add a structured origin field to every claim object, then make your templating layer refuse to render any claim missing one. That single change surfaces every place in your system where evidence is currently being dropped, usually within a day, and you fix them in priority order.

Does the Political Declaration actually require a human in the loop?

No, and this is commonly misreported. The declaration's text calls for military AI use to be accountable "within a responsible human chain of command and control." The specific phrase "a human in the loop" does not appear in the document. Its ten measures ask for auditable data sources, well defined uses, rigorous testing, and training against automation bias.

Is this the first incident of its kind?

It is the first one reported at this scale, but CNN's reporting says hallucinations like this have not been isolated across the intelligence community since these tools began proliferating. Treat this as the first one that got written up rather than the first one that happened.

Sources: The near miss was first reported by CNN Politics, Katie Bo Lillis and Zachary Cohen (September 18, 2026), which is the origin of the four sources, the "entirely false" and "almost started a war" quotes, the two AI calls, and the absence of a single verification standard. Secondary coverage and the Defense Department adoption figures come from Ars Technica, Kyle Orland (September 18, 2026) and TechCrunch, Aditya Mehta (September 18, 2026), which carries the Jake Steckler comments. Measures six and seven and the accountability language are quoted from the primary text at US Department of State, Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy (November 9, 2023). The GenAI.mil launch details come from US Department of War (December 9, 2025).