There's an assumption sitting quietly inside most engineering teams: if Google can crawl and render our site, AI crawlers can too.
It's wrong. And it's the single most expensive wrong assumption in technical SEO right now.
The gap
Googlebot renders JavaScript. It has done for years — the page goes into a render queue, headless Chromium executes the scripts, the result gets indexed. Deferred, but it works.
AI crawlers don't render anything.
A server-log study by Vercel and MERJ, published 17 December 2024, found that none of the major AI crawlers execute JavaScript. That includes OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, and Anthropic's ClaudeBot. A separate analysis of over 500 million GPTBot fetches found zero evidence of JavaScript execution.
The detail I find most telling: these crawlers do download JavaScript files. GPTBot in roughly 11.5% of requests, ClaudeBot in roughly 23.84%. They fetch the scripts and never run them.
There's no partial credit here. Either your content is in the initial HTML response, or it isn't.
The one exception is Google Gemini, which rides on Googlebot's Web Rendering Service and inherits its ability to execute JS — along with all its queue delays.
So both things are true at once
Your React site ranks on page one. Google rendered it, indexed it, ranks it.
Then a customer asks ChatGPT the exact question your page answers, and you're not in the response. Nothing is wrong with your rankings. The crawler received an empty shell:
<body>
<div id="root"></div>
<script type="module" src="/assets/index-a1b2c3.js"></script>
</body>
You are simultaneously first and invisible.
The trap inside the trap: structured data
This is where it gets specific.
You build a React site. You add react-helmet. You inject JSON-LD through it — Organization, LocalBusiness, FAQPage, the full graph:
<Helmet>
<script type="application/ld+json">
{JSON.stringify(schema)}
</script>
</Helmet>
You open DevTools, expand <head>, and there it is. Beautiful markup. Job done.
It isn't done. DevTools shows you the rendered DOM — the page after JavaScript has run. That is not what a crawler receives.
Open view-source: instead. Not the Elements panel. If your JSON-LD only exists in DevTools and not in view-source, then GPTBot, ClaudeBot and PerplexityBot never see it.
Google will still find it, because Google renders. That's why this failure is so quiet — every tool you'd normally check with says you're fine.
One nuance worth knowing: content doesn't have to be visible prose to count. Data embedded in the initial HTML as inline JSON or server-rendered payload is in the raw response and readable. What's missed is specifically what the browser builds after load.
Why this compounds
Around 92% of ChatGPT agent queries rely on Bing's index — and Bingbot's JavaScript rendering is limited. So a client-side rendered SPA risks being absent from Bing and from every AI crawler that queries it. Two failures, one root cause.
The test takes thirty seconds
curl -s https://you.com/page | grep -c "ld+json"
curl -s https://you.com/page | grep "your headline text"
If grep returns nothing, neither does the crawler.
Do it on your pricing page, your service pages, your best article. Those are the pages with the most to lose.
One layer deeper: the CDN
Everything above assumes the crawler receives the same response you do. It might not.
The curl test above uses your User-Agent and your IP. A CDN or edge middleware can serve differently based on User-Agent, headers, or geography — and AI bots come from datacenter IPs with declared bot UAs, which is exactly the traffic a WAF is tuned to challenge. Your HTML can be perfect and GPTBot still gets a 403, a challenge page, or a stale cache entry before it ever reaches the content.
So test as the bot, not as yourself:
curl -A "GPTBot" -s https://you.com/page | grep -c "ld+json"
And if you're on Cloudflare, the honest check is your server logs: filter by the AI crawlers' User-Agents and read the response codes. 200 means they see you. 403, a redirect, or a challenge means they don't — and no view-source will reveal it, because you're looking from a browser with a human fingerprint.
robots.txt doesn't save you here either: the WAF fires before robots.txt is consulted. Permission and access are different layers, and both fail silently.
(This section came out of a sharp comment from Wren Calloway below — worth reading the thread.)
The fix is not new
Server-side rendering, static generation, or prerendering. Next.js, Nuxt, Angular Universal, Django SSR — the framework matters less than the principle: put the real content in the HTML the server sends.
On our own platform the JSON-LD is rendered directly into the DOM rather than injected through Helmet, and key pages are prerendered. Not because it's elegant — because it's the difference between having schema and having schema anyone can read.
This is the same advice that's been correct for ten years. What changed is the cost of ignoring it. It used to mean waiting on Google's render queue. Now it means being absent from answers entirely, in the fastest-growing discovery channel there is.
Related reading:
