AI crawlers in WordPress logs: GPTBot shows up, Google-Extended never does

Google-Extended and Applebot-Extended, two of the names that robots.txt guides list most often for AI, cannot appear in a WordPress access log. Neither one sends a request. Google's crawler documentation states that Google-Extended "doesn't have a separate HTTP request user agent string." The second name gets even less: "Applebot-Extended does not crawl webpages," Apple's support page for Applebot reads, which leaves both names as robots.txt tokens rather than visitors and means that a log of AI crawlers in WordPress has to be built from the agents that send requests.

Disclosure: the author develops Prime SEO, a WordPress plugin that includes such a log. Every user agent below was checked against vendor documentation on September 21, 2026, and the code runs without any plugin.

Three vendors document nine user agents

OpenAI lists four agents on its crawler page. GPTBot gathers training material (the page says it exists "to make our generative AI foundation models more useful and safe"), while OAI-SearchBot builds the index behind ChatGPT search. ChatGPT-User fetches a page when a person asks for it. OAI-AdsBot checks pages submitted as ads.

Anthropic documents three, split the same way: ClaudeBot collects training data, while Claude-User and Claude-SearchBot serve live questions and search. Perplexity runs two.

Missing from Anthropic's page are anthropic-ai and Claude-Web, two names that still circulate in robots.txt templates. A rule aimed at them blocks nothing the vendor admits to sending.

User-triggered fetchers sit outside robots.txt

Crawlers that build an index obey robots.txt, by the vendors' own account. Anthropic's help page says its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt." Fetchers started by a person are a different matter.

OpenAI describes ChatGPT-User as an agent that "is not used for crawling the web in an automatic fashion," and adds that "robots.txt rules may not apply" because a user starts the action. Perplexity's bot guide is blunter about Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."

For a site owner, the split has a practical side. A Disallow line can keep a page out of a training set, yet the same page may still be fetched for a ChatGPT user five minutes later. Only a log shows that second kind of visit.

A short mu-plugin records every match

WordPress needs no plugin for a basic log. The file below goes into wp-content/mu-plugins/, hooks into template_redirect at priority 1, and compares the user agent of each request with the nine documented names.

<?php
/**
 * Plugin Name: AI Crawler Log
 * Description: Records visits from documented AI user agents.
 */

add_action( 'template_redirect', function () {
    if ( empty( $_SERVER['HTTP_USER_AGENT'] ) ) {
        return;
    }

    $agent = sanitize_text_field( wp_unslash( $_SERVER['HTTP_USER_AGENT'] ) );
    $names = array(
        'GPTBot', 'OAI-SearchBot', 'ChatGPT-User', 'OAI-AdsBot',
        'ClaudeBot', 'Claude-User', 'Claude-SearchBot',
        'PerplexityBot', 'Perplexity-User',
    );

    foreach ( $names as $name ) {
        if ( false === stripos( $agent, $name ) ) {
            continue;
        }

        $log   = get_option( 'ai_crawler_log', array() );
        $log[] = array(
            'bot'  => $name,
            'url'  => esc_url_raw( wp_unslash( $_SERVER['REQUEST_URI'] ?? '/' ) ),
            'time' => gmdate( 'Y-m-d H:i:s' ),
        );

        // Keep the last 500 visits; "false" keeps the option out of autoload.
        update_option( 'ai_crawler_log', array_slice( $log, -500 ), false );
        break;
    }
}, 1 );

Each match lands in a single option, ai_crawler_log, stored with autoload switched off so that ordinary page loads never carry it. The array_slice() call caps the record at 500 visits. On a local WordPress 7.1 test install, a request sent with the GPTBot agent string appeared in the option within the same second, while a Chrome agent left no trace. Reading the result takes one WP-CLI command: wp option get ai_crawler_log --format=json.

Owners with shell access can skip PHP and count matches in the server log instead:

grep -Eio "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User" access.log | sort | uniq -c | sort -rn

Page caches and spoofed agents distort the count

PHP-based logging has a blind spot. When a full-page cache (WP Rocket or a CDN edge, for instance) answers the request, WordPress never starts and the visit goes unrecorded. The server access log catches those hits; a PHP log on a cached site shows only cache misses.

User agent strings are also trivial to fake. Any script can call itself GPTBot, which is why all three vendors publish their IP ranges as JSON files (openai.com/gptbot.json is one of them). A visit from outside those ranges is an impostor.

The opposite error exists too. Cloudflare reported on August 4, 2025 that Perplexity used, besides its declared agents, "a generic browser intended to impersonate Google Chrome on macOS when their declared crawler was blocked." No list of names catches a crawler that refuses to give one.

Cloudflare counted a 305% rise in GPTBot requests

Traffic from these agents is no longer marginal. GPTBot "increased its share from 2.2% to 7.7%" of crawler requests in the year to May 2025, João Tomé of Cloudflare wrote in a study of crawler traffic published that July. Raw request volume from the same bot, the figure in the heading above, more than quadrupled.

Growth was uneven. In the same dataset ClaudeBot, Anthropic's training crawler, "fell from 11.7% to 5.4% of total traffic." One site's log can look nothing like the global chart, and a quiet month for one vendor says little about the others.

Blocking remains rare among large sites. Of the 3,816 robots.txt files that Cloudflare could read among top domains, 546 addressed AI bots at all. GPTBot led the refusals: the study found it "disallowed by 312 domains."

Prime SEO 1.1.9 matches 14 names, two of them in vain

Prime SEO, the plugin named in the disclosure, ships the same stripos() approach behind a dashboard screen called AI Crawler Stats, which also records requests for llms.txt and the XML sitemap. Version 1.1.9 matches 14 substrings. Two of them are Google-Extended and Applebot-Extended, which, per the documentation quoted above, can never match. Those two counters will stay at zero.

The plugin's list also predates Claude-User, Claude-SearchBot and Perplexity-User, so it undercounts user-triggered visits today; an update is planned for version 1.2.0. Background on the file these agents are meant to read is in the guide to what llms.txt is and on the llms.txt plugin for WordPress page.

Vendor lists change faster than robots.txt files

All three vendors now separate training crawlers from agents that act for a person. If that split continues, a robots.txt file written for GPTBot and ClaudeBot alone will govern a shrinking part of AI traffic, and the raw access log (not the rule file) becomes the place where a new agent name first shows up. Until a vendor page changes, nine names are enough.

Originally published on the Prime SEO blog.