AI training crawlers vs search crawlers: why the difference matters

AI crawlers do one of three jobs: training crawlers collect content that may be used to build models, search crawlers build the index an AI assistant draws on when it answers with links, and user-triggered fetchers load a single page because someone asked the assistant to. The difference matters because blocking each type costs you something different. You can opt out of training and stay visible in AI search, because most operators now give the two jobs separate controls.

The facts below come from each operator's own documentation, read on 8 November 2026. Crawler names and rules change, so treat anything here as "at the time of writing" and check the linked source before you change a live file.

What are the three types of AI crawler?

Training crawlers fetch public pages in bulk, and the content may end up in a dataset used to train future models. Disallowing them is a signal about future use: OpenAI says disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models", and Anthropic says restricting ClaudeBot signals that your future materials should be excluded from training datasets. It does not delete anything already collected.

Search crawlers build and refresh an index, much as Googlebot and Bingbot always have. When an assistant searches the web and cites sources, it draws on an index like this. If a search crawler cannot read your pages, that tool has little or nothing of yours to cite.

User-triggered fetchers are not crawling in the usual sense. They fetch one URL, in real time, because a person pasted it in or the assistant decided it needed that page to answer a question. Because a human started the request, several operators say robots.txt may not apply to them.

Some agents do more than one job, which is where most confusion starts. Applebot powers Apple's search features and its data may also help train Apple's models. Googlebot crawls for Google Search, and Google's AI features are part of Search.

Which bot does what?

This table maps the major agents to their purpose and to the control that governs them. For full user-agent strings and every smaller bot, see the AI crawler user agents list.

Sources: OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft, Common Crawl, Meta, Amazon, DuckDuckGo and Mistral. Perplexity's and DuckDuckGo's statements about training are their own and cannot be checked from outside.

What does blocking each type cost you?

Two lines in that table need detail.

Search crawlers are the ones that decide AI search visibility

OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic says disabling Claude-SearchBot may reduce your visibility in user search results, and disabling Claude-User may reduce it in user-directed web search. Perplexity tells site owners who want to appear in its results to allow PerplexityBot. A rule that blocks one of these is a real cost, so check that a template you copied does not block them by accident.

Google is the exception

Google's AI features page says AI is built into Search, so your Googlebot rules are what control crawling for AI Overviews and AI Mode. Google-Extended covers training of future Gemini models and grounding in Gemini Apps and Vertex AI, and Google says it does not affect inclusion or ranking in Google Search. So blocking Google-Extended keeps you in AI Overviews, and you cannot leave AI Overviews with robots.txt without leaving Search too. What you can do is limit how much of a page is used, with nosnippet, data-nosnippet or max-snippet. The Google-Extended guide covers the details.

Why opting out of training need not hurt search visibility

Because the controls are separate. OpenAI states that its OAI-SearchBot and GPTBot settings are independent, and its own example is a site that allows OAI-SearchBot and disallows GPTBot. Anthropic, Apple, Google and Mistral all document a training control apart from their search crawler, and Amazon says its agents respect a page-level noarchive robots meta tag, which it defines as "do not use the page for model training". Crawlito's score follows the same logic: it labels each crawler by purpose, and blocking a training crawler on purpose never lowers your search-readiness score.

One detail to know: OpenAI says that if you allow both GPTBot and OAI-SearchBot, it may use the results of one crawl for both purposes. So opting out of training does not stop OAI-SearchBot from crawling your pages; it changes how the content may be used.

How to decide your AI crawler policy

Work through these questions in order. A common outcome is "allow search, decide on training".

1. Do you want to be found and cited in AI answers? If yes, allow the search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Applebot. This is the decision that affects visibility, so make it deliberately.
2. Is your content the product? Publishers, course creators, paid research and stock libraries often opt out of training, because the content itself is what they sell. For a local plumber or a SaaS marketing site, the training decision matters less; either choice leaves search visibility as it is.
3. Do you have contracts or licences that restrict reuse? If content belongs to clients or partners, a training opt-out is a reasonable default. Use path rules if it is only part of the site.
4. Is there a section that should stay out of everything? Member areas and account pages belong behind a login, not just in robots.txt. robots.txt is a public request, not access control.
5. Will your CDN agree with your robots.txt? A firewall rule that challenges a search crawler undoes an Allow line. On Cloudflare, the AI Crawler and Agent access policies set separate actions for Search, Agent and Training: Block, Block on pages with ads, Allow, and a Disallow action for Training only that is expressed through robots.txt. New domains default to Allow for all three; the Cloudflare settings guide shows where each one lives.

Newer machine-readable signals are emerging. Since 30 October 2026, Anthropic recognises a "no AI training" preference based on an IETF draft, and Google says it intends to support the AIPREF training signal in early 2027. Both are early and may change. Named rules in robots.txt remain the documented control today, and Anthropic says a rule that names ClaudeBot takes precedence over the general signal.

Example: allow AI search, opt out of training

This file opts out of the documented training crawlers and tokens in the first group, and leaves every other crawler, search crawlers included, on the general * rules:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: MistralAI-Training
Disallow: /

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Notes before you copy it:

• Search crawlers follow the * group here because no group names them. If you later add a named group for OAI-SearchBot, it will ignore the * rules, and you need to repeat Disallow: /admin/ inside it. See how robots.txt groups work.
• meta-externalagent covers more than training. Meta says it also crawls to index content to improve products, so blocking it may have effects beyond a training opt-out.
• Amazonbot is left out on purpose. Amazon says it improves Amazon products and may be used to train its AI models, so it is mixed. Decide for your own case.
• Allow time. OpenAI and Perplexity say robots.txt changes can take around 24 hours to take effect, Meta says up to a day, and DuckDuckGo says 72 hours for DuckAssistBot.

For per-vendor variations, including how to opt out of training on one directory only, see how to allow AI search but block AI training.

How to check what your site tells each crawler

Read your live file, not the copy in your repository:

curl -s https://example.com/robots.txt

Then check each token against it. This uses Python's built-in parser, which is a quick approximation: it applies rules in file order rather than by the longest match that RFC 9309 specifies, and it does not understand wildcards. It is reliable for whole-site Disallow: / checks like this one:

python3 - <<'EOF'
from urllib.robotparser import RobotFileParser
rp = RobotFileParser("https://example.com/robots.txt"); rp.read()
for bot in ["GPTBot", "OAI-SearchBot", "ClaudeBot", "Claude-SearchBot",
            "PerplexityBot", "Googlebot", "Google-Extended", "Bingbot",
            "Applebot", "Applebot-Extended", "CCBot"]:
    print(f"{bot:18} {'allowed' if rp.can_fetch(bot, 'https://example.com/') else 'blocked'}")
EOF

Against the example file above it prints blocked for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot, and allowed for every search crawler. If a search crawler shows blocked, that is the line to fix first. One more difference from RFC 9309: if robots.txt returns 401 or 403, Python's parser reports every bot as blocked, whereas under the RFC any 4xx response means no restrictions apply. Confirm the status with curl -sI https://example.com/robots.txt before trusting the output.

robots.txt is only half the picture. OpenAI recommends allowing requests from OAI-SearchBot's published IP ranges as well, and Perplexity recommends WAF rules that match both the user agent and its published IP ranges. Check that your CDN or firewall is not challenging those requests.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot only signals whether your content may be used for training. ChatGPT search eligibility is controlled by OAI-SearchBot, and OpenAI documents the two controls as independent.

Does blocking Google-Extended keep my content out of AI Overviews?

No. Google says AI Overviews and AI Mode are part of Search and controlled by Googlebot rules. Google-Extended covers Gemini model training and grounding in Gemini Apps and Vertex AI, and it does not affect Google Search.

Can I block user-triggered fetchers with robots.txt?

Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. Anthropic is an exception: it says all three of its bots, Claude-User included, honor robots.txt.

Is PerplexityBot a training crawler?

Perplexity says no: it describes PerplexityBot as a search crawler that is not used to crawl content for AI foundation models. That is Perplexity's own statement.

Does opting out of training remove content already collected?

No. Operators describe a disallow as a signal about future use. Anthropic's wording, for example, covers excluding your future materials from training datasets.

Next step

If you are not sure which of these crawlers your live robots.txt and CDN actually let through, run the free single-page check on one of your pages. It needs no account, reads the robots.txt rule for each named AI crawler, labels each crawler by purpose and flags CDN or WAF challenge pages.