How to allow AI search but block AI training in robots.txt
A copy-paste robots.txt that blocks AI training (GPTBot, ClaudeBot, Google-Extended, CCBot) and allows AI search crawlers, with every line explained.
By Crawlito · 10 min read
In short
Training and search use separate robots.txt tokens, so you can disallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot while allowing OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot.
Google-Extended and Applebot-Extended are opt-out tokens, not crawlers: blocking them leaves Google Search, AI Overviews and Apple search untouched.
Do not put user-triggered agents such as ChatGPT-User or Perplexity-User in the training block: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
A search crawler that has its own robots.txt group ignores the * group, so repeat any private-path Disallow lines inside it.
robots.txt is a voluntary signal and operators keep adding new tokens, so review the file against your server logs regularly and use a firewall or CDN rule for anything you must enforce.
You can allow AI search and block AI training with one robots.txt file, because the major AI companies use different tokens for each job. Disallow the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) and leave the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) allowed. Your pages stay eligible for ChatGPT search, Claude, Perplexity, Google and Bing while you signal that your content should not be used to train models.
OpenAI uses exactly this split, GPTBot disallowed and OAI-SearchBot allowed, as the example in its crawler documentation. Below is a file you can copy, what each line does, and the caveats that the short version leaves out. Facts are from each operator's documentation as read on 8 November 2026.
Which tokens control training and which control search
Signals future content should be excluded from training datasets
Google
Google-Extended
Googlebot
Opts out of Gemini training and grounding; Google says it does not affect Search inclusion or ranking
Apple
Applebot-Extended
Applebot
Opts out of training Apple's foundation models; pages can still appear in Apple search
Common Crawl
CCBot
none
Stops CCBot adding your pages to Common Crawl's open dataset, which anyone can download
Perplexity
none
PerplexityBot
Perplexity says PerplexityBot is not used to crawl content for AI foundation models
Microsoft
none
Bingbot
Copilot grounds its answers through the Bing search service, so Bingbot access matters
Two of the training tokens are not crawlers at all. Google says Google-Extended has no separate user agent and is used only in robots.txt; crawling is done by Google's existing crawlers. Apple says Applebot-Extended does not crawl web pages and only decides how data Applebot already collected may be used. You will never see either of them in your logs. Google-Extended explained covers the Google side in more detail.
The copy-paste robots.txt
The file has three groups: training crawlers that are opted out, search crawlers that are allowed apart from private paths, and a * group for everyone else. Replace the private paths and the sitemap URL with your own. If you have no private paths, replace the Disallow lines in the last two groups with a single Allow: /.
The file has to live at the top level of each host, for example https://www.example.com/robots.txt. Its rules apply only to the host, protocol and port it is served from, so shop.example.com and blog.example.com each need their own copy. Anthropic says the same for ClaudeBot: add the rules to every subdomain you want to opt out.
What each line does
The training group
Several User-agent lines stacked above one set of rules form a single group under RFC 9309, the robots.txt standard. Every crawler named in it gets Disallow: /, meaning nothing on the site. Product tokens are matched case-insensitively, so gptbot and GPTBot are the same. If you prefer, write each token as its own two-line group; the result is identical.
GPTBot. OpenAI says disallowing GPTBot indicates your content should not be used to train its generative AI foundation models. Because its controls are independent, OAI-SearchBot can still surface your pages in ChatGPT search.
ClaudeBot. Anthropic's crawler article says restricting ClaudeBot signals that your future materials should be excluded from AI training datasets.
Google-Extended. Covers training future Gemini models and grounding in Gemini Apps and the Vertex AI API. It does not touch Google Search, and that includes AI Overviews and AI Mode: Google's AI features page says those are controlled by your Googlebot rules and by snippet controls such as nosnippet.
Applebot-Extended. Apple says it is not considered in search ranking and that pages which disallow it can still appear in Apple search results.
CCBot. Common Crawl's CCBot page gives this exact block, and its FAQ says CCBot checks robots.txt before fetching and obeys Crawl-delay.
The search group
Strictly, you do not need this group: crawlers that are not named fall back to *, which allows them. Naming them has two benefits. It documents your intent, and it keeps them allowed if someone later tightens the * group.
The catch is the rule that trips most people up. A crawler that finds a group naming it follows only that group and ignores *. Bing states this plainly for Bingbot, and RFC 9309 requires it. That is why the private paths appear twice: without them, OAI-SearchBot and the others would be allowed into /admin/ even though * blocks it. How robots.txt groups work walks through more examples.
A few notes on individual lines:
Claude-User sits here, not in the training block. It is a user-triggered fetcher rather than a search crawler, but Anthropic says all three of its bots honor robots.txt, and that disabling Claude-User or Claude-SearchBot may reduce your site's visibility when Claude users search.
Applebot is named on purpose. Apple says that if robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. Naming it removes the guesswork.
Googlebot covers Google Search and its AI features. Smartphone and desktop Googlebot share the one token.
The sitemap line
Sitemap: is not part of any group, and it can sit anywhere in the file. It points any crawler that supports sitemaps at your list of URLs.
Caveats the short version leaves out
User-triggered agents are a different decision
When someone asks ChatGPT, Perplexity or Claude about a specific page, a user-triggered agent fetches it. They are not the tokens these operators document for training opt-outs, and several of them do not follow robots.txt in the usual way:
Agent
What the operator says about robots.txt
ChatGPT-User
OpenAI: because actions are user-initiated, "robots.txt rules may not apply"
Anthropic: honors robots.txt; blocking it may reduce visibility
Meta-ExternalFetcher
Meta: may bypass robots.txt because the fetch was requested by a user
Adding these to the training block is not how you opt out of training: OpenAI and Anthropic name GPTBot and ClaudeBot for that, and Perplexity says Perplexity-User is not used to collect content for training. Where the rule works at all, it stops assistants from reading your page when a person asks about it. Leave them out unless you want that outcome.
New bots appear, and some tokens cover more than one job
Any list like the one above is a snapshot. Operators add new tokens and rename old ones: Anthropic's current article names ClaudeBot, Claude-SearchBot and Claude-User, while many templates still carry the older anthropic-ai and Claude-Web. Some tokens you may want to consider, with the caveats from each operator's page:
Meta-ExternalAgent: crawls for uses such as training AI models or indexing content to improve products. Meta documents Meta-WebIndexer separately for Meta AI search citations.
Amazonbot: improves Amazon products and services and may be used to train Amazon AI models. Amzn-SearchBot handles search experiences such as Alexa.
MistralAI-Training: builds Mistral's training datasets. MistralAI-Index handles indexing for Mistral search.
Bytespider: ByteDance's crawler. We found no official ByteDance documentation, and third parties report that it ignores robots.txt, so do not rely on a robots.txt rule alone.
Review the file every few months against a current AI crawler user agent list and against the user agents that actually show up in your server logs.
Some sites go the other way: User-agent: * with Disallow: /, plus named groups for the search crawlers they want. That blocks unknown training bots by default, but it also blocks every new search engine, link preview and assistant you have not named, so weigh it carefully.
robots.txt is voluntary
robots.txt asks; it does not enforce. Cloudflare's own managed robots.txt documentation says compliance is voluntary and points to its firewall tools for enforcement. The operators of the crawlers in the file above document robots.txt as the way to opt out, but a crawler that ignores it can only be stopped at the server, CDN or firewall.
A training opt-out also looks forward. Anthropic describes the ClaudeBot block as excluding your future materials, not content collected before you added the rule.
Your firewall has to agree
The reverse problem is common too. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges. A CDN bot rule that challenges those IPs can keep you out of ChatGPT search even when robots.txt is correct.
On Cloudflare, the "AI Crawler and Agent access policies" setting has a "Disallow (Training only, expressed through robots.txt)" option, and Bot Preference Sync can add a managed block to the top of your robots.txt. At the time of writing that block disallows Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. If you use it, read the robots.txt your site actually serves, not just the file at your origin. Cloudflare and AI crawlers covers the settings.
Changes can take a day or more
OpenAI says its search systems can take about 24 hours to adjust after a robots.txt change, Perplexity says up to 24 hours, and Google generally caches robots.txt for up to 24 hours. DuckDuckGo says DuckAssistBot opt-outs take about 72 hours.
Newer training signals
Anthropic says that from 30 October 2026 it recognizes a machine-readable "no AI training" signal based on an IETF draft, and that a robots.txt rule naming ClaudeBot takes precedence over it. Google says it intends to support an AIPREF training-preference signal in early 2027. These are early and still changing, so treat named robots.txt groups as the dependable method for now.
How to check the file works
Fetch it the way a crawler does. It should return a 200 with plain text, not an HTML error page or a challenge.
Under RFC 9309, a 4xx on robots.txt means no restrictions at all (your training block disappears), and a 5xx means crawlers should assume everything is disallowed (your search access disappears).
Test each token against the file. Python's built-in parser is enough for a sanity check of a file this simple:
bash
python3 - <<'EOF'
from urllib.robotparser import RobotFileParser
rp = RobotFileParser("https://www.example.com/robots.txt")
rp.read()
for bot in ["GPTBot", "ClaudeBot", "Google-Extended", "Applebot-Extended", "CCBot",
"OAI-SearchBot", "Claude-SearchBot", "PerplexityBot", "Googlebot", "Bingbot"]:
print(f"{bot:18} {rp.can_fetch(bot, 'https://www.example.com/pricing')}")
EOF
The first five should print False and the last five True. If every line prints False, check the fetch first: Python treats a 401 or 403 on robots.txt as "disallow everything", while RFC 9309 treats any 4xx as no restrictions. Python's parser is also not a complete RFC 9309 implementation, and its matching has changed between versions, so for files with wildcards or overlapping Allow and Disallow paths, use a standards-based tester. How to test your robots.txt against each AI crawler covers the options.
Check your firewall separately. Request a page with a search crawler's user agent and confirm you get your content, not a challenge. A pass here only shows that user-agent rules do not block it; IP-based rules can still apply to the real crawler.
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI says the GPTBot and OAI-SearchBot settings are independent. GPTBot controls training use; OAI-SearchBot controls whether your pages can be shown in ChatGPT search answers.
Does blocking Google-Extended keep my content out of AI Overviews?
No. Google says AI Overviews and AI Mode are part of Search and are controlled by your Googlebot rules and by nosnippet, data-nosnippet, max-snippet and noindex. Google-Extended covers Gemini model training and grounding outside Search.
Should I block ChatGPT-User or Perplexity-User to stop training?
No. GPTBot is OpenAI's training opt-out, and Perplexity says Perplexity-User is not used to collect training data. Both operators also say robots.txt may not apply to these agents because a person asked for the page, so a rule mainly risks stopping assistants from reading your page when someone asks about it.
Do I need named groups for the search crawlers?
Not strictly: unnamed crawlers follow the * group. Named groups make your intent explicit, but each one needs its own copy of your private-path Disallow lines, because a named crawler ignores *.
Will this policy hurt my visibility in AI search?
Blocking the training tokens does not, by itself, change your eligibility for any of the search products in this post. Whether an assistant then cites your pages depends on many things beyond robots.txt.
Check what your site actually serves
The file you meant to publish and the file crawlers receive are not always the same, especially behind a CDN or a platform that writes its own robots.txt. Run the free Crawlito check on a page to see the robots.txt rule each AI crawler gets, labeled by purpose, and whether a firewall challenge is in the way. It checks a single page and needs no account.
Filed under Access and robots.txt. Topics: · robots-txt, ai-crawlers, gptbot, claudebot, google-extended, applebot
AI training crawlers vs search crawlers vs user-triggered fetchers: what each does, what blocking costs, and how to opt out of training but stay in AI search.
A reference list of AI crawler user agents from OpenAI, Anthropic, Perplexity, Google, Apple and others, with each bot's purpose, robots.txt token and docs.