How robots.txt works for AI crawlers: the RFC 9309 matching rules, how GPTBot, ClaudeBot and others read the file, and three templates you can copy.
By Crawlito · 9 min read
In short
A crawler that finds a robots.txt group naming it follows only that group, so a GPTBot group means GPTBot ignores every rule under User-agent: *.
Within a group, the longest matching path wins regardless of rule order; RFC 9309 says Allow should win only when an Allow and a Disallow match with equal length.
robots.txt applies only to the exact host, protocol and port it is served from, so www, subdomains and http need their own file.
A 404 robots.txt means no restrictions, but a 5xx error makes RFC 9309 crawlers assume the whole site is disallowed.
robots.txt is a voluntary signal, not access control: user-triggered agents such as ChatGPT-User and Perplexity-User may not apply it.
robots.txt is a plain text file at the root of your host that tells each crawler, by name, which paths it may fetch. AI crawlers that respect it, including GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Googlebot, read it under the rules of RFC 9309: they follow the group that names them (or the * group if none does), then apply the longest matching path. Get those two rules right and you can allow AI search, block AI training, or opt out of both with a few lines.
This guide covers the syntax, how each major AI operator reads the file, three templates, where the file must live, how crawlers cache it, and what it cannot do. Facts about each crawler come from the operators' own documentation, read on 8 November 2026.
How robots.txt rules are read under RFC 9309
RFC 9309 is the IETF standard for the Robots Exclusion Protocol, published in September 2022. Google's robots.txt specification follows the same model, and Anthropic, for example, says its bots honour "industry-standard robots.txt directives".
Groups and user-agent matching
A file is made of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules. A crawler picks its group like this:
It looks for groups whose User-agent matches its product token, such as GPTBot or ClaudeBot. The match is case-insensitive, so gptbot works too.
If several groups match, it combines their rules into one.
If no group matches, it uses the User-agent: * group.
If there is no * group either, no rules apply and everything is allowed.
The consequence most people miss: a crawler with a named group ignores the * group completely. The * group is only a fallback, Google's specification describes the same selection, and Bing states it explicitly for bingbot.
Here GPTBot may fetch /admin/ and /checkout/, because its own group does not repeat those rules. If they should stay off limits, copy them into the GPTBot group. The robots.txt groups guide works through more cases.
You can list several user agents above one set of rules, which keeps long files readable:
Within the chosen group, the crawler checks the URL path against every rule. The most specific rule wins, meaning the one with the most characters (octets). The order of lines does not matter. If an Allow and a Disallow match with the same length, RFC 9309 says the Allow should be used. A path that matches no rule is allowed.
RFC 9309 says path matching should be case-sensitive, and Google treats rule values that way, so Disallow: /Private/ does not block /private/. Lines starting with # are comments. A Sitemap: line with an absolute URL is not tied to any user agent, and under RFC 9309 it does not end the group it sits in. Crawl-delay is not in RFC 9309 at all, and support varies by crawler, as the next section shows.
How each AI crawler reads robots.txt
Every operator below documents its own product tokens. The important split is purpose: training, search, or a fetch triggered by a user's question. The AI crawler user agents list has the full strings and IP lists.
Operator
Token
Purpose
Follows robots.txt?
OpenAI
OAI-SearchBot
ChatGPT search
Yes
OpenAI
GPTBot
Training
Yes
OpenAI
ChatGPT-User
User-triggered fetches
OpenAI: "robots.txt rules may not apply"
Anthropic
Claude-SearchBot
Search result quality
Yes
Anthropic
ClaudeBot
Training
Yes
Anthropic
Claude-User
User-triggered fetches
Yes, per Anthropic
Perplexity
PerplexityBot
Perplexity search
Yes
Perplexity
Perplexity-User
User-triggered fetches
"Generally ignores" robots.txt, per Perplexity
Google
Googlebot
Google Search, including AI Overviews and AI Mode
Yes
Google
Google-Extended
Token only: Gemini training and grounding
Yes, as a token; Google crawls with its existing user agents
Apple
Applebot
Spotlight, Siri, Safari; may also train
Yes, falls back to Googlebot rules
Apple
Applebot-Extended
Token only: opts out of Apple model training
Yes, as a token; it governs use of data Applebot crawled
Anthropic says all three of its bots honour robots.txt, supports the non-standard Crawl-delay, and asks you to add rules to every subdomain you want to opt out. It also warns that blocking Claude-SearchBot or Claude-User may reduce your visibility in user search results.
Perplexity says PerplexityBot is not used to crawl content for AI foundation models, and that Perplexity-User generally ignores robots.txt because a person asked for the page.
Google says AI Overviews and AI Mode are part of Search and are controlled by your Googlebot rules, plus nosnippet, data-nosnippet, max-snippet and noindex. Google-Extended has no user agent of its own and does not affect Search inclusion or ranking.
Apple says that if your file does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. Applebot does not follow Crawl-delay.
Bing honours Crawl-delay, as does CCBot. Google, Applebot and Amazonbot do not.
Three robots.txt templates for AI crawlers
Each template is a complete file. Replace the sitemap URL, and repeat any private paths inside every named group, for the reason explained above.
Every crawler that follows robots.txt may fetch every page. Check that nothing else in the file adds a named group with Disallow: /, because that group would override this one for the crawler it names.
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Applebot and bingbot fall through to the * group and stay allowed. Three trade-offs to know: Google-Extended also covers grounding in Gemini Apps and Vertex AI, Meta-ExternalAgent also covers Meta's product indexing, and Amazonbot is left out because Amazon says it both improves products and may train models. The allow AI search, block AI training guide covers each vendor's choice in more depth.
For the crawlers that comply, this takes you out of ChatGPT search answers (OpenAI says opted-out sites can still appear as navigational links), Anthropic says blocking Claude-SearchBot and Claude-User may reduce your visibility in its search results, and Perplexity recommends allowing PerplexityBot for a site to appear in its results. It cannot take you out of Google's AI Overviews without also leaving Google Search, because both use Googlebot; nosnippet or max-snippet limit that, but they also limit your ordinary search snippets. ChatGPT-User and Perplexity-User are listed as a signal, but their operators say robots.txt may not apply. The list is not exhaustive: new tokens appear often, so check each operator's documentation. Bytespider often appears in templates like this; no official ByteDance documentation could be found for it, so if you need it gone, block it at your firewall.
Where robots.txt must live and how crawlers fetch it
Follows at least five hops, then treats it as a 404
4xx
No restrictions: crawl anything
Same, except 429, which counts as a server error
5xx or network error
Assume everything is disallowed
Stops crawling for 12 hours, then uses the last good copy for up to 30 days
A 404 is safe. A robots.txt that returns 503 during a deploy is not: it reads as "disallow everything". A CDN challenge or block on robots.txt is a problem either way: a 403 reads as "no restrictions" and a 5xx as "disallow everything", and the same rule usually blocks your pages too. Make sure your firewall serves robots.txt to bots with a plain 200.
How long a change takes
RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable. Operators document their own windows: Google generally caches for up to 24 hours, OpenAI says its search systems need about 24 hours, Perplexity says up to 24 hours, Meta up to 24 hours, Bing at least a few hours, DuckAssistBot about 72 hours, and Amazonbot may use a copy cached within the last 30 days. Plan robots.txt changes a few days ahead of anything that depends on them.
What robots.txt cannot do
It is not access control. Compliance is voluntary. Cloudflare, Squarespace, Wix and Webflow all say so in their own docs. To actually stop a crawler, block it at the server or CDN, ideally by matching both its user agent and its published IP ranges.
It does not bind user-triggered agents. OpenAI, Perplexity, Meta and Amazon each say their user-requested fetchers may not follow robots.txt.
It does not remove pages from an index. A disallowed URL can still be listed if other sites link to it, and the crawler never sees a noindex on a page it cannot fetch. The difference is covered in robots.txt vs meta robots.
It does not recall content already collected. A training opt-out applies to future crawling.
It is public. Anyone can read it, so listing a private path in Disallow advertises it.
How to check your live robots.txt
The file a crawler sees is often not the one in your repository. Cloudflare's managed robots.txt can prepend AI rules, Shopify and WordPress generate the file, and a staging rule can leak to production.
Fetch it from every host you serve:
bash
for host in example.com www.example.com shop.example.com; do
echo "== $host"; curl -s -o /dev/null -w "%{http_code} %{content_type}\n" "https://$host/robots.txt"
done
You want 200 and text/plain on each.
Fetch it as an AI crawler, to catch firewall rules keyed on the user agent: curl -s -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://example.com/robots.txt. A challenge page instead of rules is a problem; the Cloudflare AI crawler settings guide shows where those come from.
Read the groups as each crawler would: find the group naming it, or fall back to *, then apply the longest match to a few key URLs. The robots.txt testing guide shows how with Google's open-source parser.
Check Google's view in the Search Console robots.txt report, which shows the files Google fetched for your top 20 hosts and lets you request a recrawl after a fix.
Frequently asked questions
Does a GPTBot group override my User-agent: * rules?
Yes. A crawler that finds a group naming it follows only that group and ignores *. Repeat any general rules you still want inside the named group.
Does blocking GPTBot or Google-Extended hide my site from AI search?
No. GPTBot and Google-Extended are training controls. ChatGPT search is controlled by OAI-SearchBot, and Google's AI Overviews and AI Mode by Googlebot and snippet controls.
What happens if my robots.txt returns an error?
A 4xx such as 404 means no restrictions. A 5xx or timeout makes RFC 9309 crawlers treat the whole site as disallowed, and Google pauses crawling for 12 hours before falling back to a cached copy.
Do I need a separate robots.txt for www and subdomains?
Yes. Rules apply only to the host, protocol and port that serve the file, so www.example.com, blog.example.com and http:// each need their own.
Can robots.txt stop ChatGPT or Perplexity from opening a page a user asks about?
Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Only a server or firewall rule enforces that.
Next step
To see what your live robots.txt says to each named AI crawler, and whether a CDN challenge gets in the way first, run the free single-page check. For every host and template on a site, a full audit checks up to 500 pages and lists the fixes in priority order.
Filed under Access and robots.txt. Topics: · robots-txt, ai-crawlers, gptbot, claudebot, perplexity, google-extended
A reference list of AI crawler user agents from OpenAI, Anthropic, Perplexity, Google, Apple and others, with each bot's purpose, robots.txt token and docs.