Should you block AI crawlers? A decision guide by site type
Should you block GPTBot, ClaudeBot and other AI crawlers? Decide by crawler purpose and site type, with the trade-offs and which choices you can undo later.
By Crawlito · 7 min read
In short
- Decide per crawler purpose, not per company: training crawlers, search crawlers and user-triggered fetchers have separate robots.txt tokens and different costs when blocked.
- OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links, so blocking a search crawler has a direct visibility cost.
- Blocking a training crawler is a business decision about future use: Anthropic says disallowing ClaudeBot signals that future materials should be excluded from training, and no operator describes robots.txt as removing data already collected.
- robots.txt is a request: Perplexity says Perplexity-User generally ignores it and OpenAI says it may not apply to ChatGPT-User, so anything you must enforce needs a login or a firewall rule.
- robots.txt changes are the easiest to reverse; OpenAI says its search systems adjust about 24 hours after a robots.txt update.
On this page
For most sites, the answer is: do not block AI search crawlers, decide deliberately about training crawlers, and leave user-triggered fetchers alone. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot are how AI assistants find pages to cite, so blocking them removes you from those answers. Training crawlers such as GPTBot and ClaudeBot are a separate choice about whether your future content may be used to train models, and the right answer depends on what your content is worth to you.
This guide works through that decision by crawler purpose and by type of site, with the trade-offs and the parts that are easy or hard to undo. It assumes you know the basic split; if not, start with training, search and user-triggered crawlers.
Decide by crawler purpose first
Each major AI company runs several agents, and blocking "OpenAI" or "Anthropic" as a whole is rarely what anyone wants. Here is the cost of blocking each kind, based on the operators' own documentation at the time of writing.
| Purpose | Examples | What blocking costs you | Typical default |
|---|---|---|---|
| Search and indexing | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot | Visibility in that tool's answers, or in search altogether | Allow |
| Model training | GPTBot, ClaudeBot, CCBot, plus the Google-Extended and Applebot-Extended tokens | Nothing in AI search, per the operators; your future content stays out of that operator's training data | Your call |
| User-triggered fetches | ChatGPT-User, Claude-User, Perplexity-User | The assistant cannot read your page when a person asks about it | Allow |
The operators' statements behind that table:
- OpenAI says sites opted out of
OAI-SearchBotwill not be shown in ChatGPT search answers, though they can still appear as navigational links. It describesGPTBotas used to make its foundation models more useful and safe, and says that becauseChatGPT-Useractions are started by a user, robots.txt rules may not apply. - Anthropic says disabling
ClaudeBotsignals that your future materials should be excluded from training, while disablingClaude-SearchBotorClaude-Usermay reduce your visibility in search and user-directed answers. - Perplexity says
PerplexityBotis not used to crawl content for AI foundation models and thatPerplexity-Usergenerally ignores robots.txt because a user requested the fetch. - Google says
Google-Extendeddoes not affect a site's inclusion in Google Search and is not a ranking signal there.
Decide by site type
The search question has the same answer for nearly everyone. The training question is where site types differ. These are reasonable starting points, not rules.
| Site type | AI search crawlers | Training crawlers | Why |
|---|---|---|---|
| Local business, services | Allow | Allow or block; low stakes | You want to be recommended; your pages are marketing, not the product |
| SaaS and B2B | Allow | Usually allow | Product and pricing pages exist to be read and explained accurately |
| Ecommerce | Allow | Usually allow | Product facts help assistants answer shopping questions; catalogue text is rarely the asset |
| News and magazine publishers | Allow, unless you license search access | Often block | The writing is the product; many publishers negotiate licences instead |
| Paid courses, research, data | Allow public pages | Often block | Keep paid content behind a login regardless of robots.txt |
| Documentation and developer tools | Allow | Usually allow | Developers ask assistants about your API; accurate answers reduce support load |
| Forums and user-generated content | Allow | Check your terms | You may not have rights to license users' posts for training |
| Agencies hosting client content | Allow | Follow each client's contract | The decision belongs to the content owner |
Two situations override the table:
- Contracts. If a licence, client agreement or syndication deal restricts reuse, the training block follows the contract, not a blog post.
- Load. If a crawler is hurting your servers, that is an operational problem with an operational fix. A rate limit for that crawler is better than a block that also removes you from its answers; see rate limiting AI crawlers.
Trade-offs most decision guides skip
Blocking training does not remove what was already collected
Anthropic says disallowing ClaudeBot signals that your future materials should be excluded from training, and neither OpenAI nor Anthropic describes a robots.txt rule as removing content already collected. Treat a Disallow added today as a signal about future crawls, not a deletion request. If removal matters to you, that is a conversation with the operator, not a robots.txt line.
Blocking training is not a search opt-out
OpenAI says that if you allow both GPTBot and OAI-SearchBot, it may use the results of one crawl for both purposes. If you block GPTBot and allow OAI-SearchBot, the search crawler still fetches your pages, but you have signalled that the content should not be used for training. The controls are independent by design, and allowing AI search while blocking training has a copy-paste robots.txt for that split.
Google has no AI-only switch for Search
Google's AI Overviews and AI Mode are part of Google Search and controlled by your Googlebot rules. Google-Extended covers Gemini training and grounding, not AI Overviews. The only robots.txt way out of AI Overviews is out of Google Search entirely. Page-level controls such as nosnippet limit what can be shown; Google-Extended explained covers the options.
robots.txt is a request, not a lock
robots.txt asks compliant crawlers to stay away. It does not stop a scraper that ignores it, and some user-triggered fetchers say it may not apply to them. Content that must not be read by any automated client belongs behind authentication. A firewall block enforces a decision but needs care: user-agent strings can be spoofed, and a broad bot rule can catch the search crawlers you meant to allow. Verifying real AI crawlers shows how to tell the difference.
Which choices can you undo later?
Reversibility is a good reason to start with the least drastic option.
| Choice | How to reverse it | How quickly it takes effect |
|---|---|---|
| robots.txt Disallow for a search crawler | Remove the rule | OpenAI says about 24 hours for its search systems; other operators do not publish a figure |
| robots.txt Disallow for a training crawler | Remove the rule | Applies to future crawls only |
| Firewall or CDN block | Delete or change the rule | Immediate for new requests |
| Content behind a login | Make it public | Crawlers need to rediscover and recrawl the pages |
| Allowing training crawlers | Add a Disallow later | Future use only; earlier crawls are not undone |
The asymmetry is in the last row. Blocking training later does not reach back, so if you are unsure and your content is valuable, a training block now is the more cautious choice. Blocking search later is easy to reverse, but every day it is in place you are absent from that tool's answers.
A sensible default to start from
For a business site that wants AI search visibility and no training use, this is a reasonable starting point. Adjust the training lines to your own decision:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Disallow: /account/
Disallow: /checkout/Search crawlers fall into the * group here and are allowed everywhere except the private paths. If you later give a search crawler its own group, repeat the private paths inside it, because a crawler that matches a named group ignores the * group.
How to check what you are telling crawlers now
Many sites block AI crawlers without anyone deciding to: a copied robots.txt template, a security plugin or a CDN default can each do it.
- Read your live robots.txt:
curl -s https://www.example.com/robots.txt. List every group and what it disallows. - Check for blanket rules. A
User-agent: *withDisallow: /blocks every compliant crawler, including search. - Check your CDN and firewall settings. Bot management features can block or challenge AI crawlers independently of robots.txt.
- Check your logs. Look for the tokens above and their status codes. A wall of 403 responses for
OAI-SearchBotmeans something is blocking it, whatever robots.txt says. - Write the decision down. Record which purposes you allow and why, so the next template change does not quietly reverse it.
Frequently asked questions
Will blocking GPTBot remove my site from ChatGPT search?
No. OpenAI documents GPTBot and OAI-SearchBot separately. ChatGPT search depends on OAI-SearchBot, and OpenAI says sites opted out of that crawler are not shown in ChatGPT search answers.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews are part of Search and follow your Googlebot rules.
Can I block AI crawlers on only part of my site?
Yes. robots.txt rules work by path, so you can disallow a training crawler from /premium/ and allow it elsewhere. Paid content should also sit behind a login, because robots.txt does not enforce anything.
Should a small business block AI crawlers?
Usually not the search crawlers, because being found and recommended is the point of the site. The training decision is lower stakes for most small businesses, and either choice leaves AI search visibility unchanged according to the operators.
See what your site currently allows
The free check reads your robots.txt and fetches a page as an automated client, then labels each AI crawler by purpose so you can see whether your site says what you decided. An audit does the same across the site, including firewall and CDN behaviour.
Filed under Strategy and audits. Topics: ai-crawlers, robots-txt, strategy, gptbot, claudebot