Skip to content

Should you block AI crawlers? A decision guide by site type

Should you block GPTBot, ClaudeBot and other AI crawlers? Decide by crawler purpose and site type, with the trade-offs and which choices you can undo later.

By Crawlito · 7 min read

In short

  • Decide per crawler purpose, not per company: training crawlers, search crawlers and user-triggered fetchers have separate robots.txt tokens and different costs when blocked.
  • OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links, so blocking a search crawler has a direct visibility cost.
  • Blocking a training crawler is a business decision about future use: Anthropic says disallowing ClaudeBot signals that future materials should be excluded from training, and no operator describes robots.txt as removing data already collected.
  • robots.txt is a request: Perplexity says Perplexity-User generally ignores it and OpenAI says it may not apply to ChatGPT-User, so anything you must enforce needs a login or a firewall rule.
  • robots.txt changes are the easiest to reverse; OpenAI says its search systems adjust about 24 hours after a robots.txt update.
On this page

For most sites, the answer is: do not block AI search crawlers, decide deliberately about training crawlers, and leave user-triggered fetchers alone. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot are how AI assistants find pages to cite, so blocking them removes you from those answers. Training crawlers such as GPTBot and ClaudeBot are a separate choice about whether your future content may be used to train models, and the right answer depends on what your content is worth to you.

This guide works through that decision by crawler purpose and by type of site, with the trade-offs and the parts that are easy or hard to undo. It assumes you know the basic split; if not, start with training, search and user-triggered crawlers.

Decide by crawler purpose first

Each major AI company runs several agents, and blocking "OpenAI" or "Anthropic" as a whole is rarely what anyone wants. Here is the cost of blocking each kind, based on the operators' own documentation at the time of writing.

PurposeExamplesWhat blocking costs youTypical default
Search and indexingOAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, BingbotVisibility in that tool's answers, or in search altogetherAllow
Model trainingGPTBot, ClaudeBot, CCBot, plus the Google-Extended and Applebot-Extended tokensNothing in AI search, per the operators; your future content stays out of that operator's training dataYour call
User-triggered fetchesChatGPT-User, Claude-User, Perplexity-UserThe assistant cannot read your page when a person asks about itAllow

The operators' statements behind that table:

  • OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. It describes GPTBot as used to make its foundation models more useful and safe, and says that because ChatGPT-User actions are started by a user, robots.txt rules may not apply.
  • Anthropic says disabling ClaudeBot signals that your future materials should be excluded from training, while disabling Claude-SearchBot or Claude-User may reduce your visibility in search and user-directed answers.
  • Perplexity says PerplexityBot is not used to crawl content for AI foundation models and that Perplexity-User generally ignores robots.txt because a user requested the fetch.
  • Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal there.

Decide by site type

The search question has the same answer for nearly everyone. The training question is where site types differ. These are reasonable starting points, not rules.

Site typeAI search crawlersTraining crawlersWhy
Local business, servicesAllowAllow or block; low stakesYou want to be recommended; your pages are marketing, not the product
SaaS and B2BAllowUsually allowProduct and pricing pages exist to be read and explained accurately
EcommerceAllowUsually allowProduct facts help assistants answer shopping questions; catalogue text is rarely the asset
News and magazine publishersAllow, unless you license search accessOften blockThe writing is the product; many publishers negotiate licences instead
Paid courses, research, dataAllow public pagesOften blockKeep paid content behind a login regardless of robots.txt
Documentation and developer toolsAllowUsually allowDevelopers ask assistants about your API; accurate answers reduce support load
Forums and user-generated contentAllowCheck your termsYou may not have rights to license users' posts for training
Agencies hosting client contentAllowFollow each client's contractThe decision belongs to the content owner

Two situations override the table:

  • Contracts. If a licence, client agreement or syndication deal restricts reuse, the training block follows the contract, not a blog post.
  • Load. If a crawler is hurting your servers, that is an operational problem with an operational fix. A rate limit for that crawler is better than a block that also removes you from its answers; see rate limiting AI crawlers.

Trade-offs most decision guides skip

Blocking training does not remove what was already collected

Anthropic says disallowing ClaudeBot signals that your future materials should be excluded from training, and neither OpenAI nor Anthropic describes a robots.txt rule as removing content already collected. Treat a Disallow added today as a signal about future crawls, not a deletion request. If removal matters to you, that is a conversation with the operator, not a robots.txt line.

Blocking training is not a search opt-out

OpenAI says that if you allow both GPTBot and OAI-SearchBot, it may use the results of one crawl for both purposes. If you block GPTBot and allow OAI-SearchBot, the search crawler still fetches your pages, but you have signalled that the content should not be used for training. The controls are independent by design, and allowing AI search while blocking training has a copy-paste robots.txt for that split.

Google's AI Overviews and AI Mode are part of Google Search and controlled by your Googlebot rules. Google-Extended covers Gemini training and grounding, not AI Overviews. The only robots.txt way out of AI Overviews is out of Google Search entirely. Page-level controls such as nosnippet limit what can be shown; Google-Extended explained covers the options.

robots.txt is a request, not a lock

robots.txt asks compliant crawlers to stay away. It does not stop a scraper that ignores it, and some user-triggered fetchers say it may not apply to them. Content that must not be read by any automated client belongs behind authentication. A firewall block enforces a decision but needs care: user-agent strings can be spoofed, and a broad bot rule can catch the search crawlers you meant to allow. Verifying real AI crawlers shows how to tell the difference.

Which choices can you undo later?

Reversibility is a good reason to start with the least drastic option.

ChoiceHow to reverse itHow quickly it takes effect
robots.txt Disallow for a search crawlerRemove the ruleOpenAI says about 24 hours for its search systems; other operators do not publish a figure
robots.txt Disallow for a training crawlerRemove the ruleApplies to future crawls only
Firewall or CDN blockDelete or change the ruleImmediate for new requests
Content behind a loginMake it publicCrawlers need to rediscover and recrawl the pages
Allowing training crawlersAdd a Disallow laterFuture use only; earlier crawls are not undone

The asymmetry is in the last row. Blocking training later does not reach back, so if you are unsure and your content is valuable, a training block now is the more cautious choice. Blocking search later is easy to reverse, but every day it is in place you are absent from that tool's answers.

A sensible default to start from

For a business site that wants AI search visibility and no training use, this is a reasonable starting point. Adjust the training lines to your own decision:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /account/
Disallow: /checkout/

Search crawlers fall into the * group here and are allowed everywhere except the private paths. If you later give a search crawler its own group, repeat the private paths inside it, because a crawler that matches a named group ignores the * group.

How to check what you are telling crawlers now

Many sites block AI crawlers without anyone deciding to: a copied robots.txt template, a security plugin or a CDN default can each do it.

  1. Read your live robots.txt: curl -s https://www.example.com/robots.txt. List every group and what it disallows.
  2. Check for blanket rules. A User-agent: * with Disallow: / blocks every compliant crawler, including search.
  3. Check your CDN and firewall settings. Bot management features can block or challenge AI crawlers independently of robots.txt.
  4. Check your logs. Look for the tokens above and their status codes. A wall of 403 responses for OAI-SearchBot means something is blocking it, whatever robots.txt says.
  5. Write the decision down. Record which purposes you allow and why, so the next template change does not quietly reverse it.

Frequently asked questions

No. OpenAI documents GPTBot and OAI-SearchBot separately. ChatGPT search depends on OAI-SearchBot, and OpenAI says sites opted out of that crawler are not shown in ChatGPT search answers.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews are part of Search and follow your Googlebot rules.

Can I block AI crawlers on only part of my site?

Yes. robots.txt rules work by path, so you can disallow a training crawler from /premium/ and allow it elsewhere. Paid content should also sit behind a login, because robots.txt does not enforce anything.

Should a small business block AI crawlers?

Usually not the search crawlers, because being found and recommended is the point of the site. The training decision is lower stakes for most small businesses, and either choice leaves AI search visibility unchanged according to the operators.

See what your site currently allows

The free check reads your robots.txt and fetches a page as an automated client, then labels each AI crawler by purpose so you can see whether your site says what you decided. An audit does the same across the site, including firewall and CDN behaviour.

Filed under Strategy and audits. Topics: ai-crawlers, robots-txt, strategy, gptbot, claudebot