ClaudeBot, Claude-SearchBot and Claude-User explained

ClaudeBot is Anthropic's training crawler: it collects public web content that may be used to train Claude models. Anthropic runs two other agents with different jobs. Claude-SearchBot crawls to improve Claude's search results, and Claude-User fetches a page when someone asks Claude a question that needs it. Each has its own robots.txt token, so you can opt out of training and still stay reachable for Claude's search and answers.

This post explains what each bot does according to Anthropic's support article (read on 2026-11-08), gives a robots.txt you can copy, and shows how to check that the rules and your firewall agree.

What each Anthropic crawler does

Anthropic's article names three bots and says they respect "do not crawl" signals by honoring industry-standard robots.txt directives.

The split is the same one OpenAI uses for GPTBot, OAI-SearchBot and ChatGPT-User: one agent for training, one for search, one for live fetches triggered by a person. If the three categories are new to you, training, search and user-triggered crawlers explains why the difference matters.

Two points are easy to get wrong:

• Blocking ClaudeBot is a training opt-out, not a search opt-out. Nothing in Anthropic's article ties ClaudeBot to search visibility.
• Claude-User is not a free pass. Some operators say their user-triggered fetchers may ignore robots.txt. Perplexity says Perplexity-User generally does, and OpenAI says robots.txt rules "may not apply" to ChatGPT-User. Anthropic's article says all three of its bots honor robots.txt, so by Anthropic's account a Disallow for Claude-User is honored, and it costs you the visibility that comes with it.

What about anthropic-ai and Claude-Web?

Many robots.txt templates still list anthropic-ai and Claude-Web. Anthropic's current article names only ClaudeBot, Claude-SearchBot and Claude-User. Leaving the old lines in does no harm, but a file that names only the old tokens does not address the three current bots. Check yours.

Claude-Agent is a separate identity

When Claude drives an Anthropic-hosted browser for a user, the traffic sends that browser's normal user agent plus a Claude-Agent token. Its IPs are published under the Claude-Agent entry at https://claude.com/.well-known/sail, not in the crawler list, according to Anthropic's article on Claude-Agent. That article does not say whether Claude-Agent follows robots.txt, so do not assume the rules in this post govern it.

How to block ClaudeBot but stay in Claude search

Anthropic's own example for a full ClaudeBot block is two lines in the robots.txt at the top level of your site:

User-agent: ClaudeBot
Disallow: /

To opt out of training while staying reachable for search and user fetches, name the other two bots as well. A complete file for a typical site looks like this:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /account/
Disallow: /cart/

User-agent: *
Disallow: /account/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

The first group opts ClaudeBot out of training. The second lets Claude-SearchBot and Claude-User in, apart from private paths. The * group covers every other crawler.

Why the private paths appear twice: under RFC 9309, a crawler obeys the group that matches its name and only falls back to * when no group matches. A Claude-SearchBot group with no Disallow lines would let it into /account/ even though your * group blocks it. Listing two User-agent lines above one set of rules, as in the second group, is valid and keeps the file short. How robots.txt groups work covers this in detail.

Three more rules from Anthropic's article and the standard:

1. Every subdomain needs its own file. Anthropic says to add the rules for each subdomain you want to opt out. blog.example.com/robots.txt is a separate file from www.example.com/robots.txt.
2. Matching is case-insensitive for the token but case-sensitive for paths, per RFC 9309. claudebot matches ClaudeBot; /Account/ does not match /account/.
3. Return a real 200. Under RFC 9309, a 4xx on robots.txt means "no restrictions" and a 5xx means "assume full disallow". A robots.txt that errors out can block Claude-SearchBot by accident or lift your ClaudeBot block.

For the same pattern across OpenAI, Google, Apple and Perplexity, see how to allow AI search but block AI training.

The new "no AI training" signal

As of October 30, 2026, Anthropic says it recognizes a machine-readable "no AI training" preference based on an IETF draft from the aipref working group (draft-kelley-aipref-training-preference-signal). This is early: the draft is not a final standard, Anthropic says it is rolling support out over time, and its handling may change. One rule is clear today: if your robots.txt names ClaudeBot, that rule wins over the general signal for the paths it covers. If you want a firm answer for Anthropic, write the ClaudeBot group.

Does ClaudeBot support Crawl-delay?

Yes. Anthropic says it supports the non-standard Crawl-delay extension and gives this example:

User-agent: ClaudeBot
Crawl-delay: 1

That example names ClaudeBot only. The article describes Crawl-delay support in general terms and does not give separate examples for Claude-SearchBot or Claude-User.

Two cautions before you rely on it:

• Adding a group changes the rules ClaudeBot obeys. If you add the group above and nothing else, ClaudeBot now ignores your * group, including its Disallow lines. Repeat them:

User-agent: ClaudeBot
Crawl-delay: 1
Disallow: /account/
Disallow: /cart/

• Crawl-delay is not universal. RFC 9309 does not define it. Google ignores it, Applebot and Amazonbot say they do not follow it, while Bingbot and CCBot honor it. A Crawl-delay line tuned for Anthropic does nothing for Googlebot.

If crawl load is the real problem, Crawl-delay is gentler than a block, because blocking Claude-SearchBot or Claude-User may also reduce your visibility in Claude's search. Crawl-delay and rate limits for AI crawlers compares the options, including server-side rate limits.

Why IP blocking is the wrong opt-out

Anthropic recommends robots.txt over IP blocking and warns that blocking its IPs may not work correctly or keep you opted out over time. It also says an IP block impedes its ability to read your robots.txt, so the crawler cannot learn your preference from it.

IP lists are still useful for the opposite job: proving a request really is Anthropic. Anthropic publishes its crawler source IPs at claude.com/crawling/bots.json and says a crawler from an IP on that list is coming from Anthropic. When read on 2026-11-08, the file had a creationTime of 2026-10-20 and a prefixes array of ipv4Prefix entries: one /22 block and a set of single-host /32 addresses. It does not say which bot uses which prefix, and it changes, so fetch it fresh instead of copying the ranges into a config file.

This checks one IP from your logs against the current list:

curl -s https://claude.com/crawling/bots.json -o anthropic-bots.json

python3 - 216.73.216.12 <<'EOF'
import ipaddress, json, sys
ip = ipaddress.ip_address(sys.argv[1])
data = json.load(open("anthropic-bots.json"))
nets = [ipaddress.ip_network(p["ipv4Prefix"]) for p in data["prefixes"] if "ipv4Prefix" in p]
print("on Anthropic's list" if any(ip in n for n in nets) else "NOT on Anthropic's list")
EOF

Anyone can put "ClaudeBot" in a user agent. A request that claims the name but comes from an IP outside the list is not Anthropic, whatever it says. How to verify a real AI crawler covers the same check for other operators.

How to check what Anthropic's bots actually get

A correct robots.txt is only half the job. Firewalls, CDNs and bot filters often block by user agent before robots.txt is ever read.

1. Find the bots in your logs

grep -E "ClaudeBot|Claude-SearchBot|Claude-User" /var/log/nginx/access.log \
  | awk '{print $9, $7}' | sort | uniq -c | sort -rn | head -20

With the default nginx combined log format, $9 is the status code and $7 the path. Lots of 403 or 503 responses mean something at the edge is refusing them. Match on the token, not a full user-agent string: Anthropic's article does not publish full strings, so do not filter on one copied from a blog post.

2. Request a page with the token

curl -s -o /dev/null -w "%{http_code}\n" -A "Claude-SearchBot" https://www.example.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://www.example.com/

If the first returns 403 and the second 200, a rule is blocking on the user agent. This does not prove what the real bot sees, since rules can also key on IP ranges, but it catches the common case.

3. Check your platform's AI toggles

Several platforms write Anthropic rules for you, and not always the ones you expect:

• Cloudflare: with Bot Preference Sync on, the managed robots.txt adds Disallow: / for ClaudeBot (not Claude-SearchBot or Claude-User) once any AI policy is set to Block, Block on pages with ads or Disallow (Training), according to Cloudflare's docs. The separate AI Crawler and Agent access policies set Search, Agent and Training actions at the edge. See Cloudflare and AI crawlers.
• Squarespace: the "Block known artificial intelligence crawlers" checkbox, off by default, lists ClaudeBot and anthropic-ai but not Claude-SearchBot or Claude-User, per Squarespace's help page.
• Webflow: the AI bots group under Traffic control includes both ClaudeBot and Claude-User, per Webflow's help page. Turning it off blocks the user-triggered fetcher too, not just training.

4. Make sure the content is in the HTML

Reaching the page is not the same as reading it. In Vercel and MERJ's December 2024 analysis, ClaudeBot downloaded JavaScript files (23.84% of Claude's requests) but did not execute them, so content rendered only in the browser was invisible to it. That measurement covers ClaudeBot, not Claude-SearchBot or Claude-User, so treat it as a snapshot, but the safe move is the same: keep your main text in the server response. To check, fetch the page with curl and search the raw HTML for a sentence from your main content.

Frequently asked questions

Does blocking ClaudeBot remove my site from Claude?

Not according to Anthropic's documentation. Anthropic describes a ClaudeBot block as a signal to exclude your future materials from training. Search visibility is tied to Claude-SearchBot and Claude-User, and Anthropic says blocking those may reduce it.

Does Claude-User follow robots.txt?

Anthropic's article says all three of its bots honor robots.txt, Claude-User included. That differs from Perplexity-User, which Perplexity says generally ignores robots.txt, and ChatGPT-User, where OpenAI says robots.txt rules may not apply.

What is the ClaudeBot user agent string?

Anthropic's support article gives the robots.txt tokens (ClaudeBot, Claude-SearchBot, Claude-User) but not full user-agent strings. Match on the token in logs and firewall rules, and verify the source IP against claude.com/crawling/bots.json.

Should I keep anthropic-ai and Claude-Web in robots.txt?

They are not in Anthropic's current documentation. Keeping them is harmless, but make sure the file also names ClaudeBot, and Claude-SearchBot and Claude-User if you have rules for them.

How do I report a crawler that ignores my rules?

Anthropic's article gives a contact address for questions and for crawlers that are not working as described, and asks that your email include the domain you are asking about. First confirm the IP is on Anthropic's published list, since a fake ClaudeBot is more likely than a real one ignoring robots.txt.

Next step

If you are not sure what your robots.txt, CDN and page HTML say to each of Anthropic's three bots, run the free single-page check on a key page before you change any rules. It needs no account.