AI crawler user agents list: the major bots and what each does

An AI crawler user agent is the name a bot sends in its User-Agent header and the token you write in robots.txt to give it rules. The major ones come from OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft, Common Crawl, Meta, Amazon, DuckDuckGo and Mistral, and each operator runs separate agents for training, search and user-triggered fetches. The table below lists the main agents each operator documents at the time of writing (8 November 2026), with its purpose and where it is documented.

The AI crawler user agents list

The robots.txt token is what you put after User-agent:. Matching is case-insensitive under RFC 9309, so GPTBot and gptbot are the same rule.

"Not stated" means the operator's page does not say either way. It does not mean the agent ignores robots.txt.

Training, search and user-triggered: what the purpose column means

The purpose decides what a block costs you. We cover the reasoning in depth in training, search and user-triggered crawlers; here is the short version.

Training crawlers collect pages that may be used to build future models. Disallowing GPTBot, ClaudeBot, CCBot or MistralAI-Training is an opt-out signal for training. Meta-ExternalAgent is broader: Meta uses it for AI training and for indexing content to improve its products. It does not stop an assistant from finding you in search. OpenAI says its GPTBot and OAI-SearchBot settings are independent, and its own example allows OAI-SearchBot while disallowing GPTBot. The full setup is in how to allow AI search but block AI training.

Search crawlers build the index an assistant searches when it answers. These are the agents to keep open if you want to be eligible to show up:

• OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links.
• Anthropic says disabling Claude-SearchBot may reduce a site's visibility in user search results.
• Perplexity advises allowing PerplexityBot if you want to be in its results.
• Google treats AI Overviews and AI Mode as part of Search, so they are governed by Googlebot rules and by the noindex, nosnippet, data-nosnippet and max-snippet controls (AI features and your website).

User-triggered fetchers visit a page because a person asked a question that needs it. They are not crawling the web on their own, and that changes how they treat robots.txt:

• OpenAI says that because ChatGPT-User's actions are initiated by a user, "robots.txt rules may not apply."
• Perplexity says Perplexity-User generally ignores robots.txt.
• Meta says Meta-ExternalFetcher may bypass robots.txt, and Amazon says Amzn-User may not follow all robots.txt directives.
• Anthropic is the exception here: it says all three of its bots, Claude-User included, honor robots.txt, and that disabling Claude-User may reduce visibility in user-directed web search.

Tokens that are not crawlers: Google-Extended and Applebot-Extended

Two entries in the table never send a request. You will not find them in your logs, because they exist only as robots.txt tokens.

• Google-Extended lets you say whether content Google crawls may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. Google states it does not affect inclusion in Google Search and is not a ranking signal. It does not control AI Overviews either; that is Googlebot's job. More in Google-Extended explained.
• Applebot-Extended decides whether data Applebot crawled may be used to train Apple's foundation models. Apple says pages that disallow it can still show in Apple search results and that the token is not considered in ranking.

One Apple detail catches people out: if your robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. A Disallow aimed at Googlebot can block Applebot by accident. Amazon describes a similar fallback: if robots.txt does not mention Amzn-SearchBot but allows other search bots, Amzn-SearchBot follows the rules given to those bots.

How to use these user agents in robots.txt and server logs

A named group replaces the * group for that bot. Under RFC 9309 a crawler obeys the group that matches its token and ignores *, and Bing documents the same behaviour for bingbot. So repeat any general rules inside each named group. A setup that keeps search open and opts out of the main training tokens looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /admin/

OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and bingbot are not named, so they follow the * group and stay out of /admin/ only. Changes are not instant: OpenAI says its search systems can take about 24 hours to adjust, Perplexity says changes may take up to 24 hours, Meta caches robots.txt for up to 24 hours, Amazonbot may use a copy up to 30 days old, and DuckDuckGo says a DuckAssistBot opt-out takes about 72 hours. The robots.txt guide for AI crawlers covers groups, paths and wildcards.

To see which of these agents actually visit you, count them in your access log. Match on the token, not the full string, because version numbers change: OpenAI's current examples show GPTBot/1.4 and OAI-SearchBot/1.4, while older posts quote /1.0 or /1.1.

for bot in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User \
  PerplexityBot Perplexity-User Googlebot bingbot Applebot CCBot \
  meta-externalagent meta-webindexer meta-externalfetcher Amazonbot \
  Amzn-SearchBot Amzn-User DuckAssistBot MistralAI; do
  printf '%-22s %s\n' "$bot" "$(grep -ci "$bot" /var/log/nginx/access.log)"
done

grep -c counts matching lines, and -i ignores case. A line counts if the token appears anywhere in it, including the requested URL or the referrer. Googlebot also matches variants such as Googlebot-Image, MistralAI covers all three Mistral agents, and anyone can send these strings, so treat the counts as claims until you verify them. OpenAI may add a robots.txt; marker to the GPTBot and OAI-SearchBot user agent when they fetch robots.txt, which lets you tell those requests apart. For a fuller walkthrough, see finding AI crawler visits in your server logs.

How to verify that a request really comes from that bot

A user agent string is easy to fake. Most operators publish a way to check the source IP, either a JSON list of IP ranges or a reverse DNS domain:

Reverse DNS verification is two lookups. Look up the IP, check the hostname's domain, then look up that hostname and confirm it returns the same IP:

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

The first command should return a pointer to crawl-66-249-66-1.googlebot.com, and the second should return 66.249.66.1 again. If either step fails or the domain is wrong, the request is not from Google.

The JSON lists change often (OpenAI's chatgpt-user.json was regenerated on 6 November 2026), so fetch them on a schedule instead of copying ranges into a firewall by hand. The full method, including scripts, is in how to verify a real AI crawler.

Agents missing from this list, and outdated tokens

Bytespider. Many robots.txt templates include Bytespider, which third parties attribute to ByteDance. We could not find official ByteDance documentation for it, so its purpose and robots.txt behaviour are unconfirmed and it is not in the table. Third-party reports say it ignores robots.txt, so if you want it gone, a server or WAF rule is more dependable than a Disallow line.

Older Anthropic tokens. anthropic-ai and Claude-Web still turn up in copied templates. Anthropic's current help article names only ClaudeBot, Claude-SearchBot and Claude-User. Anthropic-hosted browsers that Claude drives for users are a separate identity: they send a normal browser user agent plus a Claude-Agent token.

Copilot. Microsoft documents that Microsoft Copilot grounds answers in web information through the Bing search service. Microsoft does not document a separate Copilot crawler token, so bingbot's access to your pages is the part you control.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot only signals whether content may be used to train OpenAI's models. Eligibility for ChatGPT search is controlled by OAI-SearchBot, and OpenAI also recommends allowing requests from its published IP ranges.

Will I see Google-Extended or Applebot-Extended in my server logs?

No. Both are robots.txt tokens that never make requests. Google crawls with its existing user agents and Apple with Applebot; the tokens only control how the crawled content may be used.

Can robots.txt block ChatGPT-User or Perplexity-User?

Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person requested the page. Anthropic, by contrast, says Claude-User honors robots.txt.

Should I match the full user agent string or just the token?

Match the token. Operators change version numbers (OpenAI's docs note that versions may change), and some strings now start with a full Chrome prefix. Then confirm the IP against the operator's published list.

Is anthropic-ai still a valid Anthropic crawler?

Anthropic's current documentation names only ClaudeBot, Claude-SearchBot and Claude-User. Keeping an old token in robots.txt does no harm, but it is not a substitute for a ClaudeBot rule.

Check how your site treats each crawler

A robots.txt that looks right can still block the agents you care about, through a named group you forgot, a Googlebot rule Applebot inherits, or a CDN rule that challenges the published IPs. As a next step, run the free Crawlito check on one of your pages. It checks a single page and needs no account.