AI crawler user agents list: the major bots and what each does
A reference list of AI crawler user agents from OpenAI, Anthropic, Perplexity, Google, Apple and others, with each bot's purpose, robots.txt token and docs.
8 min read
11 guides
AI crawlers
Every AI company runs more than one crawler, and each has a different job: collecting training data, building a search index, or fetching a page because a person asked. These guides name them, explain their purpose and show how to recognize real visits.
AI crawlers in server logs: how to find and count their visits
Find AI crawlers in server logs with grep and awk: count hits per bot, spot 403 and 429 responses, verify IPs, and read Cloudflare's AI crawler data.
9 min read
AI training crawlers vs search crawlers: why the difference matters
AI training crawlers vs search crawlers vs user-triggered fetchers: what each does, what blocking costs, and how to opt out of training but stay in AI search.
9 min read
Applebot-Extended and Applebot: Apple's crawlers and AI
Applebot-Extended does not crawl: it is a robots.txt token that opts content out of Apple's model training. How it differs from Applebot, with examples.
8 min read
Bingbot and AI search: why Bing's crawler still matters
Microsoft Copilot grounds web answers in Bing search. How to check that Bingbot can reach your site, verify real Bingbot traffic and use IndexNow.
9 min read
CCBot and Common Crawl: the open dataset behind many models
What CCBot is, why Common Crawl's open web archive feeds so many AI datasets, how CCBot follows robots.txt, and what blocking it does and does not change.
9 min read
ClaudeBot, Claude-SearchBot and Claude-User explained
ClaudeBot collects training data, Claude-SearchBot indexes for search and Claude-User fetches on request. How to opt out of training and stay visible in Claude.
8 min read
Google-Extended: what it controls and what it doesn't
Google-Extended is a robots.txt token, not a crawler. It covers Gemini training and grounding, not Google Search rankings or AI Overviews. Here is how to use it.
8 min read
GPTBot vs OAI-SearchBot vs ChatGPT-User: OpenAI's crawlers explained
GPTBot vs OAI-SearchBot vs ChatGPT-User: what each OpenAI agent does, how robots.txt applies to each, and how to verify them by IP.
7 min read
How to verify GPTBot IPs and spot fake AI crawlers
User agents are easy to fake. Verify GPTBot IPs and other AI crawlers with published IP lists, reverse DNS or Cloudflare verified bots, then deal with fakes.
9 min read
PerplexityBot and Perplexity-User: how Perplexity reaches your site
PerplexityBot indexes pages for Perplexity search; Perplexity-User fetches pages on request. How to allow, block and verify each, and what blocking changes.
8 min read