How to test robots.txt against each AI crawler

To test robots.txt against AI crawlers, do three things in order: fetch the live file with curl and confirm it returns 200 as text/plain on every host, work out which group each AI token (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and the rest) actually follows, then run each token and a few real URLs through a standards-based parser. Google's Search Console only shows you what Googlebot saw, so it cannot answer the question for any other crawler.

For background on what the rules mean, start with robots.txt for AI crawlers.

Step 1: Fetch robots.txt raw with curl

The file in your repository, CMS settings or SEO plugin is not what crawlers read. They read whatever your server, CDN and firewall return at /robots.txt. A CDN can prepend its own rules (Cloudflare's managed robots.txt adds a block before your origin's file), a framework can serve an HTML fallback page, and a firewall can answer with a challenge. So test the live URL.

Start with the status code, content type and redirect chain in one line:

curl -sS -L -o /dev/null --max-time 10 \
  -w "%{http_code} | %{content_type} | %{num_redirects} redirects | %{url_effective}\n" \
  https://example.com/robots.txt

A healthy result looks like 200 | text/plain; charset=utf-8 | 1 redirects | https://www.example.com/robots.txt. Then read every hop and the body itself:

curl -sS -I -L https://example.com/robots.txt   # headers for each hop
curl -sS https://www.example.com/robots.txt     # the rules crawlers will parse
curl -sS https://www.example.com/robots.txt | wc -c   # size in bytes

Google ignores anything past 500 KiB, according to its robots.txt specification, so a generated file that has grown huge can lose rules at the end.

What each response means

The last row is the one that hides sites. A robots.txt that times out or returns 503 during a deploy reads, to a compliant crawler, as Disallow: / for everything. Our guide to status codes and redirects covers the same rules for ordinary pages.

Repeat the fetch with a crawler's user agent

Some firewalls treat bot user agents differently, so fetch the file again as an AI crawler would identify itself:

curl -sS -I \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://www.example.com/robots.txt

If this returns 403, or a cf-mitigated: challenge header (Cloudflare adds it to every challenge page), the crawler may never get to your rules. This only approximates the real request, because real crawlers also come from their operators' published IP ranges and many firewall rules look at the IP. A pass with curl is a good sign, not proof. See why a WAF challenge page means AI crawlers read nothing for what to do when it fails.

Step 2: Check every host and protocol

Google's robots.txt introduction is explicit: the file must sit in the top-level directory, and its rules apply only to the host, protocol and port where it is served. https://www.example.com/robots.txt says nothing about shop.example.com, docs.example.com or blog.example.com. Anthropic's instructions for blocking ClaudeBot make the same point: you repeat the rule for every subdomain you want to opt out.

List every hostname that serves pages you care about, then check them all in one pass:

for host in example.com www.example.com shop.example.com docs.example.com; do
  for scheme in https http; do
    url="$scheme://$host/robots.txt"
    printf "%-42s " "$url"
    curl -sS -L -o /dev/null --max-time 10 \
      -w "%{http_code} | %{content_type} | %{url_effective}\n" "$url"
  done
done

A redirect from http to https, or from the bare domain to www, is normal. What you are looking for is a host that ends on a 4xx, a 5xx, an HTML page or a different file than you expected. Help centres, status pages and shops on separate services each ship their own robots.txt.

Step 3: Walk through group selection for each AI token

Reading robots.txt from top to bottom is how most people misread it. Crawlers do not apply every rule that seems relevant. RFC 9309 sets out how a crawler picks its rules:

1. It looks for groups whose User-agent line matches its product token, case-insensitively.
2. If several groups match, it combines their rules into one group.
3. If no group matches, it obeys the User-agent: * group.
4. If there is no * group either, no rules apply.
5. Within the chosen group, the rule with the longest matching path wins, whatever its position in the file. If an Allow and a Disallow match with equal length, the Allow should win.

The consequence: once a crawler has its own group, the * group stops applying to it entirely. Bing says so for Bingbot, and it is standard behaviour. The robots.txt groups guide covers this in depth.

A worked example

Take this file:

User-agent: *
Disallow: /admin/
Disallow: /search

User-agent: GPTBot
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Googlebot
Disallow: /private/

Here is what each token is allowed to fetch for two URLs:

Two rows tend to surprise people. OAI-SearchBot can now crawl /admin/, because naming it in a group wiped out the general rules for it. And Applebot follows the Googlebot group, not *: Apple's Applebot documentation says that when robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions.

Longest match matters inside a group too:

User-agent: PerplexityBot
Disallow: /docs/
Allow: /docs/public/

/docs/public/guide is allowed because /docs/public/ is the longer match. /docs/internal is blocked. Swapping the order of the two lines changes nothing.

The AI crawler tokens to test

Not every AI agent is controlled by robots.txt, and two of the most-quoted tokens are not crawlers at all. Test the tokens in this table, but read the last column before you rely on the result. For full user-agent strings, see the AI crawler user agents list.

For user-triggered agents that do not follow robots.txt, a test result tells you about your intent, not about what will happen. Blocking them is a server or firewall decision.

Step 4: Run Google's open-source parser

For a long file, or a check you want to repeat, use a parser. Google publishes the robots.txt parser and matcher that Googlebot uses in production as an open-source C++ library, and it includes a small binary that tests one user agent and one URL against a local file.

Build it once with Bazel:

git clone https://github.com/google/robotstxt.git
cd robotstxt
bazel build :robots_main

Then download your live file and loop over the AI tokens:

curl -sS https://www.example.com/robots.txt -o /tmp/robots.txt

for ua in OAI-SearchBot GPTBot Claude-SearchBot ClaudeBot Claude-User \
          PerplexityBot Googlebot Google-Extended Bingbot Applebot \
          Applebot-Extended CCBot Meta-ExternalAgent Amazonbot DuckAssistBot; do
  ./bazel-bin/robots_main /tmp/robots.txt "$ua" "https://www.example.com/blog/post"
done

Each line prints a verdict such as user-agent 'GPTBot' with URI 'https://www.example.com/blog/post': DISALLOWED, and the exit code is 0 for allowed and 1 for disallowed, so it slots into CI. Run it for a handful of real URLs: your home page, a key product or article page, and anything under a path you have rules for.

Two limits. The README says the library and binary do not handle logic a crawler applies outside parsing and matching (its example is Googlebot-Image falling back to Googlebot's rules), so the tool will not reproduce Applebot falling back to Googlebot's group either; check that by hand. And it reflects Google's reading of the standard: other operators' parsers can differ at the edges. If a result surprises you, simplify the rule until there is only one way to read it.

Step 5: Check Google's view in Search Console

Search Console covers Google only, but it is where Google shows you exactly what it fetched. The robots.txt report lists the robots.txt files Google found for the top 20 hosts on your site, when each was last crawled, and any warnings or errors. After fixing an error you can request a recrawl of the file from the same report.

Some details save time:

• The report is available only for Domain properties or host-level URL-prefix properties, not URL-prefix properties with a path.
• The old robots.txt Tester, which let you type a URL and see a verdict, was retired in late 2023, when this report launched. To check a single URL, use URL Inspection, or the parser from step 4.
• If you use Cloudflare's managed robots.txt, Search Console may flag its Content-signal lines as "Syntax not understood". Cloudflare says it has seen no impact on crawling or SEO.

Nothing in Search Console tells you how GPTBot, ClaudeBot or PerplexityBot read the file. For those, your evidence is the parser result plus your server logs.

After you change robots.txt

Crawlers cache robots.txt, so a fix is not visible straight away. At the time of writing, OpenAI says its search systems take about 24 hours to adjust, Perplexity says up to 24 hours, Google generally caches for up to 24 hours, Bing says at least a few hours, DuckDuckGo says about 72 hours for DuckAssistBot, and Amazon says Amazonbot may use a copy cached within the last 30 days. Re-run steps 1 and 4 straight after the deploy to confirm the live file changed, then give the crawlers time.

Also remember that an Allow is permission, not an invitation. If the rules pass but a crawler still never shows up, the block is usually somewhere else: a firewall, a bot-protection setting, or a page that returns little readable HTML. Our list of robots.txt mistakes that block AI crawlers covers the common patterns.

Frequently asked questions

Can I test robots.txt for GPTBot in Google Search Console?

No. The robots.txt report shows the files Google fetched, their status and any errors, and URL Inspection tests Googlebot only. To test GPTBot or any other AI token, run the file through a parser such as Google's open-source robotstxt library with that token as the user agent.

Does a missing robots.txt block AI crawlers?

No. Under RFC 9309, a 404 or any other 4xx response means no restrictions apply, so compliant crawlers may fetch everything. The risky case is the opposite one: a 5xx or timeout tells crawlers to assume the whole site is disallowed.

Is setting GPTBot's user agent in curl enough to test as GPTBot?

It tells you how your server and CDN respond to that user-agent string, which catches many firewall blocks. It cannot reproduce rules based on IP address, so a real crawler request from the operator's published ranges may still be treated differently.

Check every AI crawler at once

The manual steps above work, but they take a while to repeat across hosts and tokens. Crawlito's free check fetches your live robots.txt and evaluates the rule for each named AI crawler on the page you enter, alongside the status code, CDN or WAF challenge pages and meta robots, with no account needed. If you need the same evidence across a whole site, the full audit covers up to 500 pages.