See your page as an AI crawler: curl with the GPTBot user agent

To see your page as an AI crawler sees it, request it with curl and the crawler's user-agent string: curl -A "<GPTBot user agent>" https://example.com/page. Add -I or -D - to read the status and headers, -L to follow redirects, and pipe the HTML through a text extractor to see the words a crawler that does not run JavaScript gets. Then compare that text with what your browser shows. If the two differ, or the response is a challenge page, you have found the problem.

This takes a few minutes per page and needs only curl and Python 3. curl ships with macOS, Windows 10 and later, and most Linux distributions; on macOS, python3 comes with the Xcode Command Line Tools. The one caveat, covered at the end, is that a spoofed user agent is not the real bot: some CDNs check the source IP, and your laptop is not on OpenAI's list.

What a curl test shows, and what it misses

curl fetches one URL and prints what the server sent. It does not run scripts, load images or build a DOM. That is a useful limitation here, because most AI crawlers behave the same way. In Vercel's analysis of AI crawler traffic from December 2024, none of the major AI crawlers rendered JavaScript, including OpenAI's crawlers, ClaudeBot and PerplexityBot. ChatGPT's and Claude's crawlers downloaded JavaScript files but did not execute them. That is a snapshot, not a permanent rule: it measured background crawlers, and user-triggered or browser-based agents may behave differently. Whether AI crawlers run JavaScript has its own post.

Two things follow from that table. A curl test is a good model of the HTML a non-rendering crawler receives. It is not a test of robots.txt, so test your robots.txt against each AI crawler as a separate step. Googlebot is different again: Google renders pages with an evergreen Chromium, so curl shows what Googlebot fetches but not what it indexes after rendering.

The AI crawler user agents to test

Use the strings the operators publish. At the time of writing, these are the documented ones. Version numbers change, so match your server rules on the token (GPTBot, OAI-SearchBot) rather than the whole string.

Sources: OpenAI's crawler overview and Perplexity's bots page. Note that OAI-SearchBot's string starts with a full Chrome-on-macOS prefix, so a rule that looks for "Chrome" to identify browsers will match it too.

Anthropic's crawler article names ClaudeBot, Claude-SearchBot and Claude-User but does not publish their full strings. For a test, send a string that contains the token, such as Mozilla/5.0 (compatible; ClaudeBot/1.0), and treat the result as a check of your token-matching rules, not an exact copy of Anthropic's request. The AI crawler user agents list covers the other operators.

How to check status, redirects and headers

Put the string in a shell variable once, so the commands stay readable:

UA='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot'
URL='https://example.com/pricing'

Read the status code and headers

-I sends a HEAD request and prints only the headers:

curl -sI -A "$UA" "$URL"

HEAD is quick, but some servers and frameworks answer HEAD differently from GET, or reject it with a 405. Crawlers send GET, so the safer version downloads the body, throws it away and prints the headers:

curl -s -o /dev/null -D - -A "$UA" "$URL"

Look for these lines in the output:

Follow the redirect chain

Without -L, curl stops at the first redirect. With it, curl follows the chain, and -w prints a summary of where it ended:

curl -s -L -o /dev/null -A "$UA" \
  -w '%{http_code} after %{num_redirects} redirects: %{url_effective}\n' \
  "http://example.com/pricing"

To see every hop, combine -I and -L and keep only the status and location lines:

curl -sIL -A "$UA" "http://example.com/pricing" | grep -i -E '^(HTTP|location)'

A chain like http to https to www to a trailing slash is three hops for every crawler visit. Watch for loops, and for redirects that send a bot to a login page, a country selector or the homepage instead of the page you asked for.

Spot a challenge or block page

A challenge page usually comes back as an error status with a short HTML body; Cloudflare's managed challenge page, for example, returns a 403. Cloudflare also adds a reliable marker: every Challenge Page response carries the header cf-mitigated: challenge, with text/html as the content type whatever was requested, according to Cloudflare's challenge detection docs.

curl -s -o /dev/null -D - -A "$UA" "$URL" | grep -i '^cf-mitigated'

That header marks challenges only. A WAF block rule or an AI crawler block returns a different response without it, so also read the page title:

curl -s -A "$UA" "$URL" | grep -i -o '<title>[^<]*</title>'

If the title names your product, the crawler got your page. If it is a generic waiting, access denied or verification message, the crawler got nothing it can use. Why a WAF challenge page means AI crawlers read nothing explains the settings that usually cause it.

How to extract the visible text and compare it with a browser

A 200 status with a full-size body can still hold almost no text, for example an empty <div id="root"> that JavaScript fills later. To see the words, strip the tags. Save this as visible_text.py. It uses only Python's standard library and skips script, style and similar elements:

import sys
from html.parser import HTMLParser

SKIP = {"script", "style", "noscript", "template", "svg"}

class Text(HTMLParser):
    def __init__(self):
        super().__init__()
        self.depth = 0
        self.lines = []
    def handle_starttag(self, tag, attrs):
        if tag in SKIP:
            self.depth += 1
    def handle_endtag(self, tag):
        if tag in SKIP and self.depth:
            self.depth -= 1
    def handle_data(self, data):
        if not self.depth and data.strip():
            self.lines.append(" ".join(data.split()))

parser = Text()
parser.feed(sys.stdin.read())
print("\n".join(parser.lines))

Then pipe curl into it:

curl -s -L --compressed -A "$UA" "$URL" | python3 visible_text.py | less
curl -s -L --compressed -A "$UA" "$URL" | python3 visible_text.py | wc -w

If you have lynx or w3m installed, lynx -dump -stdin or w3m -dump -T text/html does the same job. A faster check is to grep for one sentence you know is on the page, such as a price or a product name:

curl -s -L --compressed -A "$UA" "$URL" | grep -c 'Starter plan'

A count of 0 means that sentence is not in the raw HTML.

Now get the browser's version. Headless Chrome prints the DOM after scripts have run:

"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --headless --dump-dom "$URL" \
  | python3 visible_text.py > rendered.txt
curl -s -L --compressed -A "$UA" "$URL" | python3 visible_text.py > raw.txt
wc -w raw.txt rendered.txt
diff raw.txt rendered.txt | less

On Linux the binary is usually google-chrome or chromium. Without a command line, open the page, open DevTools and run copy(document.body.innerText) in the console, then paste it next to the curl output.

If the raw text has a fraction of the rendered word count, the missing words depend on JavaScript and a non-rendering crawler will not see them. Raw HTML vs rendered HTML covers how to read the gap and which fixes close it. One case worth knowing on Next.js App Router sites: streamed Suspense content arrives in the same response but inside a hidden element near the end of the HTML, so it can show up in your extracted text out of position.

Test every agent in one run

This script fetches one URL with each agent and prints the final status, redirect count, bytes, visible word count and whether a Cloudflare challenge was served. Save it as ua-check.sh in the same folder as visible_text.py, make it executable with chmod +x ua-check.sh, and run it as ./ua-check.sh https://example.com/page:

#!/usr/bin/env bash
URL="$1"
while IFS='|' read -r name ua; do
  : > headers.txt; : > body.html
  out=$(curl -s -L --compressed -A "$ua" -D headers.txt -o body.html \
    -w '%{http_code} %{num_redirects} %{size_download}' "$URL")
  words=$(python3 visible_text.py < body.html | wc -w | tr -d ' ')
  challenge=$(grep -i -c '^cf-mitigated: challenge' headers.txt)
  printf '%-15s %s words=%s challenge=%s\n' "$name" "$out" "$words" "$challenge"
done <<'AGENTS'
browser|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
GPTBot|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
OAI-SearchBot|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
ChatGPT-User|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
PerplexityBot|Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Perplexity-User|Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
ClaudeBot-test|Mozilla/5.0 (compatible; ClaudeBot/1.0)
AGENTS

Healthy output has the same status and roughly the same word count on every line. In this illustrative run, one agent is treated differently:

browser         200 0 48211 words=912 challenge=0
GPTBot          200 0 48211 words=912 challenge=0
OAI-SearchBot   200 0 48211 words=912 challenge=0
ChatGPT-User    200 0 48211 words=912 challenge=0
PerplexityBot   403 0 5120 words=14 challenge=1
Perplexity-User 200 0 48211 words=912 challenge=0
ClaudeBot-test  200 0 48211 words=912 challenge=0

A line like the PerplexityBot one says a rule treats that agent differently. The browser line is your baseline: if it also shows a low word count, the problem is rendering, not blocking.

Why a spoofed user agent can get a different answer

Your test sends the right header from the wrong place. Operators publish IP ranges so sites can tell real crawlers from fakes, and the guide to verifying real AI crawlers shows how to use them. That cuts both ways for your test:

• Your test is blocked, the real bot is not. A rule that blocks requests claiming a bot name from outside its published ranges will stop your curl and let the real crawler through. Cloudflare's verified bots are confirmed by Web Bot Auth signatures, published IP lists or reverse DNS, and Perplexity recommends WAF rules that match both the user agent and its published IPs.
• Your test passes, the real bot is blocked. A rule keyed on IP ranges, ASN or country can block the real crawler while your request from home gets a 200. Cloudflare's AI Crawl Control identifies crawlers by user agent only on the Free plan, but higher plans can use Bot Management detection, which a spoofed header does not satisfy in the same way.
• Rate limits and challenges vary. Cloudflare's Bot Fight Mode challenges traffic that matches known bot patterns, and a burst of tests from one IP can trip rate limits that the real crawler, spread across many IPs, may never hit.

So treat a curl test as strong evidence about your HTML and your user-agent rules, and weaker evidence about IP-based rules. To confirm what the real bot gets, look at your server or CDN logs for requests from verified crawler IPs and check their status codes.

Frequently asked questions

How do I set the GPTBot user agent in curl?

Pass the string with -A (or --user-agent): curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://example.com/. -H "User-Agent: ..." does the same thing.

Does a 200 from curl mean GPTBot can crawl my page?

No. It means your server returned the page to that header from your IP. curl ignores robots.txt, and IP-based rules can treat the real crawler differently, so check robots.txt and your logs as well.

Why does curl show less text than my browser?

curl does not run JavaScript, so content that scripts fetch or insert after load is missing from its output. Most AI crawlers in Vercel's December 2024 analysis did not render JavaScript either, so the curl text is the better guide to what they read.

Should I use -I or a full GET to check headers?

Use curl -s -o /dev/null -D - for a GET that prints headers. -I sends HEAD, which some servers answer differently or reject, while crawlers send GET.

Next step

If you would rather not run these commands by hand, run the free check on one public page; it needs no account. For a whole site, the full audit runs deterministic checks across up to 500 pages, adds a human review, and delivers a PDF with the evidence and prioritized fixes.