How to verify GPTBot IPs and spot fake AI crawlers

To verify a real AI crawler, ignore the user-agent string and check the request's source IP. For OpenAI, Anthropic and Perplexity, compare the IP with the JSON list of ranges the operator publishes. For Googlebot, Bingbot, Applebot and CCBot, run a reverse DNS lookup on the IP, check the hostname's domain, then run a forward lookup and confirm it returns the same IP. A request that claims a crawler's name but fails the check its operator documents is a fake, and blocking it does not block the real crawler.

This matters in two directions. Fakes inflate your "AI crawler" numbers and can scrape or probe your site under a trusted name. And if you block bots by name without checking, you also block the real OAI-SearchBot or PerplexityBot, the crawlers OpenAI and Perplexity use to surface sites in ChatGPT search and Perplexity search results.

Why the user agent is not proof

A user agent is a header the client writes about itself. This request looks exactly like OpenAI's training crawler to any rule that only reads the header:

curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://example.com/

A scraper can borrow a crawler's name to get past any rule that allows that bot by name alone. When you count AI crawler visits in your logs, as described in finding AI crawler visits in server logs, you are counting claims until you have checked the IPs behind them.

The fix is to tie each claim to something the client cannot fake easily: the IP address the TCP connection came from. Operators let you check that in one of two ways, and the AI crawler user agents list says which method each one supports.

Which crawler operators publish IP lists or DNS names

At the time of writing, these are the documented verification methods. When we fetched them on 8 November 2026, all the JSON files shared one shape: a creationTime field and a prefixes array of ipv4Prefix or ipv6Prefix CIDR entries (Common Crawl's file adds a couple of extra fields).

Sources: OpenAI's crawler overview, Anthropic's crawler article, Perplexity's bots page, Google's verification guide, Bing's verification help, Apple's Applebot page and Common Crawl's CCBot page. Amazon documents a separate IP list for each of Amazonbot, Amzn-SearchBot and Amzn-User on its Amazonbot page; its current page describes no reverse DNS method.

A few details are easy to miss:

• Anthropic's list is shared. bots.json does not say which bot uses which prefix, so it tells you a request is from Anthropic, not whether it was ClaudeBot or Claude-User. Anthropic also documents a separate identity, Claude-Agent, for browsers it hosts on behalf of users, with IPs listed under the Claude-Agent entry at claude.com/.well-known/sail rather than in bots.json.
• Google-Extended and Applebot-Extended are not crawlers. They are robots.txt tokens only. A request whose user agent says Google-Extended is fake by definition, because Google says the token has no separate user agent string.
• The lists change. When we checked, OpenAI's chatgpt-user.json had been regenerated on 6 November 2026. Bing says to refresh its list daily because it can change at any time. Never paste ranges into config by hand: fetch them on a schedule.

How to verify GPTBot IPs against OpenAI's JSON list

This shell function checks an IP against any of the JSON files above. It handles IPv4 and IPv6 and needs only curl and Python 3:

check_ip() {
  curl -s "$2" | python3 -c '
import json, sys, ipaddress
ip = ipaddress.ip_address(sys.argv[1])
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"), strict=False)
        for p in json.load(sys.stdin)["prefixes"]]
print("match" if any(ip in n for n in nets) else "no match")
' "$1"
}

check_ip 203.0.113.7 https://openai.com/gptbot.json

Replace 203.0.113.7 with the IP from your log line and pick the file that matches the claimed agent: gptbot.json for GPTBot, searchbot.json for OAI-SearchBot, perplexitybot.json for PerplexityBot, bots.json for any Claude agent. match means the request came from that operator's published range. no match on a request that claims the name means it did not come from the ranges the operator publishes for that agent.

Check against the file for the agent it claims. An IP in chatgpt-user.json that sends a GPTBot user agent is odd and worth a second look, even though it is an OpenAI address.

For a whole log, extract the distinct IPs first and loop over them:

grep "GPTBot" access.log | awk '{print $1}' | sort -u | while read -r ip; do
  echo "$ip $(check_ip "$ip" https://openai.com/gptbot.json)"
done

The function downloads the JSON on every call, which is fine for a few dozen IPs. For more, save the file once and read it locally. If your server sits behind a CDN or load balancer, the first field of the log line may be the proxy's address rather than the client's: log the forwarded client IP (for example with nginx's realip module) before you check anything.

How to verify Googlebot, Bingbot and Applebot with reverse DNS

Search engines that predate the AI crawlers mostly use forward-confirmed reverse DNS. The reverse lookup tells you the hostname the IP owner chose, and the forward lookup proves the hostname really belongs to that IP. You need both: anyone who controls an IP block can set its reverse DNS to say googlebot.com, but they cannot make Google's DNS point back at them.

Google's own example, using the host command:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The hostname is in googlebot.com and the forward lookup returns the original IP, so the request is from Googlebot. Google's guide lists the hostname patterns per crawler type: crawl-***.googlebot.com or geo-crawl-***.geo.googlebot.com for common crawlers like Googlebot, rate-limited-proxy-***.google.com for special-case crawlers, and ***.gae.googleusercontent.com or google-proxy-***.google.com for user-triggered fetchers.

To script the same check with dig:

verify_fcrdns() {
  ip="$1"; suffix="$2"
  name=$(dig +short -x "$ip" | grep -v '\.arpa\.$' | head -n1 | sed 's/\.$//')
  case "$name" in
    *"$suffix") ;;
    *) echo "FAIL: $ip reverse-resolves to '${name:-nothing}'"; return 1 ;;
  esac
  if dig +short "$name" A "$name" AAAA | grep -qxF "$ip"; then
    echo "OK: $ip is $name"
  else
    echo "FAIL: $name does not resolve back to $ip"; return 1
  fi
}

verify_fcrdns 66.249.66.1 .googlebot.com       # Googlebot
verify_fcrdns 157.55.39.1 .search.msn.com      # Bingbot
verify_fcrdns 17.241.208.161 .applebot.apple.com  # Applebot

The first IP is Google's own example, and the other two sit inside the ranges in bingbot.json and applebot.json at the time of writing. DNS records change, so test with an IP from your own logs. Keep the leading dot in the suffix: matching googlebot.com without it would also accept a hostname such as fakegooglebot.com. For IPv6 addresses, dig may print the address in a different form from your log, so the IP list method is more reliable there. Common Crawl says reverse DNS is not yet supported for CCBot over IPv6 at all, so use ccbot.json for those.

Bing also offers a web form, Verify Bingbot, if you only need to check one address.

How Cloudflare verified bots help

If your site is behind Cloudflare, it does much of this work for you. Cloudflare marks a bot as verified when it has confirmed the bot identifies itself honestly, through a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS, and behaves without abuse, including obeying robots.txt.

In WAF custom rules, the cf.client.bot field is true for a known good bot and carries the same signal as cf.bot_management.verified_bot. A rule that blocks impostors looks like this:

(http.user_agent contains "GPTBot" and not cf.client.bot)
or (http.user_agent contains "ClaudeBot" and not cf.client.bot)
or (http.user_agent contains "PerplexityBot" and not cf.client.bot)

Set the action to Block. Before you deploy it, confirm in Security Analytics that the real crawler's requests show as verified on your zone, because a bot missing from Cloudflare's verified list would be blocked too. Perplexity's own advice is the mirror image: build allow rules that match both the user agent and the current published IP ranges.

Two cautions. Cloudflare's Bot Fight Mode cannot be skipped by WAF custom rules, so an allow rule will not rescue a real crawler it challenges. And Cloudflare's AI crawler policies act on both verified bots and matching unverified ones, so check them before you assume a block came from your own rule. The Cloudflare AI crawler settings guide covers where each of these lives.

What to do with fake AI crawlers

Once you know a request is a fake, treat it as an unknown scraper using a borrowed name, not as the crawler it claims to be.

1. Do not touch robots.txt. Fakes do not read it, and changing it only affects the real crawlers. Anthropic's article makes a related point from the other side: it recommends robots.txt over IP blocking for opting out of its real bots, because IP blocking may not work reliably.
2. Block the combination, not the name. Refuse requests that claim a crawler name from an IP outside its published list or DNS domain. Blocking the name alone stops the real OAI-SearchBot and PerplexityBot as well.
3. Return a plain 403. A clear status is easy to spot in logs. Avoid challenge pages for anything that might be a real crawler: crawlers cannot solve them, and Anthropic says it will not attempt to bypass CAPTCHAs.
4. Rate-limit the rest. Traffic that claims no crawler name but behaves like one belongs in ordinary rate limits. Rate limiting AI crawlers shows how to set limits without catching verified crawlers.
5. Keep the evidence. Record the IP, claimed user agent, paths and verification result. It separates real AI crawler visits from noise when you report on them.

On your own server, nginx can enforce rule 2 with a generated allowlist. Refresh the include file on a schedule and reload nginx afterwards:

curl -fsS https://openai.com/gptbot.json | python3 -c '
import json, sys
for p in json.load(sys.stdin)["prefixes"]:
    print(p.get("ipv4Prefix") or p.get("ipv6Prefix"), "1;")
' > /etc/nginx/gptbot-ips.conf.new \
  && mv /etc/nginx/gptbot-ips.conf.new /etc/nginx/gptbot-ips.conf \
  && nginx -t && nginx -s reload

Writing to a .new file first means a failed download leaves the old list in place instead of an empty one, which would mark every real GPTBot request as fake.

The geo and map blocks go in the http context and the if goes inside your server block:

geo $gptbot_ip {
    default 0;
    include /etc/nginx/gptbot-ips.conf;
}

map "$http_user_agent:$gptbot_ip" $fake_gptbot {
    default        0;
    "~GPTBot.*:0$" 1;
}

if ($fake_gptbot) {
    return 403;
}

Requests that say GPTBot from an IP outside OpenAI's list get a 403. Real GPTBot traffic and normal visitors are unaffected. geo reads $remote_addr by default, so behind a CDN or load balancer configure the realip module first, or every request will appear to come from the proxy. Repeat the pattern for each agent you care about. If you go further and allowlist only known IPs for a whole section of the site, read geo-blocking and IP allowlists first: stale lists are a common reason real crawlers get locked out.

Frequently asked questions

Can I verify GPTBot with reverse DNS?

OpenAI does not document a reverse DNS method. It publishes IP ranges instead, so check the source IP against gptbot.json, and use searchbot.json or chatgpt-user.json for its other agents.

Is a request with a Googlebot user agent from a Google Cloud IP real Googlebot?

Not necessarily. Anyone can rent Google Cloud servers. Googlebot itself reverse-resolves to googlebot.com names, and only certain user-triggered fetchers resolve to gae.googleusercontent.com, so run the full reverse and forward check before trusting it.

Should I block every unverified request that claims to be an AI crawler?

Blocking a claimed name from an IP outside the operator's published list is low risk, because operators publish those lists so you can recognize their crawlers. Check first that you are reading the current list for that exact agent, since the files change often.

Does blocking fake crawlers affect my visibility in AI search?

No. A request from outside an operator's published ranges or DNS domain is not that operator's crawler, so blocking it does not block ChatGPT search, Perplexity or Claude. What affects visibility is whether the verified crawlers can reach your pages, so make sure your blocking rules check the IP and not only the name.

How often should I refresh the IP lists?

Fetch them automatically at least daily. Bing explicitly says to refresh its list daily because it can change at any time, and OpenAI's chatgpt-user.json was regenerated two days before this post was written.

Next step

If you are not sure whether your CDN or firewall treats the real crawlers differently from the fakes, run the free single-page check as a next step: it needs no account and shows the robots.txt rule each named AI crawler gets and whether a challenge page is in the way. If you want verified visit counts rather than claims, Crawlito's monthly retainer includes a verified AI-crawler visit report from your server or Cloudflare logs. For the details of OpenAI's four agents, see GPTBot, OAI-SearchBot and ChatGPT-User explained.