Geo blocking crawlers: how country and IP rules shut out AI

Geo blocking crawlers usually happens by accident. AI crawlers connect from cloud data centers, most of them in the US, so a rule that blocks the US, allows only your own country, or blocks hosting networks can return a 403 to GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot while your local visitors see a normal site. The fix is to keep the geo rule and exempt verified crawlers by their published IP ranges, or to narrow the rule so it no longer catches them.

This is one of the harder blocks to spot, because nothing looks wrong from where you sit. The rest of this post shows where the crawlers come from, how to detect the block, and how to write the exemption.

Where AI crawlers connect from

Operators publish their crawler IP ranges, and you can look up who owns each range and where geolocation databases place it. We did that on 8 November 2026, taking the first address of every prefix in each official JSON file and looking it up with ipinfo.io:

Two patterns matter. The search and training crawlers in the table run from US addresses on Microsoft, Amazon and Google Cloud networks. And the user-triggered ChatGPT-User agent fetches from many regions, so a country allowlist can fail some of those fetches and not others.

Google says the same of its own crawler. Its locale-adaptive pages guide states that Googlebot's default IP addresses appear to be based in the USA, and that Googlebot also crawls from IP addresses outside the USA.

Treat the table as a snapshot. The lists change often, and geolocation databases disagree with each other: Cloudflare, MaxMind and ipinfo.io can place the same address in different countries. Your firewall uses its own database, not ours.

Which rules block crawlers by location or network

Geo and network rules turn up in more places than most site owners expect. Check each layer:

The common patterns that catch AI crawlers:

1. Allow only your own market. A Dutch shop that allows only EU traffic blocks every search and training crawler in the table above, and lets through only the ChatGPT-User fetches that happen to come from EU addresses.
2. Block the US. Sites that do this for legal or licensing reasons block nearly all of these crawlers.
3. Block hosting and cloud networks. Blocking Microsoft, Amazon and Google Cloud ASNs stops many scrapers, and also OpenAI's, Anthropic's and Perplexity's crawlers, because they run on exactly those networks.
4. Allowlist office or partner IPs. A whole site, or a docs or pricing section, open only to known IPs is invisible to every crawler.

On Cloudflare, note two details from the IP Access Rules documentation: blocking by country there is available only on Enterprise plans, so most country blocks live in WAF custom rules, and an IP that Cloudflare allows globally overrides an IP Access Rules country block but not a country block made with a custom rule.

Why geo-blocking is a silent block

A geo block is invisible from inside the allowed region. Your browser loads the site, your uptime monitor in the same country reports green, and your team sees nothing odd. The crawler gets a 403, and nobody tells you.

The logs often hide it too. When the block happens at the CDN, the request never reaches your origin, so your server logs simply show no AI crawler visits at all. That looks like "the crawlers have not found us yet" when the real answer is "the edge turns them away".

The effect on AI search is direct. OpenAI's crawler documentation recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges, so a correct robots.txt is not enough if the network refuses the connection. A blocked OAI-SearchBot cannot read your pages for ChatGPT search, and a blocked ChatGPT-User or Claude-User cannot fetch the page a user just asked about.

How to detect a geo or IP block

Read your edge logs for crawler user agents

Start where the block happens. In Cloudflare, open the security events log and filter by user agent containing OAI-SearchBot, GPTBot, ClaudeBot or PerplexityBot. Each event names the rule that acted on it. On CloudFront, look for 403 in the sc-status field of the standard logs. AWS notes that these logs alone cannot tell a geo rejection apart from other 403s, so compare the c-ip client address with the crawler lists.

If you only have origin logs in the combined format, this counts status codes per crawler:

awk -F'"' '{ split($3, s, " ")
  if (match($6, /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User/))
    print substr($6, RSTART, RLENGTH), s[1] }' access.log | sort | uniq -c | sort -rn

A crawler with only 403 lines is being refused at the origin. A crawler with no lines at all, on a site that has been live for weeks, points to a block further up. The guide to AI crawler visits in server logs covers log formats in more detail.

Test from a US cloud region

To see what a crawler sees, send the request from where crawlers live. Start a small VM or cloud shell in a US region of Azure, AWS or Google Cloud, then compare its result with the same request from your office:

UA="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"
curl -sS -o /dev/null -D - -A "$UA" https://example.com/ | head -n 20

A 200 from your office and a 403 from the VM points to a geo or network rule. A 403 with a cf-mitigated: challenge header is a Cloudflare challenge rather than a block, which is a different problem covered in WAF challenge pages and AI crawlers. The curl guide to seeing your page as a crawler has more variations.

This test runs from the right country and network but not from the crawler's own IPs. If your rule already exempts the published ranges correctly, your VM should still be blocked, and that is the expected result.

How to exempt verified crawlers by IP range

The safe exemption ties the request to the operator's published ranges or to a verified-bot signal. Never exempt by user agent alone: anyone can send GPTBot in a header, which is why verifying real AI crawlers relies on the source IP.

Cloudflare custom rules

Add the exemption to the block rule itself. With the action set to Block, this expression blocks traffic from outside Germany, Austria and Switzerland unless Cloudflare identifies it as a known good bot:

not (ip.src.country in {"DE" "AT" "CH"}) and not cf.client.bot

The same idea for a hosting-network block:

ip.src.asnum in {8075 14618 16509 396982} and not cf.client.bot

The ip.src.country and ip.src.asnum fields replace the deprecated ip.geoip.country and ip.geoip.asnum. Before you rely on cf.client.bot, check in Security Analytics that the crawlers you care about show as verified on your zone. If one does not, keep an IP list built from the operator JSON files, for example $ai_crawler_ips, and add and not ip.src in $ai_crawler_ips to the expression. Cloudflare's AI crawler policies and Bot Fight Mode act separately from these rules, and Bot Fight Mode cannot be skipped with a custom rule, so check them too with the Cloudflare AI crawler settings guide.

nginx with a generated allowlist

Generate an nginx geo file from the official lists on a schedule, so nobody pastes ranges by hand. All these files share the same prefixes format:

#!/bin/sh
set -e
out=/etc/nginx/crawler-ips.conf
tmp=$(mktemp "$out.XXXXXX")
trap 'rm -f "$tmp"' EXIT
for url in \
  https://openai.com/searchbot.json \
  https://openai.com/chatgpt-user.json \
  https://openai.com/gptbot.json \
  https://claude.com/crawling/bots.json \
  https://www.perplexity.ai/perplexitybot.json \
  https://www.perplexity.ai/perplexity-user.json \
  https://developers.google.com/static/crawling/ipranges/common-crawlers.json \
  https://www.bing.com/toolbox/bingbot.json \
  https://search.developer.apple.com/applebot.json
do
  curl -fsS "$url" | python3 -c '
import json, sys
for p in json.load(sys.stdin)["prefixes"]:
    print((p.get("ipv4Prefix") or p.get("ipv6Prefix")) + " 1;")'
done > "$tmp"
[ -f "$out" ] && cp "$out" "$out.bak"
mv "$tmp" "$out"
if ! nginx -t; then
  [ -f "$out.bak" ] && mv "$out.bak" "$out"
  exit 1
fi
nginx -s reload

If any download fails, the JSON parse fails with it and set -e stops the script before the old file is replaced; if the new file fails nginx -t, the previous one is restored. At the time of writing these nine lists produce about 700 lines. Then combine the allowlist with your country rule. This example uses the third-party ngx_http_geoip2_module and a MaxMind country database:

geoip2 /usr/share/GeoIP/GeoLite2-Country.mmdb {
    $geoip2_data_country_code country iso_code;
}

geo $verified_crawler {
    default 0;
    include /etc/nginx/crawler-ips.conf;
}

map $geoip2_data_country_code $outside_market {
    default 1;
    DE 0;
    AT 0;
    CH 0;
}

map "$outside_market:$verified_crawler" $geo_deny {
    default 0;
    "1:0"   1;
}

server {
    # ...
    if ($geo_deny) { return 403; }
}

The geo block and geoip2 both read the connecting address by default. If nginx sits behind a CDN or load balancer, set up the realip module first, or every request will look like it comes from the proxy.

Include only the crawlers you want to let through. A network exemption for GPTBot does not decide whether your content may be used for training; robots.txt does, as explained in allowing AI search but blocking training. Anthropic makes a related point in its crawler help article: it recommends robots.txt rather than IP blocking for opting out, saying IP blocking may not work correctly or persistently.

CloudFront

CloudFront's geographic restrictions apply to the entire distribution and have no IP exception, and AWS says that requests they block are never forwarded to AWS WAF. To exempt crawlers, turn the CloudFront restriction off and recreate the country rule in AWS WAF. The geo match documentation gives the pattern: a Block rule with an AND statement that combines a geo match on the countries you block with a NOT around an IP set statement. Build the IP set from the same JSON lists.

When a geo block should stay in place

Some blocks exist for legal reasons, such as licensing or sanctions. Google's guidance for that case is to treat Googlebot like any other visitor from the country it appears to come from, so its example says a site that blocks US users should also block Googlebot when it crawls from the US. If your block is a legal requirement, check with whoever owns that requirement before you exempt any crawler, and accept that crawlers will see what users in that country see.

If the block is a security measure against scrapers, a verified-crawler exemption costs little protection: traffic from an operator's published ranges comes from that operator, not from a scraper borrowing its name. You still choose which operators to let through, and user-triggered agents such as ChatGPT-User and Perplexity-User may not follow robots.txt. Rate limits are often a better tool than a country ban against scrapers.

Frequently asked questions

Do AI crawlers crawl from my country?

Mostly not. On 8 November 2026, ipinfo.io placed every published GPTBot, OAI-SearchBot, PerplexityBot and Anthropic crawler prefix in the US, while ChatGPT-User prefixes were spread over 21 countries. Assume a US data center unless the operator says otherwise.

Is it safe to allowlist crawlers by user agent?

No. A user agent is a header any client can write, so a user-agent exemption lets every scraper that borrows a crawler's name through your geo rule. Match the published IP ranges or a verified-bot signal such as Cloudflare's cf.client.bot.

How often should I refresh crawler IP lists?

Daily is a reasonable default. Bing says to refresh its list daily, and OpenAI's chatgpt-user.json had been regenerated two days before we checked it.

Does blocking hosting providers block Googlebot?

Usually not, because Googlebot runs on Google's own network (AS15169 for the addresses we checked), not on Google Cloud's AS396982. Anthropic's crawlers, however, used Google Cloud addresses for 13 of their 14 prefixes at the time of writing, so a Google Cloud block catches them.

Check what your site returns to crawlers

If you are not sure whether a country or network rule is in the way, run the free check on one page to see the status and any block page it returns, or book a full audit, which reports access problems like this across up to 500 pages with the evidence for each.