AI crawler IP ranges: keep a firewall allowlist in sync
Where OpenAI, Perplexity, Anthropic, Google and Bing publish crawler IP ranges, and how to build a firewall allowlist that updates itself safely.
By Crawlito · 7 min read
In short
- OpenAI publishes one IP file per crawler (gptbot.json, searchbot.json, chatgpt-user.json), and Perplexity publishes perplexitybot.json and perplexity-user.json.
- Anthropic publishes a single bots.json for all its crawlers, but says blocking by IP may not work as an opt-out and points site owners to robots.txt instead.
- Google and Bing publish JSON lists too, alongside reverse DNS verification, and all these files share a creationTime field and a prefixes array.
- A safe allowlist job fetches each file on a schedule, rejects empty or malformed lists, keeps the last good copy and replaces the firewall list in one step.
- Allow by IP and user agent together, one IP set per crawler, so a scraper that borrows a crawler's name from another network is not let through.
On this page
- Where AI crawler IP range files are published
- What each operator says about allowlisting
- How to build an allowlist job that updates itself
- Using the list in nginx, Cloudflare and AWS WAF
- Common allowlist mistakes
- How to check that the allowlist works
- Frequently asked questions
- Check your crawler access end to end
Most AI crawler operators now publish their IP ranges as JSON files: OpenAI has one per crawler, Perplexity has two, Anthropic has one shared file, and Google and Bing publish lists alongside their DNS verification. To keep a firewall allowlist correct, fetch those files on a schedule, check that each one parsed and is not empty, and replace the list in your firewall in one step. Never paste ranges into config by hand, because the files change and a stale list quietly blocks the crawler you meant to let in.
This post is about the allowlist itself: which files to use, how to turn them into nginx, Cloudflare or AWS WAF rules, and how to stop the job from breaking your site when a download fails. If you only need to check whether one log line came from a real crawler, how to verify real AI crawlers covers that with reverse DNS and a one-off script.
Where AI crawler IP range files are published
At the time of writing, these are the files the operators document. Each is linked from the operator's own crawler page.
| Operator | Crawler | IP file | Documented in |
|---|---|---|---|
| OpenAI | OAI-SearchBot | searchbot.json | OpenAI crawlers overview |
| OpenAI | GPTBot | gptbot.json | same |
| OpenAI | ChatGPT-User | chatgpt-user.json | same |
| OpenAI | OAI-AdsBot | adsbot.json | same |
| Perplexity | PerplexityBot | perplexitybot.json | Perplexity crawlers |
| Perplexity | Perplexity-User | perplexity-user.json | same |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User | bots.json (one shared file) | Anthropic crawler article |
| Googlebot and other common crawlers | common-crawlers.json | Verify Google's crawlers | |
| Microsoft | Bingbot | bingbot.json | Bing Webmaster Tools |
Google publishes more than one file. Besides common-crawlers.json, its verification page lists special-crawlers.json (for example AdsBot), user-triggered-fetchers.json, user-triggered-fetchers-google.json and user-triggered-agents.json, all under https://developers.google.com/static/crawling/ipranges/. Pick the file that matches the crawler you want to allow, not Google's general goog.json, which covers other Google services.
All of these files use the same shape, which makes one parser enough:
{
"creationTime": "2026-10-20T00:00:00Z",
"prefixes": [
{ "ipv4Prefix": "203.0.113.0/24" },
{ "ipv6Prefix": "2001:db8::/32" }
]
}The creationTime value tells you when the operator last regenerated the list. Log it on every run: it is the quickest way to see that a list changed and to explain a sudden block later.
What each operator says about allowlisting
The operators do not all take the same position, and it changes what the list is good for.
- OpenAI recommends allowing requests from its published OAI-SearchBot ranges if you want your site to appear in ChatGPT search. Its overview gives a separate file per crawler, so you can allow OAI-SearchBot and ChatGPT-User while treating GPTBot differently. See GPTBot, OAI-SearchBot and ChatGPT-User for what each one does.
- Perplexity gives the most specific firewall advice. Its crawler page says to match both the user agent and the IP ranges, give each bot its own IP set, use the current ranges from the JSON endpoints and automate the refresh. For Cloudflare it suggests a custom rule with a Skip action placed above rules that block or challenge, and it warns that Bot Fight Mode cannot be bypassed by custom rules.
- Anthropic publishes
bots.jsonand says a source IP on that list indicates the request came from Anthropic. It does not split the list by crawler. It also warns that blocking its IP addresses may not work as a lasting opt-out, because doing so stops it from reading your robots.txt, and it directs site owners to robots.txt for opting out. - Google and Bing document reverse and forward DNS as a verification method as well as their JSON lists. For a firewall, the JSON list is easier, because a firewall cannot run a DNS lookup on each new connection.
Anthropic's point applies more widely than to Anthropic. An IP list is good for letting a crawler through a firewall that would otherwise block it. It is a poor way to say no, because robots.txt is the control each of these operators documents for opting out.
How to build an allowlist job that updates itself
The job has four steps: fetch, validate, render, swap. The validation step is the one people skip, and it is the one that prevents an outage. If a download returns an HTML error page or an empty array and you write that straight into your firewall, the allowlist becomes empty and every rule that depends on it starts blocking real crawlers.
This script fetches one file, refuses anything that does not parse or has no prefixes, and writes an nginx geo include only when the new list is valid:
#!/usr/bin/env bash
set -euo pipefail
name="$1" # e.g. oai-searchbot
url="$2" # e.g. https://openai.com/searchbot.json
dir=/etc/nginx/ai-ips
tmp=$(mktemp)
curl -fsS --max-time 30 "$url" -o "$tmp"
python3 - "$tmp" "$dir/$name.conf.new" <<'PY'
import json, sys, ipaddress
data = json.load(open(sys.argv[1]))
nets = []
for p in data["prefixes"]:
cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
nets.append(ipaddress.ip_network(cidr, strict=False))
if not nets:
sys.exit("empty prefix list, keeping the old file")
with open(sys.argv[2], "w") as out:
out.write(f"# creationTime {data.get('creationTime')}\n")
for n in nets:
out.write(f"{n} 1;\n")
print(f"{len(nets)} prefixes, creationTime {data.get('creationTime')}")
PY
mv "$dir/$name.conf.new" "$dir/$name.conf"
nginx -t && systemctl reload nginx
rm -f "$tmp"Because set -e stops the script at the first failure, a failed download or a rejected list leaves the previous .conf file in place. That is the behaviour you want: an old list that is mostly right is better than a new one that is empty. Run the script from cron or a systemd timer, once per crawler. Operators do not publish a refresh schedule for most of these files, so a daily run is a sensible default, and more often does no harm.
Add two more checks once it works:
- Alert on large shrinkage. If a list drops from 40 prefixes to 2, keep the old one and send yourself a message instead of applying it.
- Alert on failure streaks. One failed fetch is noise. A week of failures means your allowlist is drifting from reality.
Using the list in nginx, Cloudflare and AWS WAF
nginx
The geo module maps the client address to a value, and it accepts an include. Combine it with the user agent so a request needs both to pass:
geo $oai_searchbot_ip {
default 0;
include /etc/nginx/ai-ips/oai-searchbot.conf;
}
map "$oai_searchbot_ip:$http_user_agent" $allow_oai_searchbot {
default 0;
"~^1:.*OAI-SearchBot" 1;
}Use $allow_oai_searchbot to skip your rate limit or deny rule for that crawler. If nginx sits behind a load balancer or CDN, configure the realip module first, or geo will see the proxy's address instead of the crawler's.
Cloudflare
Cloudflare's custom IP lists accept IPv4 and IPv6 addresses and CIDR ranges, and you reference a list in a rule expression as ip.src in $list_name. The Lists API has an "Update all list items" operation, PUT accounts/{account_id}/rules/lists/{list_id}/items, which deletes the current items and replaces them in one asynchronous operation. That is the swap step: build the full array of items from the JSON file, validate it, then send it in one call.
A Skip rule for PerplexityBot, placed above your blocking rules, would use an expression like this:
(http.user_agent contains "PerplexityBot" and ip.src in $perplexitybot)Keep in mind the limit both Cloudflare and Perplexity document: Bot Fight Mode does not run on the Ruleset Engine, so a Skip rule does not exempt anything from it. The Cloudflare AI crawler settings guide explains where each bot setting lives.
AWS WAF
Perplexity's advice for AWS WAF is to create an IP set per bot and an Allow rule that joins the IP set match and a user-agent match with an AND statement, with a lower priority number than your blocking rules. AWS WAF IP sets hold either IPv4 or IPv6 addresses, so a file with both needs two sets. Updates go through aws wafv2 update-ip-set, which replaces the address list and needs the lock token from a fresh get-ip-set call. Our AWS WAF Bot Control guide covers where these rules sit next to Bot Control.
Common allowlist mistakes
| Mistake | What happens | Fix |
|---|---|---|
| Ranges pasted into config once | New prefixes are blocked as the operator adds them | Fetch on a schedule |
| Allowing by user agent only | Any scraper can send PerplexityBot and pass | Require IP and user agent together |
| One combined list for all OpenAI bots | Allowing ChatGPT-User also allows GPTBot addresses | One list per file, one rule per crawler |
| Using Anthropic's list to tell bots apart | bots.json is shared, so you cannot | Use the user agent to tell them apart, the list to confirm Anthropic |
| Writing an empty list on a failed fetch | Every crawler is blocked until someone notices | Validate before swapping, keep the last good copy |
| Allowlist in the WAF, block in a security plugin | Requests pass the edge and fail at the origin | Check every layer, as in geo-blocking and IP allowlists |
How to check that the allowlist works
Do not wait for the crawler to visit. After each change:
- Confirm the generated file has the prefix count and
creationTimeyou expect. - Pick one IP from the operator's file and test your rule logic against it, for example with Cloudflare's Security Analytics or your own access log after a crawl.
- Search your logs for the crawler's user agent and look at the status codes. A run of
403or challenge responses for requests from listed IPs means a rule above or below your allowlist still catches them.
Frequently asked questions
How often do AI crawler IP ranges change?
Most operators do not publish a schedule. Each file has a creationTime field, and comparing it between runs shows when a list was regenerated. A daily fetch is a reasonable default.
Does Anthropic publish separate IP lists for ClaudeBot and Claude-SearchBot?
No. At the time of writing Anthropic publishes one bots.json file for its crawlers. It tells you a request came from Anthropic, and the user agent tells you which crawler it claims to be.
Can I block AI training crawlers by IP instead of robots.txt?
You can, but Anthropic warns that IP blocking may not work as a lasting opt-out because it stops the crawler reading your robots.txt. robots.txt is the opt-out the operators document, and it lets you treat training and search crawlers differently.
Does allowlisting an IP range make a crawler visit my site?
No. An allowlist only removes a barrier. Whether and how often a crawler fetches your pages is decided by the operator.
Is there an IP list for Google-Extended?
No. Google-Extended is a robots.txt token, not a separate crawler, so there are no requests or IP addresses to allow for it.
Check your crawler access end to end
An allowlist is one layer among several, and a crawler can pass your WAF only to be challenged by a bot setting or blocked by robots.txt further on. The free check tests a single page against the main AI crawlers with no account, and a full audit reviews robots.txt, edge rules and responses across up to 500 pages.
Filed under AI crawlers. Topics: ai-crawlers, waf, cloudflare, bot-protection, gptbot, oai-searchbot