Anubis bot protection: does it block Googlebot and AI search?
Anubis's default policy lets verified Googlebot and Bingbot through but denies OAI-SearchBot and other AI search crawlers. How to check it and allow them.
By Crawlito · 10 min read
In short
- At the time of writing, the default Anubis policy allows Googlebot, Bingbot, Applebot and DuckDuckBot only when both the user agent and the source IP match a published range.
- The same default policy denies OAI-SearchBot, Claude-SearchBot, PerplexityBot and ChatGPT-User by user agent, so a default install keeps AI search crawlers out even when robots.txt allows them.
- Anubis evaluates rules in order and the first match wins, so an allow rule for a search crawler has to sit above the import that denies AI crawlers.
- Anubis sends challenge and denial pages with HTTP 200, so a status-code check cannot tell you whether a crawler got your content.
- The IP ranges in Anubis's allow rules are copies; when we compared them with OpenAI's and Bing's published lists, some prefixes were missing.
On this page
- What Anubis is and how the challenge works
- Why a challenge stops crawlers that fetch raw HTML
- What the default policy allows and denies
- How to check what Googlebot and AI search bots receive
- How to allow search crawlers with policy rules
- Trade-offs: blocking scrapers or losing AI search visibility
- How to monitor the challenge after changes
- Frequently asked questions
- Check what crawlers see on your site
At the time of writing, the default Anubis policy lets verified Googlebot, Bingbot and several other search crawlers through, and denies the crawlers behind AI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user-triggered ChatGPT-User. So a default install can keep ChatGPT search away from your pages even when your robots.txt allows it. You can change that with a few policy rules, but rule order matters and the IP ranges Anubis ships are snapshots.
This post is based on the default policy in the Anubis repository as we read it on 11 November 2026. Your installed version may differ, so the checks below show you how to read your own.
What Anubis is and how the challenge works
Anubis is an open-source (MIT licence) filter that sits in front of a site and checks incoming connections with one or more challenges, to protect the site from scraper bots. Its README calls it a fallback for sites that cannot use a service like Cloudflare, and warns that it may also block well-known useful bots, such as the Internet Archive's, unless you allowlist them in a bot policy.
The default challenge is proof of work. According to the project's proof-of-work explanation, the server sends a challenge, the visitor's browser adds a number to it and hashes the result with SHA-256 until the hash starts with enough zeros, and the server checks the answer with one hash. The proof-of-work docs say the fast and slow methods need JavaScript. A separate metarefresh method needs none, and its docs page says it is not enabled by default.
Which challenge a visitor gets depends on a weight. In the default policy, any user agent containing Mozilla or Opera gains 10 weight. The default thresholds then allow a weight of 0 or less, use metarefresh for weights above 0 and below 10, and use proof of work from 10 upward, at difficulty 2 for 10 to 19, 4 for 20 to 29 and 6 beyond that. The weight rules in the default file each add 10, so the metarefresh band is not reached by those rules alone. In practice, a request that looks like a browser and matches no allow rule gets a proof-of-work challenge.
Why a challenge stops crawlers that fetch raw HTML
A challenged request does not receive your page. It receives the challenge page, and a client has to run the script to get through. A crawler that reads the raw HTML sees only the interstitial, whatever your robots.txt says. We cover the general mechanism in why a WAF challenge page shows AI bots nothing, and what is known about script execution in do AI crawlers run JavaScript?.
That is why the default policy does not make search engines solve anything. It recognises them by identity instead. A comment in the default file says Google is on the allow list so that it does not try to bypass Anubis. The next sections show which crawlers get that treatment, and which do not.
What the default policy allows and denies
The table summarises the default file, data/botPolicies.yaml, and the snippets it imports, as of 11 November 2026.
| Who | Default action | Where it comes from |
|---|---|---|
| Googlebot, Applebot, Bingbot, DuckDuckBot, Qwantbot, Kagi, Mojeek, Marginalia, Yandex, the Internet Archive, Common Crawl | Allow, if the identity check passes (user agent plus IP range; IP range alone for the Internet Archive; reverse DNS for Yandex) | _allow-good.yaml |
| OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer | Deny | ai-search.yaml |
| GPTBot, ClaudeBot | Deny | ai-training.yaml |
| ChatGPT-User, Claude-User, MistralAI-User, Perplexity-User | Deny | clients/ai.yaml |
| Other AI agents on a long list, such as Amazonbot and Bytespider | Deny | ai-catchall.yaml |
/robots.txt, /sitemap.xml, /.well-known/, favicons | Allow, but only for requests that no earlier row matched | keep-internet-working.yaml |
Anything else with Mozilla or Opera in the user agent | Challenge, by weight | generic-browser rule |
Three details decide whether a crawler gets through.
Order. Anubis's default-allow documentation says rules are evaluated in order and the first match wins. In the default file, the AI deny import comes before the search-engine allow import. A request that matches nothing is allowed, which the docs call the implicit rule. The import that allows /robots.txt and /sitemap.xml comes after the AI deny imports, so a user agent that is denied is denied on those paths too.
Two conditions. An allow rule for Googlebot or OAI-SearchBot has a user_agent_regex and a remote_addresses list. The policy docs describe this as allowing a search engine if and only if its IP address matches the published ones. A request with the right user agent from another IP does not match that rule, so it moves on to the later rules. Not every allow rule works this way: the Internet Archive rule checks only the IP range, and Yandex is verified by reverse DNS.
Copied ranges. The IP lists are copied into the snippets, and the (data) folder is a copy as of the time your Anubis was compiled. We compared the snippets with the operators' own files on 11 November 2026:
- Every one of the 317 prefixes in Google's
common-crawlers.jsonwas covered by the Googlebot snippet. - 4 of the 39 prefixes in OpenAI's
searchbot.jsonwere not in the OpenAI search snippet. - 1 of the 29 prefixes in Bing's
bingbot.jsonwas not in the Bingbot snippet.
We cannot say how much real traffic comes from the missing ranges, only that a copy lags behind its source.
There is also a user-agent detail. The OpenAI search snippet matches OAI-SearchBot/1\.0, while OpenAI's crawler documentation shows OAI-SearchBot/1.4 in its example string. Version numbers are not fixed, so compare the version in your logs with the version in your rule: if they differ, the allow rule will not match.
How to check what Googlebot and AI search bots receive
Start with a curl test, then confirm with evidence from the real crawlers. The test shows what a given user agent gets from your IP address, which is useful for deny rules and challenge rules. It cannot test an allow rule that also checks the IP, because your IP is not Google's.
UA_BROWSER='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
UA_SEARCH="$UA_BROWSER; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"
body=$(mktemp)
for ua in "$UA_BROWSER" "$UA_SEARCH"; do
code=$(curl -s -o "$body" -w '%{http_code}' -A "$ua" https://example.com/)
echo "$code $(grep -o '<title>[^<]*</title>' "$body")"
doneDo not trust the status code. The policy docs describe the deny response as an error message that scrapers think is a success, and the default file sets both challenge and denial responses to HTTP 200. Read the page title instead. This is what we saw from Anubis's own documentation site on 11 November 2026, using commands like the ones above. The page titles end with an exclamation mark, which we left off the table:
| User agent | Status | Page title |
|---|---|---|
| Chrome-style browser string | 200 | Making sure you're not a bot (challenge page) |
| OAI-SearchBot string | 200 | Oh noes (with an "Access Denied" message and an error code) |
| GPTBot string | 200 | Oh noes (with an "Access Denied" message and an error code) |
That is one site's configuration, and the denials match the default file. The titles are page text, so a site can customise them. Look for a title or body that is not your own content.
For Googlebot, use evidence from Google rather than a spoofed string:
- Google Search Console. The URL Inspection live test shows the page fetch result, the raw HTML and, on a successful test, a screenshot. Google's help page describes the screenshot as the page as Google-InspectionTool sees it, a different user agent from Googlebot, so treat the result as a strong hint, not proof of what Googlebot receives.
- Your logs. Look for Googlebot requests whose IPs verify as Google's, using Google's DNS check or its published ranges, and see what they received. How to verify real AI crawlers shows the method for several operators.
How to allow search crawlers with policy rules
First, see the policy you are actually running. Anubis can write out its compiled-in data, including botPolicies.yaml and the crawler snippets:
anubis --extract-resources ./var/anubis-resourcesThen choose one of two routes.
Route 1: import the moderate AI policy. The repository ships ai-block-moderate.yaml, which by its own description blocks AI bots used for training or with undocumented purposes, and permits those with documented non-training use and published IP allowlists. It imports allow rules for OAI-SearchBot and for ChatGPT-User and MistralAI-User, and still denies GPTBot and ClaudeBot. Replace the ai-block-aggressive.yaml import with (data)/meta/ai-block-moderate.yaml, then check the version-string issue above. One catch: in the file as we read it, the catch-all list that moderate imports first also names PerplexityBot and Perplexity-User, so under the first-match rule those two look denied even though moderate also imports Perplexity allow rules. Test them rather than assuming.
Route 2: write your own rule above the deny. This gives you control of the version and the ranges. The rule needs both conditions, and it must come before the import that denies AI crawlers:
bots:
- name: openai-searchbot-verified
user_agent_regex: OAI-SearchBot/[0-9.]+
action: ALLOW
remote_addresses:
- 192.0.2.0/28 # placeholder: replace with the generated list below
- import: (data)/bots/_deny-pathological.yaml
# ...your other default imports, in their original order
- import: (data)/meta/ai-block-aggressive.yaml
- import: (data)/crawlers/_allow-good.yaml
- import: (data)/common/keep-internet-working.yaml
# ...the rest of your existing policy, unchangedGenerate the address list from OpenAI's file rather than pasting it by hand. The file lists each range as an ipv4Prefix entry:
curl -s https://openai.com/searchbot.json \
| jq -r '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) | " - " + .'Re-run it on a schedule, because OpenAI can change the list. The same pattern works for Google's and Bing's files if your compiled copy is older than their lists.
Claude-SearchBot needs a decision. A comment in Anubis's own files notes that it has no published IP allowlist, so Anubis ships no verified allow rule for it. An allow rule on the user agent alone would also admit anyone who copies that string.
robots.txt is a separate layer. In the default policy, a request that no earlier rule matched can fetch /robots.txt, but a user agent on the deny list is denied there too: we saw the "Access Denied" page for GPTBot and OAI-SearchBot strings on /robots.txt on Anubis's documentation site on 11 November 2026. So allowing a crawler in robots.txt does nothing while Anubis denies it, and a crawler that Anubis lets in is still subject to your robots.txt rules. A comment in Anubis's AI policy also warns that some training opt-outs, such as Google's, work only through robots.txt flags and not by blocking a user agent.
Trade-offs: blocking scrapers or losing AI search visibility
The aggressive default does not distinguish training crawlers from search crawlers. If you want to stay out of model training but remain eligible for AI search, you need the narrower policy above. OpenAI says its search crawler and its training crawler are independent robots.txt settings, and the edge rules should reflect the same split.
- Allowing a crawler removes a barrier, nothing more. Whether any assistant uses your pages in an answer depends on factors outside your site. See why ChatGPT can't see your website for the other causes to rule out.
- User-agent-only rules are spoofable; IP rules go stale. Combine both, and refresh the lists.
- User-triggered agents are a separate choice. OpenAI says robots.txt rules may not apply to ChatGPT-User, so your edge rule is the control that applies. Anubis denies it by default.
- Load still matters. If you opened the door because the challenge cost you visibility, you may still need limits on crawl rates. Rate limiting AI crawlers covers that without blocking search.
How to monitor the challenge after changes
- Diff your policy against upstream after each upgrade. Run
--extract-resourceson the new version and compare the crawler snippets and the import order. - Check crawler hits in your logs. Look for the tokens Googlebot, Bingbot and OAI-SearchBot, and keep only requests whose IPs verify. If a verified crawler appears only on
robots.txt, a challenge or deny rule may be stopping it before it reaches your pages, because the default policy lets unmatched requests fetch/robots.txtregardless of the challenge. - Use Anubis's own signals. Rule names appear in its Prometheus metrics when you enable the metrics server, and it passes
X-Anubis-Rule,X-Anubis-ActionandX-Anubis-Statusheaders to the service behind it. Requests that were denied, or challenged and not yet passed, are answered by Anubis and do not normally reach your backend, so the metrics are the place to see them. - Repeat the curl test from section three after every policy or version change, and check Search Console for fetch errors on key URLs.
Frequently asked questions
Does Anubis block Googlebot by default?
Not when the request comes from Google's published ranges. The default policy allows Googlebot if the user agent and the IP both match, and at the time of writing the Googlebot snippet covered every prefix in Google's published list. A Googlebot user agent from any other IP does not match that rule.
Does Anubis block ChatGPT search?
The default policy denies OAI-SearchBot by user agent, and it does so before it reaches the search-engine allow list. To let it through, add an allow rule above the deny, or switch to the moderate policy, then check the user-agent version matches.
Is the proof-of-work challenge the same as a CAPTCHA?
No. It asks the visitor's browser to run a hash computation in JavaScript, not to solve a puzzle by hand. For a crawler that does not run scripts, though, the result is the same: it receives the challenge page instead of your content.
Can I use the no-JavaScript challenge so crawlers get through?
The metarefresh challenge works without JavaScript, but its docs page says it is not enabled by default and warns about a higher false-positive rate with headless Chrome. We have not verified how any search crawler treats it, so allowing by verified identity is the more predictable route.
If I allow a crawler in Anubis, will it appear in answers?
Allowing it only lets the crawler read your pages. Whether a page is used in a search result or an AI answer is up to the operator and depends on many things you do not control.
Check what crawlers see on your site
The free check reads one page the way AI crawlers do, with no account: robots.txt rules for each named AI crawler, robots meta tags and headers, and whether the main text is in the raw HTML. It runs from Crawlito's servers, so IP-based allow rules treat it differently from the real crawlers, and your logs remain the evidence. If you want the whole site checked, an audit covers up to 500 pages with evidence and prioritized fixes.
Filed under CDN and bot protection. Topics: bot-protection, ai-crawlers, waf, javascript, troubleshooting, checklist