WAF blocking crawlers: why a challenge page shows AI bots nothing

When a web application firewall (WAF) challenges a request, it does not send your page. It sends a small HTML page with a script or a puzzle that a browser must solve before it gets the real content. AI crawlers such as OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot fetch HTML without running that script, so they receive the challenge page and nothing else. In a browser the site looks fine, while the crawler gets a short "security check" page, often with a 403 status on Cloudflare or a 202 on AWS WAF.

This is one of the causes covered in why ChatGPT can't see your website, and one of the hardest to spot, because nothing in robots.txt or your page source looks wrong.

How JS challenges, CAPTCHAs and managed challenges work

All three types hold the request at the edge and ask the client to prove it is a real browser. Only the proof differs.

Cloudflare describes its challenge pages this way: a non-interactive challenge needs nothing from the visitor except the JavaScript their browser processes, an interactive one needs the visitor to act, and a managed challenge lets Cloudflare choose based on the request's signals. After a pass, the browser holds a clearance cookie (cf_clearance on Cloudflare) and its next requests go through.

AWS WAF works the same way. Its CAPTCHA and Challenge actions check for a valid token. Without one, AWS WAF stops the request and returns an interstitial; once it is solved, the script updates the token and resubmits the original request. AWS notes that these scripts can only be handled properly by a client that expects HTML content.

Common triggers on Cloudflare include Bot Fight Mode, Super Bot Fight Mode with "Definitely automated" traffic set to Managed Challenge, custom rules and rate limiting rules with a challenge action (by country, ASN or path), and I'm Under Attack mode, which challenges every visitor. On AWS WAF, the targeted Bot Control rules and your own rules with a CAPTCHA or Challenge action do the same.

Why AI crawlers fail every challenge

A crawler is an HTTP client that requests a URL and reads the response. It does not open a browser tab. Three things stop it:

1. It does not run JavaScript. Vercel and MERJ's December 2024 analysis found that none of the major AI crawlers it measured rendered JavaScript at the time, including OpenAI's crawlers, ClaudeBot and PerplexityBot. For more detail, see do AI crawlers run JavaScript?
2. It does not solve puzzles. Anthropic states in its crawler documentation that it will not attempt to bypass CAPTCHAs on the sites it crawls.
3. It is meant to fail. Challenges exist to stop automated clients. A crawler that passed them reliably would be the kind of bot they are built to catch.

The result is that the crawler records whatever the challenge page contains. Operators do not document what they do with such a response, but none of your content is in it, so there is nothing to index, quote or link to. OpenAI makes the dependency explicit: to appear in ChatGPT search, it recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges. A correct robots.txt does not help if the WAF challenges those IPs.

What the crawler actually receives

Cloudflare

Cloudflare adds one header to every challenge page response, whatever the type. Its detection guide says the header is cf-mitigated, its only valid value is challenge, and the response is text/html whatever resource was requested, even for an image or a JSON endpoint.

Cloudflare's documentation does not promise a status code. In the tests run for this article the challenge responses came back as 403, which is common, but do not rely on the status alone. Here are the headers a GPTBot user agent received from one Cloudflare-protected site, trimmed:

HTTP/2 403
content-type: text/html; charset=UTF-8
content-length: 27613
cf-mitigated: challenge
server: cloudflare
x-content-type-options: nosniff

Most of those 27 KB are inline script. Strip it and the visible text was one line: a security check message ending in "Enable JavaScript and cookies to continue". That line is everything the crawler learned about the page.

Two cautions. cf-mitigated: challenge marks challenges only: a WAF Block rule or an AI Crawl Control block sends its own response without it (Cloudflare's Block action returns 403 for most features and 429 for rate limiting rules). And a 403 with no header can still be Cloudflare, so read the body and the server header as well.

AWS WAF and others

AWS WAF uses its own header and different status codes, according to its action reference:

The AWS 202 is worth noticing. It is a 2xx status, so a monitoring tool that only checks for "2xx means fine" reports a healthy page while crawlers receive an interstitial. Other CDNs and WAFs have their own markers; when in doubt, compare the body a crawler receives with the body a browser receives.

How to detect a WAF challenge with curl

Fetch the page several times with different user agents and compare the status, the marker headers and the amount of text. Save this as waf-check.sh and run it with bash waf-check.sh https://example.com/pricing. It needs only bash, curl and perl, which macOS and most Linux systems include. It sends Accept: text/html because AWS WAF only returns the interstitial body to requests that accept HTML:

url="${1:?give a URL}"
body=$(mktemp)
while IFS='|' read -r name ua; do
  headers=$(curl -sSL -o "$body" -D - -A "$ua" -H 'Accept: text/html' --max-time 20 "$url")
  status=$(printf '%s\n' "$headers" | awk 'toupper($1) ~ /^HTTP\// {code=$2} END {print code}')
  marker=$(printf '%s\n' "$headers" | grep -iE '^(cf-mitigated|x-amzn-waf-action):' | tr -d '\r' | tr '\n' ' ')
  words=$(perl -0777 -pe 's/<(script|style)\b.*?<\/\1>//gis; s/<[^>]+>/ /g' "$body" | wc -w | tr -d ' ')
  title=$(perl -0777 -ne 'print $1 if /<title[^>]*>\s*(.*?)\s*<\/title>/is' "$body" | cut -c1-40)
  printf '%-15s %-4s words=%-6s %-28s %s\n' "$name" "$status" "$words" "$marker" "$title"
done <<'EOF'
browser|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
OAI-SearchBot|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
GPTBot|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
ChatGPT-User|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
PerplexityBot|Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Perplexity-User|Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
EOF
rm -f "$body"

The user-agent strings are the ones OpenAI and Perplexity document at the time of writing; version numbers change, so WAF rules should match the token, not the full string. Anthropic does not publish full strings for its bots, so if you add ClaudeBot, a string that contains the ClaudeBot token is only an approximation.

Here is output from one site behind a Cloudflare challenge, captured during research for this article, with the site's name removed from the titles:

browser         403  words=23     cf-mitigated: challenge      Security Check
OAI-SearchBot   403  words=23     cf-mitigated: challenge      Security Check
GPTBot          403  words=23     cf-mitigated: challenge      Security Check
ChatGPT-User    403  words=23     cf-mitigated: challenge      Security Check
PerplexityBot   200  words=1289                                Job Search
Perplexity-User 403  words=23     cf-mitigated: challenge      Security Check

How to read the result

• 403, 202 or 405 plus a marker header, and only a few dozen words: a challenge. This is what the crawler gets when the same rule matches it.
• 200 and a normal word count: the rule did not match this request. Compare the count with the browser row.
• A crawler row that passes while the browser row is challenged, like PerplexityBot above: the site probably has an allow rule that trusts the user-agent string. Your curl request is not from Perplexity, so that rule also lets through anyone who copies the string. See how to verify a real AI crawler.

The limit of this test is the IP address. Your request comes from your laptop or server, not from OpenAI's or Perplexity's networks. Rules that look at the user agent show up in curl. Rules that depend on IP reputation, verified-bot status or country do not, and they can treat the real crawler better or worse than your test. If you can, run the script from a cloud server as well as from your own connection, because many challenges target data-centre IPs, and confirm with the WAF's own logs. See your page as an AI crawler, with curl covers the wider set of curl checks.

Confirm in the WAF event log

• Cloudflare: open Security Events for the zone and filter by user agent (for example "contains GPTBot") and by action. Look for Managed Challenge, Non-Interactive Challenge (formerly JS Challenge) or Interactive Challenge events, and note which rule or product (Bot Fight Mode, Super Bot Fight Mode, a custom rule) issued them.
• AWS WAF: query your WAF logs for requests whose terminating action is CAPTCHA or Challenge and whose labels include awswaf:managed:aws:bot-control:bot:verified. Those are verified bots being challenged, usually by one of your own rules. Also look for blocks by the Bot Control CategoryAI rule, covered below.

How to allow verified bots safely

The goal is narrow: let the real crawlers you want through the challenge, and keep challenging everyone else, including anything that only claims to be a crawler.

On Cloudflare

Cloudflare tracks known, verified bots. The cf.client.bot field indicates a request from a known good bot or crawler, and cf.verified_bot_category gives its category. The verified bot categories that work in custom rules at the time of writing include AI Crawler, AI Search, AI Assistant and Search Engine Crawler. Cloudflare now also classifies bots by behavior and keeps these legacy categories for compatibility, so check which category each bot you care about falls in before you rely on it.

1. Exempt verified bots from your own challenge rules. Add a condition to each custom rule that uses a challenge action:

   (ip.src.country in {"CN" "RU"} or http.request.uri.path contains "/search") and not cf.client.bot

2. Let AI search crawlers past Super Bot Fight Mode. On Pro and higher, check that the Verified bots setting is Allow. To exempt chosen categories from the rest of Super Bot Fight Mode, create a custom rule with the Skip action, choose All Super Bot Fight Mode rules (and All rate limiting rules if they apply), and match:

   cf.verified_bot_category in {"AI Search" "Search Engine Crawler"}

   Add "AI Assistant" if you want user-triggered fetches to get through too. Training crawlers ("AI Crawler") are a separate choice; allowing AI search while blocking AI training explains the trade-off.

3. Check Bot Fight Mode. On the Free plan, Bot Fight Mode cannot be skipped with WAF custom rules or Page Rules, because it does not run on the Ruleset Engine. Perplexity points this out in its WAF guidance. If Bot Fight Mode is challenging the crawlers you want, your choices are to turn it off or to move to a plan with Super Bot Fight Mode, where verified bots can be allowed. Cloudflare's AI crawler settings walks through the rest of the dashboard, including AI Crawl Control and the AI Crawler and Agent access policies, which block rather than challenge.

On AWS WAF

The Bot Control rule group labels bots it can verify with awswaf:managed:aws:bot-control:bot:verified and does not block them, with one exception that matters here: the CategoryAI rule applies its Block action to AI bots whether they are verified or not. That is a block, not a challenge, but the crawler gets nothing either way. To let selected AI crawlers in:

1. Override the CategoryAI rule's action to Count, so it labels requests instead of blocking them.
2. After the rule group, add a rule that blocks requests carrying awswaf:managed:aws:bot-control:bot:category:ai unless they also carry the verified label and the bot:name: label of a crawler you want. AWS does not list the name values, so read them from the labels in your logs first.

The targeted rules (names starting with TGT_), which issue most of Bot Control's challenges and CAPTCHAs, only inspect requests that are not from verified bots.

On any other WAF or server

Match the user-agent token and the source IP together, as Perplexity recommends, and fetch the IP lists automatically instead of pasting them in. A starting point for three of OpenAI's agents, using jq (OAI-AdsBot has its own list at adsbot.json):

for f in searchbot gptbot chatgpt-user; do
  curl -s "https://openai.com/$f.json" | jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix'
done | sort -u > openai-ips.txt

Load the result into your WAF's IP list on a schedule. Anthropic (bots.json) and Perplexity (perplexitybot.json and perplexity-user.json) publish files in the same format.

Two things to avoid: allowing by user agent alone, which lets any scraper through with a copied string, and switching the WAF off for the whole site. Neither is needed.

Frequently asked questions

Does a WAF challenge hurt my Google rankings?

It can, if the challenge catches Googlebot, because Google cannot crawl a page it only sees as a challenge. Cloudflare and AWS both treat Googlebot as a verified bot, but your own custom challenge rules can still catch it, which is why the not cf.client.bot condition is worth adding to every challenge rule.

Is a challenge better than a block for AI crawlers?

No. Neither delivers your content. A block at least states intent clearly; a challenge aimed at humans that also catches crawlers is usually an accident.

Will robots.txt fix a challenge?

No. robots.txt tells crawlers what they may fetch, and the WAF answers before the request reaches your server. If the WAF challenges the request, the crawler never gets your content, whatever robots.txt says, and if it also challenges /robots.txt, the crawler cannot read your rules either.

How long until crawlers see my page after I fix the rule?

The fixed rule applies to requests that arrive after the change is deployed. When each crawler comes back to the page depends on its own schedule, which operators do not publish.

Check what crawlers get from your site

Run the curl script above on your homepage and one deep page, then check your WAF events for the bots you care about. As a next step, the free check tests one public page with no account needed. For a whole site, the audit runs deterministic checks on up to 500 pages, with human review and a PDF of the evidence and prioritized fixes.