Cloudflare and AI crawlers: the settings that block them

Cloudflare can block AI bots in at least six places: the AI Crawler and Agent access policies, AI Crawl Control, Bot Fight Mode or Super Bot Fight Mode, the managed robots.txt, your own WAF custom rules and rate limiting rules. Only the first two and the managed robots.txt are built around the split between search and training crawlers, and Bot Fight Mode cannot be overridden by any WAF rule. To let AI search crawlers in while keeping training crawlers out, set the Search policy to Allow, the Training policy to Block or Disallow, and check that bot protection is not challenging the search crawlers you allowed.

Feature names and defaults below are as documented by Cloudflare at the time of writing. They have changed several times, so guides that mention "AI Audit" or a single "Block AI bots" switch are out of date.

Which Cloudflare settings affect AI crawlers

Sources: Cloudflare's docs on AI crawler policies, AI Crawl Control, Bot Fight Mode, Super Bot Fight Mode, managed robots.txt and rate limiting rules.

Before you change anything, decide which crawlers you want. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether your pages can appear in AI search answers. Training crawlers such as GPTBot and ClaudeBot only collect content that may be used for model training. Training, search and user-triggered crawlers explains the split for each operator.

AI Crawler and Agent access policies

This is the setting that used to be called "Block AI bots", then "Configure AI bot policies". It lives under Security > Settings: filter by Bot traffic and open AI Crawler and Agent access policies. Instead of one switch, it sets an action for three behaviors:

• Search: crawlers that collect or index content to answer questions later.
• Agent: automated activity acting in real time for a person, such as chat fetchers and browser agents. Personal Agents inherit this policy unless you set them separately, and Cloudflare recommends Allow for them.
• Training: crawlers that take content to train or fine-tune a model.

Each behavior can be set to Block, Block on pages with ads or Allow (do not block). Training also has Disallow, which expresses the opt-out through robots.txt. Cloudflare says that with Bot Preference Sync on, Disallow also blocks training crawlers at the edge, but mixed-purpose crawlers used for both search and training are excluded from that edge block.

Three details matter for AI search:

1. Each policy covers verified bots with that behavior and matching unverified bots. A Block on Search is not limited to one operator.
2. Cloudflare counts mixed-purpose crawlers, used for both search and training, under Training. A Training Block therefore stops them too, while a Training Disallow does not block them at the edge. Check Security Analytics to see which crawlers were actually blocked.
3. New domains default to Allow for all three. The often repeated claim that Cloudflare blocks AI crawlers by default does not match the current docs. They can be set during onboarding or changed later, so read your zone's current actions.

To allow AI search but not training: set Search to Allow, Agent to Allow, and Training to Block or Disallow. How to allow AI search but block AI training covers the matching robots.txt for sites that are not on Cloudflare.

AI Crawl Control: per-crawler allow and block

AI Crawl Control is the current name of what Cloudflare used to call AI Audit. It is listed as available on all plans and has a table of AI crawlers with their category, request count, robots.txt violations and an Allow or Block action for each one.

Two things to know when reading it:

• On the Free plan, it identifies crawlers by user-agent string only. Higher plans can use Bot Management detection. Free-plan numbers are claimed identity, not verified identity, which matters because user agents are easy to fake. How to verify a real AI crawler shows how to check.
• Its blocks run as WAF custom rules, which Cloudflare evaluates before its bot products. On paid plans, once at least one crawler is blocked, you can set the block response to 403 Forbidden or 402 Payment Required with a plain-text body.

Pay per crawl, which answers unpaid crawlers with 402 Payment Required, is part of AI Crawl Control but is still a beta you apply for.

To check it: open AI Crawl Control and look at the Action column for OAI-SearchBot, Claude-SearchBot, PerplexityBot and the others you care about. Cloudflare says the request count includes unsuccessful requests, which can come from any rule or response error. A high count for a crawler you allow can point to a different setting or an origin error.

Bot Fight Mode and Super Bot Fight Mode

These are the settings most likely to block AI search crawlers by accident, because they are not aimed at AI.

Bot Fight Mode (Free plan)

Bot Fight Mode challenges traffic that matches known bot patterns with computationally expensive challenges, across the whole domain. The limit that catches people out: Cloudflare says it cannot be bypassed or skipped with WAF custom rules or Page Rules, because it does not run on the Ruleset Engine. Skip, Bypass and Allow actions have no effect on it. Perplexity's crawler docs make the same point.

Cloudflare's page does not say how Bot Fight Mode treats verified bots, so do not assume it lets them through. It does say Bot Fight Mode will not trigger if an IP Access rule matches the request first, but crawler IP lists change often, so that route needs a job that keeps the rules in sync.

To check it: go to Security > Analytics, open the Events tab and filter for requests whose user agent contains a crawler name. Requests that Bot Fight Mode challenged show "Bot Fight Mode" in the Service field. To fix it: turn the Bot Fight Mode toggle off under Security > Settings (filter by Bot traffic), or move to a plan with Super Bot Fight Mode, where exceptions work.

Super Bot Fight Mode (Pro and above)

Super Bot Fight Mode sorts traffic into three groups: Definitely automated, Likely automated and Verified bots, each with its own action. On Pro, the Likely automated group cannot be blocked or challenged; Business and Enterprise without Bot Management can. Make sure Verified bots is set to Allow if you want verified search crawlers through.

Unlike Bot Fight Mode, it respects WAF custom rules with the Skip action. A rule like this lets OpenAI's search crawler bypass it, but only when Cloudflare has verified the request:

(http.user_agent contains "OAI-SearchBot" and cf.client.bot)
or (http.user_agent contains "PerplexityBot" and cf.client.bot)
or (http.user_agent contains "Claude-SearchBot" and cf.client.bot)

Set the action to Skip so matching requests bypass the Super Bot Fight Mode phase. The cf.client.bot condition matters: without it, anyone who sends OAI-SearchBot in a header skips your bot protection. If a crawler is not in Cloudflare's verified list, match its published IP ranges instead, such as openai.com/searchbot.json, using an IP list that a scheduled job refreshes.

Managed robots.txt and Content Signals

The managed robots.txt feature is turned on with the Enable Bot Preference Sync toggle. Cloudflare's plan tables call it "Instruct AI bot traffic with robots.txt". It only adds rules when at least one AI policy is set to Block, Block on pages with ads or Disallow. When it does, Cloudflare places this block before your origin's own robots.txt:

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

The list follows Cloudflare's documentation at the time of writing. The main AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot) are not named, and Google-Extended does not affect Google Search. Some listed tokens are mixed-purpose: Meta says meta-externalagent crawls for AI training and to index content for its products. If your origin has no robots.txt, Cloudflare serves one with these managed rules.

The Content-signal line comes from Cloudflare's Content Signals Policy, not from RFC 9309, and does not set ai-input. The use=reference value is an experimental extension Cloudflare is testing. Cloudflare notes Google Search Console may report "Syntax not understood" for it and says it has seen no effect on crawling or SEO.

To check it: fetch the file the world sees, not the one in your repository:

curl -s https://example.com/robots.txt | head -40

Compliance with robots.txt is voluntary, as Cloudflare says itself. Rules there ask; only edge blocks enforce.

WAF custom rules and challenge actions

Your own custom rules run before Cloudflare's bot products, and many accidental blocks live there: an old scraper rule matching bot in the user agent catches every AI crawler.

The actions that stop a crawler are Block and the three challenges: Managed Challenge, Non-Interactive Challenge and Interactive Challenge (Cloudflare's action list). Crawlers generally cannot solve challenges, and Anthropic says it will not attempt to bypass CAPTCHAs. A challenged crawler reads an HTML challenge page, not your content. Why a WAF challenge page means AI crawlers read nothing goes into detail.

To find rules that hit AI crawlers, use the verified bot category field. This expression matches Cloudflare's legacy categories for AI and search bots:

(cf.verified_bot_category in {"AI Crawler" "AI Search" "AI Assistant" "Search Engine Crawler"})

Cloudflare keeps "AI Search" for backward compatibility; new search crawlers go under its newer Search behavior, which is not a WAF field. Use the expression in a temporary rule to see what matches, or add and not cf.client.bot to broad scraper rules so they leave verified bots alone. Put any Skip rule above rules that block or challenge.

To test from outside: request a page with a crawler's user agent and look at the status and headers:

curl -s -o /dev/null -D - \
  -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  https://example.com/ | grep -iE "^HTTP|cf-mitigated"

A cf-mitigated: challenge header means Cloudflare served a challenge page. It marks challenges only, not outright blocks, so a 403 without it can still be a Cloudflare block. Your request comes from your own IP, not the crawler's, so rules that verify IPs can treat the real bot differently. Security Analytics shows what the real crawler got.

Rate limiting rules

Rate limiting rules block or challenge a client once it passes a request threshold. The Free plan gets one rule, Pro two and Business five, and Free and Pro count requests by IP only. Cloudflare's own docs warn that applying rate limiting to verified bots might affect SEO.

A crawler fetching many pages from a few IPs can trip a limit set for humans. Exclude verified bots in the rule expression. Free-plan rules can use only the path and verified bot fields, so this works on every plan:

(http.request.uri.path contains "/blog/" and not cf.client.bot)

If a crawler's load is the real problem, a looser limit for that crawler alone beats blocking it. Rate limiting AI crawlers covers thresholds and 429 responses. AI Labyrinth, by contrast, is not a block at all: Cloudflare logs its events and does not block or challenge the request.

Frequently asked questions

Does Cloudflare block AI crawlers by default?

Not according to the current documentation: for new domains, the Search, Agent and Training policies all default to "Allow (do not block)". Bot Fight Mode and your own rules can still block or challenge crawlers, and those are separate settings.

Will blocking AI training bots on Cloudflare remove me from ChatGPT search?

Blocking GPTBot does not. OpenAI says GPTBot and OAI-SearchBot are controlled independently, and ChatGPT search depends on OAI-SearchBot. Setting Cloudflare's Search policy to Block, or a bot rule that challenges OAI-SearchBot, does affect it.

Why does my WAF allow rule not help GPTBot or PerplexityBot?

If you are on the Free plan with Bot Fight Mode on, no WAF custom rule can skip it. Turn Bot Fight Mode off, or use Super Bot Fight Mode on a paid plan, where a Skip rule works.

Is the managed robots.txt enough to stop training crawlers?

It only asks. Crawlers that honor robots.txt will stop, but compliance is voluntary. To enforce the opt-out, use the Training policy's Block action or a block in AI Crawl Control.

Check what crawlers actually get

The dashboard rarely shows the combined result of these settings. Run the free single-page check to see the status code, any challenge header and the robots.txt rule for each AI crawler on one page. For the whole site, a full audit runs the same checks across up to 500 pages and comes as a PDF with the evidence and prioritized fixes. If AI search tools still cannot reach you after this, why ChatGPT can't see your website works through the other common causes.