Cloudflare Disallow AI Training vs Block: effect on Googlebot
Cloudflare's 15 September 2026 change means Block can now stop Googlebot, Bingbot and Applebot. Disallow AI Training keeps search. Here is how to check.
By Crawlito · 9 min read
In short
- Per Cloudflare's 15 September 2026 announcement, Block and Block on pages with ads now apply to mixed-use crawlers such as Googlebot, Bingbot and Applebot, so they can affect search as well as training.
- Disallow AI Training is the setting Cloudflare points to for stopping training while keeping search: it publishes a no-training preference in robots.txt and keeps Accountable mixed-use crawlers allowed for search.
- Cloudflare says existing zones migrate automatically: a legacy Block AI Bots setting that was on maps to Search Allow and Training Disallow AI Training, and a Training Block set through the granular controls also becomes Disallow AI Training.
- Disallow AI Training does not yet send a no-training signal to Bing through robots.txt: Cloudflare says Microsoft's mechanism is targeted for early 2027, and Bing's current option is the NOARCHIVE meta tag.
- Cloudflare's documentation does not describe how a block on Googlebot appears in your logs, so confirm with Security Events and the Crawl stats report in Search Console.
On this page
- What changed in Cloudflare's AI crawler settings on 15 September 2026
- Block vs Disallow AI Training: which crawlers each one stops
- Why Googlebot, Bingbot and Applebot are called Accountable
- How to check which setting your zone uses now
- How to stop AI training without losing Googlebot
- Why Bing and Google-Extended opt-outs still need their own signals
- How to confirm Googlebot is not getting 403s
- Frequently asked questions
- Check what your site returns
Choosing Block for training crawlers in Cloudflare can now stop Googlebot, Bingbot and Applebot as well, because Cloudflare changed on 15 September 2026 how Block treats crawlers that do both search and training. Disallow AI Training is the setting Cloudflare recommends for opting out of training while keeping search crawling. Cloudflare says most existing zones carry over without a change, but if anyone on your team picked Block by hand, check it.
This post covers what changed, how the settings differ, how to see which one your zone uses, and how to confirm Googlebot is getting through. Setting names and behavior come from Cloudflare's announcement and its docs at the time of writing, and Cloudflare may change them again.
What changed in Cloudflare's AI crawler settings on 15 September 2026
Cloudflare sorts bot behavior into three controls: Search (crawling to build a search index), Training (crawling to train or fine-tune a model) and Agent (tools acting for a person in real time). It also introduced the term mixed-use crawler for one crawler that does both search and training. Cloudflare names Applebot, Bingbot and Googlebot as mixed-use.
Before 15 September, Cloudflare says, the Block and Block on pages with ads actions did not apply to mixed-use crawlers, because blocking them could also affect search discoverability. From that date they apply to all training crawlers, mixed-use ones included, and Cloudflare says either action "impacts search as well as training."
Two other changes landed the same day. The "Block AI Bots" setting is deprecated in favor of the Search, Training and Agent controls, and Managed Robots.txt is deprecated in favor of Bot Preference Sync. Cloudflare says the changes apply to Bot Management and AI Crawl Control, on all plans.
Block vs Disallow AI Training: which crawlers each one stops
Here is how Cloudflare describes each domain-level action.
| Action | What it does | Googlebot, Bingbot, Applebot |
|---|---|---|
| Allow | All crawlers allowed unless another setting or a WAF rule blocks them | Allowed |
| Disallow AI Training | Bot Preference Sync publishes a no-training preference in robots.txt; other training crawlers are blocked | Accountable mixed-use crawlers stay allowed for search |
| Block on pages with ads | Crawlers, mixed-use ones included, are blocked only on pages detected as serving ads | Blocked on those pages |
| Block | All crawlers, mixed-use ones included, are blocked | Blocked |
Disallow AI Training is available for Training only, not for Search or Agent. Cloudflare says there is no ad-page version of it, because the list of pages with ads is too large and changes too often to express in robots.txt. For Block on pages with ads, how Cloudflare detects an ad is its own call, so test pages rather than assuming.
The practical rule from Cloudflare: to stop training but keep search, choose Disallow AI Training. To remove mixed-use crawlers entirely, including from search, choose Block.
Why Googlebot, Bingbot and Applebot are called Accountable
Cloudflare's Accountable label is its own, not a standard from Google, Microsoft or Apple. It applies to operators that offer, or commit to offer, four things:
- An opt-out from AI training through robots.txt or a similar standard.
- An opt-out from AI summaries, set with the operator today and, Cloudflare says, through Cloudflare next year.
- URL-level visibility into which pages were made available for training, plus metrics on how content appears in search.
- Assurance that opting out of training does not affect traditional search results.
Cloudflare lists Applebot, Googlebot and Bingbot as Accountable, along with the training crawlers from Amazon, Anthropic, Meta and OpenAI. Those four operators run separate search and training crawlers, so a training block does not touch their search bots. Cloudflare also says some of this is still being delivered: URL-level tools for Applebot and Googlebot are not fully available yet, and AI summary controls are partial.
That is why Disallow AI Training leaves the three mixed-use crawlers alone for search: the opt-out for training happens through a separate signal, and each operator says using it does not change search. Google's crawler documentation says the Google-Extended token does not affect inclusion in Google Search and is not a ranking signal. Cloudflare relays similar statements from Apple (Applebot-Extended) and Microsoft (NOARCHIVE). See Google-Extended explained and Applebot-Extended for the details of each token.
How to check which setting your zone uses now
In the Cloudflare dashboard, open your zone, go to Security > Settings, filter by Bot traffic and open AI Crawler and Agent access policies, per Cloudflare's docs. Read the action next to Search, Training and Agent.
What you should see depends on your history, based on Cloudflare's migration rules:
| Your zone before | After 15 September |
|---|---|
| Legacy Block AI Bots off | Search Allow, Training Allow, Agent Allow |
| Legacy Block AI Bots set to Block, or Block on pages with ads | Search Allow, Training Disallow AI Training, Agent Block on pages with ads |
| Granular controls, Training Block or Block on pages with ads | Training becomes Disallow AI Training; Search and Agent keep their actions |
| Granular controls, Search set to Block | Unchanged: Search stays Block |
In the last row, Cloudflare's migration leaves Search unchanged, so a Search Block someone set by hand stays in place and is worth checking. Cloudflare says that if you want mixed-use crawlers gone entirely, you must now select Block yourself, so any Block on a live site is a deliberate choice somebody made.
Also read your live robots.txt, since Bot Preference Sync edits it:
curl -s https://example.com/robots.txt | head -40Look for Disallow: / groups and note which tokens they name. At the time of writing, the example in Cloudflare's managed robots.txt documentation names training tokens such as GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended, and does not name Googlebot or Bingbot. The list can change, so a Googlebot group with Disallow: / would be a red flag. For how the settings interact, see Cloudflare and AI crawlers: the settings that block them.
How to stop AI training without losing Googlebot
- Open AI Crawler and Agent access policies and set Search to Allow.
- Set Training to Disallow AI Training, not Block.
- Set Agent to the action you want. Cloudflare's presets use Allow for sites without ad monetization and Block on pages with ads for sites with it.
- Make sure Bot Preference Sync is on, since that is what publishes the preference.
- Save, then re-fetch robots.txt to see the result.
Cloudflare's presets for new domains follow the same idea: Search stays Allow for both ad and non-ad sites, and the ad-monetized preset sets Training to Disallow AI Training.
Cloudflare also notes that robots.txt alone cannot identify crawlers, work out their purpose or enforce compliance, and says its network enforcement fills that gap. If you run a site outside Cloudflare, the robots.txt version of this split uses the same idea with separate tokens.
Why Bing and Google-Extended opt-outs still need their own signals
Disallow AI Training does not do the whole job for every operator. What Cloudflare says about each:
- Bing: until Microsoft ships robots.txt support for a no-training preference, which Cloudflare says is targeted for early 2027, Disallow AI Training does not convey a no-training preference to Bing through robots.txt. Today, Cloudflare says Bing's options are the NOARCHIVE meta tag and the Block URLs or Content Removal tools in Bing Webmaster Tools. Microsoft says NOARCHIVE does not affect ranking. See Bingbot and AI search.
- Google: the training opt-out is a
Google-ExtendedDisallow rule in robots.txt. Google-Extended is not a separate crawler: Google's docs say it has no separate HTTP user agent string, so it is not a user agent you will look for in your logs. Cloudflare also says Google offers a toggle in its webmaster portal to exclude a site's content from generative search results. - Apple: the training opt-out is an
Applebot-ExtendedDisallow rule. Keeping content out of summaries is anosnippetdirective in the page HTML.
So after choosing Disallow AI Training, decide whether you also want these operator-specific signals, and set them yourself where they exist.
How to confirm Googlebot is not getting 403s
Cloudflare's announcement does not say how a block on Googlebot shows up, so check the evidence from both sides.
On Cloudflare: open your zone's Analytics page, select the Events tab and filter for Googlebot, Bingbot or Applebot by user agent. Cloudflare's security events documentation describes filtering by Action, Service, user agent, IP and path. Free plans can only view sampled logs, event data is kept for 31 days and results may be sampled, so use a narrow time window. Look at the Action and the Service columns on any event for a crawler you meant to allow.
On your server: search your access logs for Googlebot and look at the status codes. Before you trust the user agent, verify the IP. Google documents this in Verifying Googlebot: a reverse DNS lookup that ends in googlebot.com, google.com or googleusercontent.com, then a forward lookup that returns the same IP. Google also publishes IP range files such as common-crawlers.json for automatic matching.
In Search Console: open Settings > Crawl stats. The report is available for Domain properties and root-level URL-prefix properties only. Check Host status, then the response table: Google counts 200, 301, 302, 304 and similar redirect codes as good and calls other 4xx client errors bad. Click a response row for example URLs.
By request: you can test status and headers with a crawler's user agent.
curl -s -o /dev/null -D - -A "Googlebot/2.1 (+http://www.google.com/bot.html)" https://example.com/ | grep -iE "^HTTP|cf-mitigated"This request comes from your own IP, not Google's, so Cloudflare's verified-bot checks will not treat it as the real Googlebot. Use it for the status and headers your page returns, not as proof about Googlebot itself. Security Events and the Crawl stats report are better evidence.
Frequently asked questions
Will the legacy Block AI Bots setting now block Googlebot?
Not according to Cloudflare's migration rules. A zone that had the legacy setting on (Block, or Block on pages with ads) and never used the granular controls moves to Search Allow and Training Disallow AI Training, with Agent set to Block on pages with ads. Check your zone anyway, because a Block chosen by hand would behave differently.
Does Disallow AI Training stop Googlebot, Bingbot or Applebot from indexing my site?
Cloudflare says Accountable mixed-use crawlers remain allowed for search under Disallow AI Training. Google states that Google-Extended does not affect Search inclusion or ranking, and Cloudflare relays similar statements from Apple and Microsoft for their opt-out signals.
Does Disallow AI Training opt me out of Bing's AI training?
Not through robots.txt yet. Cloudflare says Microsoft's robots.txt mechanism is targeted for early 2027. In the meantime, Cloudflare names the NOARCHIVE meta tag and Bing's content removal tools as the current controls.
Can I still block these crawlers on purpose?
Yes. Choose Block if you want mixed-use crawlers gone entirely, search included. Cloudflare says that is now the effect of Block, so only do it if losing search access to those engines is what you want.
Is Disallow AI Training an enforced block?
Partly. It publishes a robots.txt preference, which is voluntary, and Cloudflare says it also blocks other training crawlers at the edge, except mixed-purpose ones. For mixed-use crawlers it relies on their operators honoring the signal.
Check what your site returns
The free check reads one page the way AI crawlers do, with no account: the robots.txt rules for the named AI crawlers, the status code and any challenge header. It runs from Crawlito's servers, not Google's, so it cannot show what Googlebot itself received. For that, use Security Events and the Crawl stats report. If you want the whole site checked, an audit covers up to 500 pages with evidence and prioritized fixes.
Filed under CDN and bot protection. Topics: cloudflare, bot-protection, googlebot, ai-crawlers, robots-txt, troubleshooting