Cloudflare and AI crawlers: the settings that block them
Which Cloudflare settings block AI bots: AI crawler policies, AI Crawl Control, Bot Fight Mode, managed robots.txt and WAF rules, and how to allow the ones you want.
10 min read
6 guides
CDN and bot protection
Bot protection is meant to stop scrapers, but it often stops search crawlers too. People get through, while automated visitors receive a challenge page with nothing to read. These guides show how to keep protection on without hiding your site.
Cookie walls, age gates and pop-ups: what crawlers actually see
Cookie walls, age gates and pop-ups can replace your content in the HTML crawlers read. When interstitials are harmless, when they hide pages, and how to check.
9 min read
Geo blocking crawlers: how country and IP rules shut out AI
Geo blocking crawlers is easy to do by accident: most AI crawlers connect from US cloud data centers. How to find country and ASN blocks and exempt verified bots.
9 min read
Paywalls and AI crawlers: what gated content they can read
How hard, metered and lead-in paywalls and login walls decide what AI crawlers read, what isAccessibleForFree markup does, and how to choose what to expose.
9 min read
Rate limit AI crawlers without locking out AI search
How to rate limit AI crawlers safely: throttle with 429 and Retry-After, give each bot its own budget, cache to cut origin load, and watch for blocked search bots.
9 min read
WAF blocking crawlers: why a challenge page shows AI bots nothing
A WAF challenge page stops AI crawlers that don't run JavaScript. See what they receive, how to detect it with curl and how to allow verified bots safely.
10 min read