Do CAPTCHAs block Google and AI crawlers? Where to put them
reCAPTCHA, hCaptcha and Cloudflare Turnstile belong on forms, not page loads. How a CAPTCHA gate hides content from crawlers and how to check your pages.
By Crawlito · 6 min read
In short
- Crawlers cannot solve CAPTCHAs, so a CAPTCHA shown before page content means crawlers receive the challenge instead of the page.
- reCAPTCHA, hCaptcha and Cloudflare Turnstile are designed to protect actions such as form submissions, and each requires a server-side verification call to work.
- A widget embedded in a form does not hide the rest of the page; the risk comes from full-page interstitials and challenge rules that gate every URL.
- Cloudflare marks challenge responses with a cf-mitigated: challenge header, which makes them easy to detect with curl.
- Protect the endpoint that receives the form with server-side verification instead of gating the page that displays it.
On this page
Yes, a CAPTCHA blocks crawlers if it stands between them and your content. Googlebot, Bingbot and AI crawlers such as GPTBot or ClaudeBot do not solve challenges, so a page that shows a CAPTCHA before its content is, to them, a page whose content is the CAPTCHA. A CAPTCHA widget inside a contact or signup form is different: the rest of the page is still in the HTML, and crawlers never submit forms anyway.
So the rule is simple: put CAPTCHAs on actions, not on page loads. This guide explains how reCAPTCHA, hCaptcha and Cloudflare Turnstile are meant to be used, the setups that accidentally gate content, and how to check whether your pages are affected.
How CAPTCHA services are designed to be used
The three common services all work the same way at the level that matters here: a script on the page produces a token, the browser sends the token along with the form, and your server checks the token with the provider before accepting the submission.
| Service | Where the provider says to use it | Server-side check |
|---|---|---|
| Google reCAPTCHA v3 | On actions such as login or form submission; can also run in the page background for analytics | Your server verifies the token and reads a score from 0.0 to 1.0 |
| hCaptcha | An h-captcha container placed inside an HTML form | POST to https://api.hcaptcha.com/siteverify |
| Cloudflare Turnstile | A widget embedded on your page, with or without Cloudflare's CDN | POST to https://challenges.cloudflare.com/turnstile/v0/siteverify |
A few details from the providers' own documentation:
- reCAPTCHA v3 "will never interrupt your users". It returns a score, where 1.0 is very likely a good interaction and 0.0 very likely a bot, and your code decides what to do. Tokens expire after two minutes, so Google suggests calling
executewhen the user takes the action, not on page load. - hCaptcha's documentation tells you to include the empty
.h-captchacontainer inside an HTML form, and to read theh-captcha-responseparameter on your server. - Cloudflare Turnstile runs non-interactive JavaScript challenges and usually shows the visitor no puzzle. Cloudflare's server-side validation guide is explicit: "You must call the Siteverify API to complete your Turnstile implementation." Tokens are valid for 300 seconds and intended for single use.
The common thread is that none of these is meant to decide whether someone may read a page. They decide whether a submission should be trusted.
Which setups hide content from crawlers
A widget in a form is safe for crawling. The problems start when a challenge runs before the content is delivered.
Full-page CAPTCHA interstitials
Some plugins and custom scripts show a "verify you are human" screen on the first visit and load the real page only after it is passed. If the content arrives only after the challenge, crawlers get the interstitial HTML. Check whether the real content is in the initial HTML response or loaded after verification.
Challenge rules on every URL
WAF and bot-management products can issue a JavaScript or interactive challenge for whole paths or whole sites. These are not CAPTCHA widgets you placed, but they have the same effect: the crawler receives a challenge page, often with a 403 or 503 status, instead of your content. Our guide to WAF challenge pages covers this in depth, and Cloudflare's AI crawler settings explains the switches that apply site-wide.
Content rendered by the CAPTCHA callback
A subtler version: the page loads normally, but a script only fetches and inserts the article, prices or download links after the CAPTCHA callback fires. The HTML a crawler receives is a shell with a widget. This matters most for crawlers that do not run JavaScript at all, which our article on whether AI crawlers run JavaScript discusses.
Search, filter and "load more" forms
A CAPTCHA on your site search box is fine, and usually a good idea, because internal search results are not pages you want crawled anyway. The problem appears when ordinary navigation runs through a form: a store locator that needs a CAPTCHA before listing branches, a "load more" button that posts a protected request, or a filter panel that is the only route to category pages. Crawlers follow links, not form submissions, so any page reachable only through a protected form is effectively unreachable. Give those pages plain links as well.
Gating documents and downloads
PDFs, spec sheets and price lists behind a CAPTCHA are invisible to crawlers. If you want them findable, link them directly and protect only what truly needs protecting.
How to check whether your content pages are gated
Start by fetching a page the way a non-rendering crawler does:
curl -s -A "GPTBot/1.1" https://www.example.com/guides/setup | grep -ciE "captcha|turnstile|cf-chl|verify you are human"A non-zero count on a content page is worth a closer look. Then look at the status code and headers:
curl -sI -A "GPTBot/1.1" https://www.example.com/guides/setup | grep -iE "^HTTP|cf-mitigated|content-length"Cloudflare documents that challenge responses carry a cf-mitigated: challenge header. If you see it, the page is gated by a challenge rule, not by a widget.
Finally, confirm the content itself is in the HTML. Pick a sentence from the middle of the page and search for it:
curl -s -A "GPTBot/1.1" https://www.example.com/guides/setup | grep -c "a sentence from the page body"Zero means the text is not in the response a crawler gets. Our walkthrough on seeing your page as an AI crawler with curl has more checks along these lines.
Use this checklist when you read the results:
| Result | Likely cause | Next step |
|---|---|---|
| Content present, widget markup in a form | Form CAPTCHA, working as intended | Nothing |
Challenge page, cf-mitigated: challenge | WAF or bot rule on the path | Exclude content paths, or allow verified bots |
| 200 with an interstitial and no content | Full-page CAPTCHA plugin | Move the CAPTCHA to forms only |
| Content missing, widget present, no challenge header | Content loaded after the CAPTCHA callback | Server-render the content |
Requests from your machine do not carry a real crawler's IP address, so security products that score IPs may treat the real crawler differently. Your CDN or server logs show what crawlers actually received.
How to fix it: protect the action, not the page
- Remove page-level CAPTCHAs from content URLs. Articles, product pages, docs and pricing should load without a challenge.
- Keep the widget in the form, and verify the token on the server that receives the submission. The protection lives at the endpoint, so the page that displays the form can be public.
- Scope WAF challenge rules to sensitive paths such as
/login,/wp-login.php,/checkoutor/api/, and exclude/robots.txtand your sitemaps. - Allow verified search and AI crawlers you want to reach the site, using your provider's verified-bot features rather than user-agent strings alone.
- Use rate limits for abuse on public pages. If scrapers are the concern, per-IP limits hurt them more than they hurt a crawler that fetches a page every few seconds. See rate limiting AI crawlers.
On Turnstile specifically, adding the widget to a page does not by itself challenge crawlers: it runs only in browsers that execute the script, and it protects nothing until your server calls Siteverify. A crawler that reads the HTML just sees an empty widget container next to your content.
Frequently asked questions
Does reCAPTCHA v3 on every page hurt SEO?
reCAPTCHA v3 does not interrupt visitors or crawlers; it only returns a score to your code. It hurts crawling only if your code uses a low score to hide content or block the page. Do not gate page content on the score.
Can Googlebot solve a CAPTCHA?
Neither Google nor the AI crawler operators describe their crawlers as solving CAPTCHAs, so treat a challenge as a block. If it stands between the crawler and the content, the crawler receives the challenge page.
Is Cloudflare Turnstile the same as a Cloudflare challenge page?
They share Cloudflare's challenge platform, but they are used differently. Turnstile is a widget you place in your own page, usually a form, while a challenge page replaces the response for a whole request when a WAF or bot rule fires.
Will a CAPTCHA on my contact form stop crawlers indexing the contact page?
No. The page's text, address and phone number are still in the HTML. Crawlers do not submit forms, so the CAPTCHA never comes into play for them.
Check your important pages
CAPTCHA gates are easy to add by accident, especially through security plugins and WAF presets. Run the free check on a page you care about to see whether crawlers get your content or a challenge.
Filed under CDN and bot protection. Topics: bot-protection, waf, cloudflare, ai-crawlers, troubleshooting