Skip to content

Do CAPTCHAs block Google and AI crawlers? Where to put them

reCAPTCHA, hCaptcha and Cloudflare Turnstile belong on forms, not page loads. How a CAPTCHA gate hides content from crawlers and how to check your pages.

By Crawlito · 6 min read

In short

  • Crawlers cannot solve CAPTCHAs, so a CAPTCHA shown before page content means crawlers receive the challenge instead of the page.
  • reCAPTCHA, hCaptcha and Cloudflare Turnstile are designed to protect actions such as form submissions, and each requires a server-side verification call to work.
  • A widget embedded in a form does not hide the rest of the page; the risk comes from full-page interstitials and challenge rules that gate every URL.
  • Cloudflare marks challenge responses with a cf-mitigated: challenge header, which makes them easy to detect with curl.
  • Protect the endpoint that receives the form with server-side verification instead of gating the page that displays it.
On this page

Yes, a CAPTCHA blocks crawlers if it stands between them and your content. Googlebot, Bingbot and AI crawlers such as GPTBot or ClaudeBot do not solve challenges, so a page that shows a CAPTCHA before its content is, to them, a page whose content is the CAPTCHA. A CAPTCHA widget inside a contact or signup form is different: the rest of the page is still in the HTML, and crawlers never submit forms anyway.

So the rule is simple: put CAPTCHAs on actions, not on page loads. This guide explains how reCAPTCHA, hCaptcha and Cloudflare Turnstile are meant to be used, the setups that accidentally gate content, and how to check whether your pages are affected.

How CAPTCHA services are designed to be used

The three common services all work the same way at the level that matters here: a script on the page produces a token, the browser sends the token along with the form, and your server checks the token with the provider before accepting the submission.

ServiceWhere the provider says to use itServer-side check
Google reCAPTCHA v3On actions such as login or form submission; can also run in the page background for analyticsYour server verifies the token and reads a score from 0.0 to 1.0
hCaptchaAn h-captcha container placed inside an HTML formPOST to https://api.hcaptcha.com/siteverify
Cloudflare TurnstileA widget embedded on your page, with or without Cloudflare's CDNPOST to https://challenges.cloudflare.com/turnstile/v0/siteverify

A few details from the providers' own documentation:

  • reCAPTCHA v3 "will never interrupt your users". It returns a score, where 1.0 is very likely a good interaction and 0.0 very likely a bot, and your code decides what to do. Tokens expire after two minutes, so Google suggests calling execute when the user takes the action, not on page load.
  • hCaptcha's documentation tells you to include the empty .h-captcha container inside an HTML form, and to read the h-captcha-response parameter on your server.
  • Cloudflare Turnstile runs non-interactive JavaScript challenges and usually shows the visitor no puzzle. Cloudflare's server-side validation guide is explicit: "You must call the Siteverify API to complete your Turnstile implementation." Tokens are valid for 300 seconds and intended for single use.

The common thread is that none of these is meant to decide whether someone may read a page. They decide whether a submission should be trusted.

Which setups hide content from crawlers

A widget in a form is safe for crawling. The problems start when a challenge runs before the content is delivered.

Full-page CAPTCHA interstitials

Some plugins and custom scripts show a "verify you are human" screen on the first visit and load the real page only after it is passed. If the content arrives only after the challenge, crawlers get the interstitial HTML. Check whether the real content is in the initial HTML response or loaded after verification.

Challenge rules on every URL

WAF and bot-management products can issue a JavaScript or interactive challenge for whole paths or whole sites. These are not CAPTCHA widgets you placed, but they have the same effect: the crawler receives a challenge page, often with a 403 or 503 status, instead of your content. Our guide to WAF challenge pages covers this in depth, and Cloudflare's AI crawler settings explains the switches that apply site-wide.

Content rendered by the CAPTCHA callback

A subtler version: the page loads normally, but a script only fetches and inserts the article, prices or download links after the CAPTCHA callback fires. The HTML a crawler receives is a shell with a widget. This matters most for crawlers that do not run JavaScript at all, which our article on whether AI crawlers run JavaScript discusses.

Search, filter and "load more" forms

A CAPTCHA on your site search box is fine, and usually a good idea, because internal search results are not pages you want crawled anyway. The problem appears when ordinary navigation runs through a form: a store locator that needs a CAPTCHA before listing branches, a "load more" button that posts a protected request, or a filter panel that is the only route to category pages. Crawlers follow links, not form submissions, so any page reachable only through a protected form is effectively unreachable. Give those pages plain links as well.

Gating documents and downloads

PDFs, spec sheets and price lists behind a CAPTCHA are invisible to crawlers. If you want them findable, link them directly and protect only what truly needs protecting.

How to check whether your content pages are gated

Start by fetching a page the way a non-rendering crawler does:

bash
curl -s -A "GPTBot/1.1" https://www.example.com/guides/setup | grep -ciE "captcha|turnstile|cf-chl|verify you are human"

A non-zero count on a content page is worth a closer look. Then look at the status code and headers:

bash
curl -sI -A "GPTBot/1.1" https://www.example.com/guides/setup | grep -iE "^HTTP|cf-mitigated|content-length"

Cloudflare documents that challenge responses carry a cf-mitigated: challenge header. If you see it, the page is gated by a challenge rule, not by a widget.

Finally, confirm the content itself is in the HTML. Pick a sentence from the middle of the page and search for it:

bash
curl -s -A "GPTBot/1.1" https://www.example.com/guides/setup | grep -c "a sentence from the page body"

Zero means the text is not in the response a crawler gets. Our walkthrough on seeing your page as an AI crawler with curl has more checks along these lines.

Use this checklist when you read the results:

ResultLikely causeNext step
Content present, widget markup in a formForm CAPTCHA, working as intendedNothing
Challenge page, cf-mitigated: challengeWAF or bot rule on the pathExclude content paths, or allow verified bots
200 with an interstitial and no contentFull-page CAPTCHA pluginMove the CAPTCHA to forms only
Content missing, widget present, no challenge headerContent loaded after the CAPTCHA callbackServer-render the content

Requests from your machine do not carry a real crawler's IP address, so security products that score IPs may treat the real crawler differently. Your CDN or server logs show what crawlers actually received.

How to fix it: protect the action, not the page

  1. Remove page-level CAPTCHAs from content URLs. Articles, product pages, docs and pricing should load without a challenge.
  2. Keep the widget in the form, and verify the token on the server that receives the submission. The protection lives at the endpoint, so the page that displays the form can be public.
  3. Scope WAF challenge rules to sensitive paths such as /login, /wp-login.php, /checkout or /api/, and exclude /robots.txt and your sitemaps.
  4. Allow verified search and AI crawlers you want to reach the site, using your provider's verified-bot features rather than user-agent strings alone.
  5. Use rate limits for abuse on public pages. If scrapers are the concern, per-IP limits hurt them more than they hurt a crawler that fetches a page every few seconds. See rate limiting AI crawlers.

On Turnstile specifically, adding the widget to a page does not by itself challenge crawlers: it runs only in browsers that execute the script, and it protects nothing until your server calls Siteverify. A crawler that reads the HTML just sees an empty widget container next to your content.

Frequently asked questions

Does reCAPTCHA v3 on every page hurt SEO?

reCAPTCHA v3 does not interrupt visitors or crawlers; it only returns a score to your code. It hurts crawling only if your code uses a low score to hide content or block the page. Do not gate page content on the score.

Can Googlebot solve a CAPTCHA?

Neither Google nor the AI crawler operators describe their crawlers as solving CAPTCHAs, so treat a challenge as a block. If it stands between the crawler and the content, the crawler receives the challenge page.

Is Cloudflare Turnstile the same as a Cloudflare challenge page?

They share Cloudflare's challenge platform, but they are used differently. Turnstile is a widget you place in your own page, usually a form, while a challenge page replaces the response for a whole request when a WAF or bot rule fires.

Will a CAPTCHA on my contact form stop crawlers indexing the contact page?

No. The page's text, address and phone number are still in the HTML. Crawlers do not submit forms, so the CAPTCHA never comes into play for them.

Check your important pages

CAPTCHA gates are easy to add by accident, especially through security plugins and WAF presets. Run the free check on a page you care about to see whether crawlers get your content or a challenge.

Filed under CDN and bot protection. Topics: bot-protection, waf, cloudflare, ai-crawlers, troubleshooting