Cookie walls, age gates and pop-ups: what crawlers actually see

A cookie banner, age gate or pop-up only hurts crawlers when it replaces your content instead of covering it. If the page's text is in the raw HTML and the banner is drawn over it, a crawler reads the text and ignores the banner. If the server redirects to a consent page, or sends the article only after a "yes" has been stored in a cookie, the crawler gets the gate on every URL and nothing else.

The difference is invisible in a normal browser, because you clicked "Accept" months ago and the cookie is still there. This guide shows the patterns that withhold content, how to check what a crawler is actually handed, and how to keep the gate for people without hiding pages from machines.

Why crawlers never get past the gate

Interstitials assume a person who reads, decides and clicks. Crawlers do none of that.

• They do not click. No crawler presses "Accept", "I am over 18" or "No thanks". Whatever the page shows before that click is what the crawler records.
• Assume they carry no cookies. Google is explicit about its own renderer: in Fix Search-related JavaScript problems it says HTTP cookies, Local Storage and Session Storage are all cleared across page loads. OpenAI, Anthropic and Perplexity do not document cookie handling for their crawlers at the time of writing, so the safe assumption is that every request arrives as a first-time visitor.
• They decline permission prompts. The same Google page says to expect Googlebot to decline user permission requests, so nothing should depend on a visitor granting one.
• The big AI crawlers did not run JavaScript when measured. In a December 2024 analysis, Vercel and MERJ found that none of the major AI crawlers they measured, including OpenAI's crawlers, ClaudeBot and PerplexityBot, executed JavaScript. Googlebot does render JavaScript, so it is the exception. That cuts both ways: a banner drawn by a script is invisible to them, and so is content that a script only adds after consent.

So the question for any gate is simple: what does the raw HTML contain for a visitor with no cookies?

Harmless banners and harmful walls

Most consent and age prompts fall into one of five patterns. Only the first two are safe.

The safe patterns

When the gate is a layer on top of real content, crawlers read the content. If the banner markup is in the HTML, its text gets read too, which is a minor issue: put the banner after the main content in the source, or render it with a script, so "We use cookies to improve your experience" is not the first sentence of every page.

The harmful patterns

The last three all fail for the same reason: the server decides what to send based on a choice the crawler can never make.

The redirect is the worst. Every URL answers with a 302 to the same gate page, so a crawler sees one page for the whole site. Google's guidance on intrusive interstitials warns against exactly this. It says not to redirect incoming requests to a different page to collect consent, and that redirecting all URLs to a single page will remove all but that one page from search results, because Googlebot can only fetch that one page. The same logic applies to any crawler. Our guide to status codes and redirects covers how crawlers treat 302s in general.

A cookie wall can also trap crawlers in a loop. A common setup redirects to the gate, sets a cookie there, and redirects back. A client that does not store the cookie lands on the gate again, and again, until it gives up.

How to check what a crawler sees with curl

curl sends no cookies unless you ask it to, which makes it a fair stand-in for a first visit by a crawler. Start with the status codes and redirects:

curl -sIL https://example.com/products/blue-widget

A healthy result is a single 200 response with no redirect in front of it. A result like this means a consent redirect:

HTTP/2 302
location: https://example.com/consent?return=%2Fproducts%2Fblue-widget

HTTP/2 200
content-type: text/html; charset=utf-8

To get the final URL, status and number of hops on one line:

curl -s -o /dev/null -L -w '%{http_code} %{num_redirects} %{url_effective}\n' \
  https://example.com/products/blue-widget

If url_effective is a consent or age page, crawlers are not reaching the content.

Next, check whether a sentence from the real page is in the HTML. Pick something specific from the body, not the navigation:

curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://example.com/products/blue-widget | grep -c "machined from a single aluminium block"

A count of 0 means the text is not in what a request with GPTBot's user agent receives. Check that the sentence has no HTML entities or line breaks in the source, or you will get a false 0. Running the same command with a normal browser user agent tells you whether the gate applies to everyone or only to bots. The GPTBot string is OpenAI's documented example from its crawler page; the version number may change.

Finally, compare the title and a rough word count for a few different URLs:

for path in / /about /products/blue-widget /blog/some-post; do
  html=$(curl -s "https://example.com$path")
  title=$(printf '%s' "$html" | grep -o '<title>[^<]*' | head -1)
  words=$(printf '%s' "$html" | sed -e 's/<[^>]*>/ /g' | wc -w)
  echo "$path | $title | $words words"
done

If every URL returns the same title ("Please confirm your age") and a similar small word count, the server is sending the gate template instead of the pages. The word count is crude, since it includes scripts and styles, but identical numbers across different pages are a clear signal. For more ways to fetch a page as a crawler, see how to see your page as an AI crawler, with curl.

How to check in a browser with JavaScript off

curl shows the HTML; a no-JavaScript browser view shows what that HTML looks like once laid out. Both matter.

1. Open a private window, so no earlier consent cookie is sent.
2. Open Chrome DevTools, press Ctrl+Shift+P (Cmd+Shift+P on a Mac), type "Disable JavaScript" and run it.
3. Reload the page with DevTools still open.
4. Look for the main text. If the page is blank, shows only the banner, or shows "Enable JavaScript to continue", AI crawlers that do not render get the same.
5. Turn JavaScript back on, reload, and compare. Anything that appears only now is content a non-rendering crawler misses.

You can also use view-source: in the address bar of a private window to see the unprocessed HTML. The difference between the two views is explained in raw HTML vs rendered HTML.

A gate that only appears in some countries needs a check from that region, through a VPN or a remote testing service. Some consent setups show a wall only to visitors from certain regions, such as the EU, which ties into the problems described in geo-blocking and IP allowlists.

How to fix a cookie wall without dropping consent

A consent banner exists to control cookies and trackers. Nothing about that requires the article text to be missing from the HTML. The fix is to separate the two.

1. Render the content on the server for every visitor. The HTML for a first-time visitor with no cookies should contain the full page.
2. Gate the scripts, not the page. Load analytics, ad and marketing tags only after consent. That is what the banner is for.
3. Show the banner as an overlay on the same URL. No redirect, no separate consent page.
4. Keep banner text out of the way. Render it with a script, or place its markup after the main content. Google supports data-nosnippet on span, div and section elements, per its robots meta documentation, if you want to keep banner text out of Google snippets.

A minimal safe structure looks like this:

<body>
  <main>
    <h1>Blue widget</h1>
    <p>Machined from a single aluminium block...</p>
  </main>

  <div id="consent" data-nosnippet hidden>
    <p>We use cookies for analytics. <button>Accept</button> <button>Decline</button></p>
  </div>
  <script>
    // Show the banner only if no choice has been stored yet.
    if (!document.cookie.includes("consent=")) {
      document.getElementById("consent").hidden = false;
    }
  </script>
</body>

If your consent platform has an option that blocks the page or redirects to a preference page until a choice is made, switch it to a banner or overlay mode, then repeat the curl checks.

How to handle age gates and legally required interstitials

Age gates are harder, because for some products the law may require that content stays hidden until a visitor confirms their age. Google treats these as mandatory interstitials and exempts them from its interstitial guidelines, but still recommends good practice where possible: overlay the gate on the content rather than redirecting, so Google can still index at least some of the content.

You have three options, from simplest to most involved:

• Overlay the gate on server-rendered content. The page text is in the HTML and the gate covers it in the browser. This is Google's recommended pattern when it is acceptable for your product.
• Show crawlers the content without the gate. For content behind a mandatory age gate for adults, Google's interstitials page recommends letting Googlebot crawl without triggering the gate, after verifying that requests really come from Googlebot (reverse and forward DNS, or Google's published IP ranges). No AI operator documents this pattern, so extending it to AI crawlers is your own decision. If you do, verify them by their published IP ranges rather than trusting the user agent: OpenAI, Anthropic and Perplexity all publish lists, covered in how to verify a real AI crawler. Serve the same content you would show a verified adult, nothing different.
• Keep the content fully gated and accept the trade-off. If you cannot show the text to anyone unverified, crawlers will index the gate, not the pages. Make the gate page itself useful: a clear title, a short description of the site, and links to pages that are not age-restricted.

Do not let the gate return a 200 with identical text on thousands of URLs. If those URLs are meant to stay out of search, it is cleaner to say so with noindex than to let crawlers record the same gate on every one.

Promotional pop-ups, such as newsletter sign-ups and discount offers, are a different case. If they are drawn by a script over server-rendered content, crawlers read the page normally. For users, though, Google's guidance says not to obscure the whole page and to use banners that take up only a small fraction of the screen instead of full-page interstitials.

Login walls, paywalls and bot challenges are related problems with their own fixes: see paywalls and login walls.

Frequently asked questions

Do AI crawlers accept cookie banners?

No. Crawlers do not click buttons, and none of the major AI operators documents keeping cookies between requests. They read whatever the server sends to a first-time visitor with no cookies, so the content has to be in that response.

Is a JavaScript cookie banner bad for crawlers?

Usually not. If the page text is in the server's HTML and the banner is added by a script, crawlers that do not run JavaScript never see the banner and read the text. It only becomes a problem when the script also controls whether the content appears.

Is showing crawlers content without the age gate cloaking?

Not when the content matches what a verified visitor gets: Google's interstitials guidance itself recommends letting verified Googlebot requests past a mandatory adult age gate. Serving crawlers different content from what verified visitors see is what creates the cloaking risk, and crawlers should be verified by IP or DNS rather than by user agent alone.

How do I know if my consent tool redirects?

Run curl -sIL on a few page URLs; curl sends no cookies by default, and if the gate is regional, run it from that region. A 301, 302, 303 or 307 to a consent, privacy or preference URL means the tool is redirecting, and crawlers end up on that page.

Does a cookie wall affect Google AI Overviews?

It can. Google says a page must be indexed and eligible for a snippet to appear as a link in AI Overviews or AI Mode, so a page that Googlebot only sees as a consent gate has nothing to offer there.

Check your own pages

Run the curl commands above on your home page and two or three deep pages, and repeat them in a private window with JavaScript off. If the content is missing for a cookie-less visitor, that is the first thing to fix. To check a single page without an account, run the free Crawlito check; for a whole site, an audit covers up to 500 pages with evidence and prioritized fixes.