Paywalls and AI crawlers: what gated content they can read

AI crawlers read what your server sends to an anonymous request, and nothing more. They have no account and no subscription, so behind a hard paywall or a login wall they see the teaser, the login form or an error. Behind a paywall that only hides the article in the browser, they often see the whole thing. Which case you are in depends on how your paywall is built, not on what a visitor sees on screen.

This guide covers the common paywall models, what each puts in the raw HTML, what Google's paywalled content markup is for, and how to decide what to expose to AI search. It is about crawlers and fetchers that request pages anonymously; browser-based agents that a person drives inside their own logged-in session are a separate case. It covers technical options only, not licensing or legal questions. Sources were read on 8 November 2026.

What each paywall model puts in the raw HTML

The test that matters is the first HTML response, before any JavaScript runs. In a December 2024 analysis of crawler traffic on its network, Vercel and MERJ found that none of the major AI crawlers they measured, including OpenAI's crawlers, ClaudeBot and PerplexityBot, rendered JavaScript at the time. For those crawlers, the raw HTML is the page. (Googlebot does render JavaScript, using a headless Chromium.)

Two rows surprise people.

Client-side paywalls leak. If your article ships in the HTML and a script blurs it, removes it from view or covers it with an overlay, a crawler that does not run scripts never sees the overlay. Google's own guide to fixing Search-related JavaScript problems says that including the full content in the server response and hiding it with JavaScript "isn't a reliable way to limit access to the content", and tells publishers to provide the full content only once the subscription status is confirmed.

Meters depend on state the crawler may not keep. A meter that counts reads with a cookie can only count if the client sends that cookie back. AI crawler operators do not document whether they keep cookies between requests, so do not assume either way. Test it, as shown below.

How login walls look to a crawler

A login wall usually takes one of three forms, and each reads differently:

1. A redirect to /login (301, 302 or 307). The crawler follows it and finds a login form, so the original URL has no content of its own. See status codes, redirects and AI crawlers for how chains are handled.
2. A 401 or 403 response. The crawler gets an error and moves on. This is the clearest signal that the page is private.
3. A 200 response with a login form in place of the content. This is the worst case: the URL looks like a working page whose content is a sign-in box, which can end up indexed as such.

None of the operators document a way to give their crawler an account, and Anthropic states that its crawlers do not attempt to bypass CAPTCHAs (Anthropic's crawler article). Treat anything behind a login as unreadable to AI search.

The common mistake is not the members area itself but what ends up inside it by accident: pricing, documentation, help articles or product specs that sit behind a login because the whole app subdomain does. If you want those pages to be eligible for AI answers, they have to be served publicly.

What isAccessibleForFree markup does, and what it does not

Google's paywalled content structured data exists to solve one problem: telling paywalled content apart from cloaking, which that page says violates Google's spam policies. Google's flexible sampling guidance describes cloaking as serving Googlebot different content from what users get. If you let Googlebot read the full article so it can be indexed, while visitors see a paywall, the markup explains the difference.

The only required property is isAccessibleForFree, set to false. The recommended hasPart property points at the gated section with a WebPageElement and a class selector. The JSON-LD goes in a <script type="application/ld+json"> element on the same page:

<article>
  <div class="non-paywall">
    <p>The first two paragraphs everyone can read.</p>
  </div>
  <div class="paywall">
    <p>The rest of the article, for subscribers.</p>
  </div>
</article>

{
  "@context": "https://schema.org",
  "@type": "NewsArticle",
  "headline": "Example headline",
  "datePublished": "2026-11-08",
  "isAccessibleForFree": false,
  "hasPart": {
    "@type": "WebPageElement",
    "isAccessibleForFree": false,
    "cssSelector": ".paywall"
  }
}

Google's rules for it, from the same page:

• It works with CreativeWork and more specific types such as Article, NewsArticle, Blog, Course, Review and WebPage.
• Use only .class selectors in cssSelector, and do not nest gated sections. If a page has several gated sections, list them as an array.
• Validate the markup with the Rich Results Test, and use the URL Inspection tool in Search Console to see how Google crawls and renders the URL.
• If you do not want the gated text available in the browser at serving time, use a paywall that does not send it.

What the markup does not do matters just as much:

• It hides nothing. It is a label. If the full text is in the HTML, every crawler that fetches the page can read it, marked or not.
• It does not grant or limit AI use. Google's paywall page notes that AI Overviews and AI Mode are "subject to Search's preview controls". Limiting what they may use is the job of nosnippet, max-snippet and data-nosnippet, covered in noindex, nosnippet and X-Robots-Tag.
• Other AI operators do not document reading it. At the time of writing, OpenAI's crawler documentation and Perplexity's say nothing about paywalls, subscriptions or isAccessibleForFree.

If you serve the full text to Googlebot but do not want the gated part shown in Google's snippets or AI features, you can add data-nosnippet to the gated block. Google lists data-nosnippet among the controls that limit what appears from a page in Search, AI Overviews and AI Mode included, and supports it only on span, div and section elements. Its paywall documentation does not describe this combination, so treat it as an option to test rather than a documented pattern:

<div class="paywall" data-nosnippet>
  <p>The rest of the article, for subscribers.</p>
</div>

How to check what crawlers get from a gated page

Start with the raw response. These commands fetch a page the way a non-rendering crawler would: one request, no JavaScript, no cookies.

The status code and response headers, including any redirect to a login page (-o /dev/null -D - prints the headers of a normal GET request and discards the body):

curl -s -o /dev/null -D - -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://example.com/news/some-article

Whether a sentence from the gated section is in the raw HTML (a count of 1 or more means yes):

curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://example.com/news/some-article | grep -c "a sentence from paragraph six"

Whether the paywall markup is present:

curl -s https://example.com/news/some-article | grep -o '"isAccessibleForFree"[^,}]*'

Then work through these checks:

1. Pick a sentence from deep in the article, past the lead-in. If grep finds it in the anonymous response, the gated text is public to any crawler, whatever the browser shows.
2. Run the request several times. If a metered page returns the full text every time, the meter is not stopping a client that sends no cookies.
3. Compare with the rendered page in your browser's developer tools. A big gap in one direction or the other tells you where the paywall runs. Raw HTML vs rendered HTML explains how to read the difference.
4. Check your server logs for real crawler requests to gated URLs and the status codes they got. A curl request with a crawler's user agent comes from your IP, so a WAF or a bot rule may treat it differently from the real crawler. How to verify a real AI crawler shows how to confirm a log entry is genuine.

Deciding what to expose to AI search

There is no setting that lets AI search summarise an article without seeing it. Whatever you send a crawler can be read. Google documents snippet controls that also apply to its AI features, but at the time of writing OpenAI, Anthropic and Perplexity document nothing comparable to max-snippet or data-nosnippet in their crawler pages. So the decision is about what you send, page type by page type.

A few practical points shape the choice:

• Make the public part worth reading. A lead-in that states the main finding, a specific title and an accurate meta description give an AI system something true to work with. A bare "Subscribe to continue" gives it nothing.
• Search and training are separate decisions. You can allow OAI-SearchBot while disallowing GPTBot; OpenAI documents the two controls as independent. The same split exists for other operators, as set out in how to allow AI search but block AI training.
• User-triggered fetchers may not follow robots.txt. OpenAI says that for ChatGPT-User "robots.txt rules may not apply", and Perplexity says Perplexity-User generally ignores robots.txt because a user asked for the page. If text must stay private, keep it out of the anonymous response rather than relying on a Disallow line.
• Allow-listing by IP needs upkeep. The IP files change often (OpenAI's chatgpt-user.json was regenerated on 6 November 2026), so fetch them on a schedule instead of copying ranges by hand.

Frequently asked questions

Can ChatGPT read articles behind my paywall?

Only what your server sends without a subscription. If the paywall is enforced on the server, OpenAI's crawlers get the teaser; if the full text is in the HTML and hidden by a script, they can read all of it.

Does isAccessibleForFree stop AI crawlers from using my content?

No. It is a label for Google that explains why Googlebot sees more than a visitor does. It does not block any crawler, and OpenAI and Perplexity do not document reading it.

Do I need paywall markup if Googlebot only sees the teaser?

The markup's main job is to explain a gap between what Googlebot and visitors see. If Googlebot gets the same teaser as everyone else there is no gap to explain, though marking the page as not free still describes it accurately.

Will a login page hurt my AI search readiness?

Not for pages that are meant to be private. It becomes a problem when public material such as pricing, docs or help articles sits behind the same login, or when the login form is served with a 200 status at the article's URL.

Check what your gated pages send

Start with one gated URL in the free check, a single-page check that needs no account, and compare the result with the curl tests above. If your paywall differs by section or template, a full-site audit covers up to 500 pages and returns a PDF with the evidence and a prioritized list of fixes.