Skip to content

Is serving AI crawlers different content cloaking?

Serving AI crawlers different content can count as cloaking. What Google's spam policy says, which differences are fine, which are risky, and how to audit.

By Crawlito · 8 min read

In short

  • Google defines cloaking as presenting different content to users and search engines with the intent to manipulate rankings and mislead users, and lists inserting text or keywords only when the requesting user agent is a search engine as an example.
  • Google says it generally does not treat dynamic rendering as cloaking when the crawler version produces similar content, but serving completely different content to users and crawlers can be considered cloaking.
  • Google does not count a paywall as cloaking when it can see the full content as a person with access would and the site follows its flexible sampling guidance.
  • At the time of writing, the OpenAI, Anthropic and Perplexity crawler pages document user agents, IP ranges and robots.txt, and say nothing about serving their crawlers different content.
  • Google's AI optimization guide says you do not need Markdown, AI text files or special markup to appear in Google Search, including its generative AI features.
On this page

It can be. Google defines cloaking as presenting different content to users and to search engines in order to manipulate rankings and mislead users. Giving a crawler a prerendered copy of the same page is a different thing: Google says Googlebot does not view dynamic rendering as cloaking when it produces similar content. A crawler-only summary, extra claims or hidden keywords sit on the wrong side of the line. The crawler pages from OpenAI, Anthropic and Perplexity say nothing on the subject, so Google's text is the clearest written rule, and the safe default is the same content for every visitor.

This guide covers what Google's policy says, what the AI operators document, which differences are accepted, which are risky, and how to compare what a crawler receives with what a browser receives. It describes policy text, not a guarantee about how any system will treat your site.

What does Google's spam policy say about cloaking?

Google's spam policies define cloaking as presenting different content to users and search engines with the intent to manipulate search rankings and mislead users. It gives two examples:

  • Showing search engines a page about travel destinations while users see a page about discount drugs.
  • Inserting text or keywords into a page only when the requesting user agent is a search engine, not a human visitor.

The second example is the one closest to AI crawlers. A rule of the form "if the user agent contains a crawler name, add or swap content" resembles that pattern. Google's text is about search engines and about intent to manipulate, and it does not say how it would judge a rule aimed at other crawlers, so treat such a rule as a risk.

The same page says that if your site uses technologies search engines struggle with, such as JavaScript, you should follow Google's recommendations for making that content accessible "without cloaking". It also says Google detects spam with automated systems and human review, that this can lead to a manual action, and that violating sites may rank lower or not appear in results at all.

This is a Google Search policy, but it reaches AI features. Google's AI features documentation says a page has to be indexed and eligible to show with a snippet to be a supporting link in AI Overviews or AI Mode, and that there are no additional technical requirements. A page that Google drops from results cannot serve as a supporting link there.

The definition includes intent, but do not build your plan on it. Google's dynamic rendering page gives a plain test: a site that serves a page about cats to users and a page about dogs to crawlers is cloaking.

What do the AI crawler operators document?

Each operator publishes a crawler page. None of the ones checked for this guide, at the time of writing, covers whether you may serve its crawlers different content.

OperatorDocumentsSays about different content
GoogleGooglebot, verification by DNS lookup or IP range files, spam policiesCloaking is a spam policy violation
OpenAIOAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot user agents, published IP files, robots.txt behavior (crawler overview)Nothing
AnthropicClaudeBot, Claude-SearchBot and Claude-User, published IP ranges, robots.txt behavior (Anthropic's crawler article)Nothing
PerplexityPerplexityBot and Perplexity-User, published IP files, robots.txt behavior (Perplexity's crawler guide)Nothing

A missing rule is not permission. It means there is no documented safe harbor. Two practical points follow:

  • Googlebot can still see the difference. If your detection list matches AI crawlers and a mistake or a catch-all rule also matches Googlebot, Google sees the crawler version.
  • People follow the links. ChatGPT-User and Perplexity-User fetch pages when a person's request needs them. If an answer repeats text that exists only in a crawler version, the reader who clicks through will not find it.

Which differences are usually fine?

Some differences are documented and accepted, because the content users get is equivalent or because Google has described a pattern and said how to do it.

DifferenceWhat Google documents
Paywalls and content gatingNot cloaking if Google can see the full content behind the paywall just like a person with access, and you follow flexible sampling. Paywalled content markup helps Google tell a paywall from cloaking.
Different HTML per deviceDynamic serving uses one URL, user-agent detection and a Vary: User-Agent header. Google's mobile guidance says the versions should carry equivalent content, titles, meta descriptions and robots meta tags. It recommends responsive design as the easiest pattern.
A prerendered snapshot for botsGenerally not treated as cloaking when the dynamic rendering produces similar content. Google still calls it a workaround and recommends server-side rendering, static rendering or hydration.
Accordions, tabs and slidersGoogle's hidden text policy says design elements that show and hide content for visitors do not violate it.

The paywall row rests on Googlebot seeing what a subscriber sees, and Google's own text says nothing about other crawlers. The detail of that tradeoff is in paywalls and AI crawlers.

Which differences are risky?

These are the patterns that match what Google describes as cloaking or related spam, or that create mismatches Google's guidelines warn about.

  • A crawler-only summary or "AI-friendly" rewrite on the same URL. Text added only when the requester is a crawler is close to the user-agent example in the spam policy. It is also unnecessary: Google's AI optimization guide says you do not have to write content in a special way for generative AI search.
  • Hidden keywords. Google's policy names white text on a white background, text positioned off-screen and font size or opacity set to 0 as hidden text abuse. Filling a page with keywords or numbers to manipulate rankings is keyword stuffing.
  • Claims only the crawler sees. Awards, prices, specifications or availability that appear in the bot version and not on the visible page. If that content is also in structured data, it breaks Google's rule not to mark up content that is not visible to readers of the page.
  • A user-agent allowlist that anyone can fake. A user agent is a header any client can set. If a rule hands out a special version on that header alone, any visitor can get it. Google's verification page describes checking IPs by reverse DNS or published IP ranges, and OpenAI, Anthropic and Perplexity each publish IP ranges for their crawlers. Verifying real AI crawlers shows how.

What is a safer pattern?

Make the difference as small as you can, then verify what is left.

  1. One HTML source for everyone. Server-side rendering or static generation gives crawlers and people the same document. Google's JavaScript basics calls server-side or pre-rendering still a great idea, partly because not all bots can run JavaScript.
  2. If you prerender, keep the snapshot equal. Same headings, text, links, structured data and status codes, refreshed when the page changes. Prerendering for crawlers covers the setup and the stale snapshot risk.
  3. Negotiate by what the client asks for, not who it says it is. Cloudflare's Markdown for Agents returns a Markdown conversion of the same HTML page when a client sends Accept: text/markdown, adds Vary: Accept so caches keep both variants, and is available on Pro, Business and Enterprise plans at the time of writing. Cloudflare's page does not discuss cloaking, and the Google pages checked for this guide do not say how Google treats such responses, so treat it as a lower-risk design, not a cleared one.
  4. Publish alternates openly. A public Markdown page or an llms.txt file that any visitor can open is not a hidden version. Google's guide says its Search ignores such files, so do not expect them to help there.
  5. If you must vary, verify the crawler. For a paywall, confirm requests by IP or DNS rather than user agent alone, and mark the page up as Google describes.

How do you audit what crawlers receive?

Fetch the page twice, once as a browser and once as the crawler you worry about, then compare the text. This script keeps each fetch and diffs the words:

bash
URL="https://example.com/pricing"
BROWSER="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
BOT="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"

curl -s -L -A "$BROWSER" "$URL" -o browser.html
curl -s -L -A "$BOT" "$URL" -o bot.html

words() { sed -e 's/<[^>]*>/ /g' "$1" | tr -s '[:space:]' '\n' | sort -u; }

wc -w browser.html bot.html
diff <(words browser.html) <(words bot.html) | head -40

Read the result in this order:

  1. Status codes and redirects. Run curl -sI -A "$BOT" "$URL" and compare it with the browser run. A 403, 429 or challenge page for one and a 200 for the other is an access problem, covered in see your page as an AI crawler with curl.
  2. Headings, prices, dates and links. A line that exists in only one diff side is the finding. Words that differ because of a timestamp, a nonce or a rotating banner are noise.
  3. Title, canonical, robots meta and JSON-LD. Search the saved files for <title, rel="canonical", name="robots" and application/ld+json. Structured data that describes content the other version lacks is the mismatch to fix.
  4. A baseline. Fetch twice with the same user agent. If those two copies differ, personalization or testing is adding variance, and the browser-versus-bot gap means less.

Know the limits. The test sends a user-agent string from your own IP, so rules based on IP ranges or bot verification can treat the real crawler differently. It also compares raw HTML only, not what a browser builds after running JavaScript. Your server or CDN logs show what real crawlers received.

Frequently asked questions

Is prerendering for AI crawlers cloaking?

Not in Google's view when the crawler version produces similar content to what users see. Google generally does not treat dynamic rendering as cloaking on that condition, though it calls the approach a workaround.

Is serving a Markdown version of a page to AI crawlers cloaking?

Google's spam policy tests whether you present different content to users and search engines to manipulate rankings and mislead people. A Markdown copy of the same text, offered by Accept header or at a public URL, differs in format, not in substance, but none of the Google pages checked for this guide clears or bans it. Google says you do not need Markdown to appear in Search, because Search itself does not use such files.

Is it cloaking to allow search crawlers and block training crawlers?

Google's cloaking text is about presenting different content, not about which crawlers you let in. Separate rules for search and training are a documented pattern: OpenAI says its GPTBot and OAI-SearchBot settings are independent. See allowing AI search while blocking AI training.

What happens if Google decides a site is cloaking?

Google says violating sites may rank lower or not appear in results at all, and that violations can lead to a manual action. It does not publish how it weighs individual cases.

Do ChatGPT and Perplexity have a cloaking policy?

Not on the crawler pages checked at the time of writing. That is silence, not approval, so keep one version of the content for every visitor.

Check what a crawler receives

The free check reads one page the way AI crawlers do, with no account, and shows the text in the raw HTML your server sends. If you want the comparison across a whole site, with evidence and prioritized fixes, an audit covers up to 500 pages.

Filed under Access and robots.txt. Topics: ai-crawlers, google, rendering, bot-protection