noindex, nosnippet and X-Robots-Tag: what they mean for AI search

nosnippet, max-snippet, data-nosnippet and noindex are serving controls, not crawling controls. robots.txt decides whether a crawler may fetch a URL; these rules, set in a robots meta tag or an X-Robots-Tag HTTP header, decide what a search engine may do with a page it has fetched. Google documents that its snippet controls also govern AI Overviews and AI Mode. Other AI operators document far less, so for them you can rely only on what each one states, which this guide sets out operator by operator.

Facts below come from Google's robots meta tag documentation and AI features page, plus each operator's own pages, read on 8 November 2026.

Crawling controls vs indexing and serving controls

Most confusion comes from treating these as one layer. They are three:

The order matters. A crawler reads robots.txt first, then fetches the page, then reads the meta tag or header. Google is explicit about the consequence: if a page is disallowed in robots.txt, "any information about indexing or serving rules will not be found" and those rules are ignored. A blocked URL can still be listed from links elsewhere, just without the content. The longer comparison is in robots.txt vs meta robots.

What each directive does in Google Search and AI features

Google's AI features page says you can limit what appears from your pages in Search, AI features included, with nosnippet, data-nosnippet and max-snippet, and block indexing with noindex. AI Overviews and AI Mode are part of Search, so there is no separate AI tag to add.

Two rules from Google's documentation decide edge cases:

• The more restrictive rule wins. If a page carries both nosnippet and max-snippet:50, nosnippet applies.
• Only two crawler names work in the meta tag. Google supports googlebot and googlebot-news as the name value; other names are ignored. <meta name="googlebot" content="nosnippet"> affects Google only, while <meta name="robots"> addresses every crawler that reads it.

The cost of nosnippet

nosnippet is not an AI-only switch. It removes your text snippet from ordinary results too, and Google says a page must be "eligible to be shown in Google Search with a snippet" to appear as a supporting link in AI Overviews or AI Mode. Use it on pages where that trade is acceptable, such as licensed content or member material that is partly public. For the wider eligibility picture see Google AI Overviews: the technical requirements.

Partial controls with data-nosnippet and max-snippet

If you want the page in AI answers but a block of text kept out, mark that block:

<p>Our standard plan starts at $49 per month.</p>
<div data-nosnippet>
  <p>Volume pricing for partners is shared on request.</p>
</div>

data-nosnippet is valid only on span, div and section. On a p, li or table it does nothing, so wrap the element instead. To cap length rather than hide text:

<meta name="robots" content="max-snippet:160">

Google-Extended is not a snippet control

Blocking the Google-Extended token in robots.txt does not keep content out of AI Overviews or AI Mode. Google says it covers Gemini training and grounding in Gemini Apps and Vertex AI, and does not affect Search inclusion. Google-Extended explained covers what it does control.

How to set X-Robots-Tag for PDFs and other non-HTML files

A PDF, image, Word file or JSON feed has no <head>, so a meta tag is impossible. The X-Robots-Tag response header carries the same rules for any file type, and works for HTML too:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, nofollow

You can send several headers, and address one crawler by prefixing its name, following the pattern of Google's examples:

X-Robots-Tag: googlebot: nosnippet
X-Robots-Tag: otherbot: noindex, nofollow
X-Robots-Tag: unavailable_after: 31 Dec 2026 23:59:59 GMT

Apache

Google's example for every PDF on a site, in .htaccess or the server config with mod_headers enabled:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

nginx

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

location = /reports/annual-2026.pdf {
  add_header X-Robots-Tag "nosnippet";
}

nginx uses the exact-match location = block for that one file, so the annual report gets nosnippet only, while every other PDF gets noindex, nofollow. An add_header inside a location block also means headers set in the parent block are no longer inherited there, so repeat any security headers you rely on.

Next.js

In next.config.js, the headers() function adds response headers by path:

module.exports = {
  async headers() {
    return [
      {
        source: "/downloads/:file*",
        headers: [{ key: "X-Robots-Tag", value: "noindex" }],
      },
    ];
  },
};

Pick the rule to match the goal. noindex keeps a PDF out of Google entirely; nosnippet keeps it listed but stops its text being quoted or used as a direct AI input. And do not also disallow the PDF in robots.txt, or crawlers never see the header.

Which AI operators document support for these rules

Google is the only operator with detailed, current documentation tying snippet controls to its AI answers. Here is what each operator says, and where it says nothing.

Three cautions follow from this table:

1. Absence is not refusal. Anthropic and Perplexity may read noindex; they simply do not document it. Do not rely on an undocumented control for anything sensitive.
2. The Bing post predates the Copilot name. It was written for Bing Chat in September 2023. Check Bing's current webmaster guidelines before relying on its exact wording.
3. User-triggered fetchers are a separate case. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. Neither documents how these agents treat meta robots. The OpenAI crawlers guide has the full detail.

If content must not reach any AI tool, the only reliable control is not serving it publicly: put it behind authentication, as covered in paywalls and login walls.

The robots.txt trap: noindex on a blocked page

The most common mistake is stacking both controls:

User-agent: *
Disallow: /old-campaign/

<meta name="robots" content="noindex">

The intent is "remove these pages". The effect is that crawlers stop fetching them, never see the noindex, and may keep listing the bare URLs found through links. Google and OpenAI both say the crawler must be able to fetch a page to read its tag. To remove pages, allow crawling, serve noindex, wait for the pages to drop out, and only then add a Disallow if you still want one.

The reverse mistake is just as costly: a noindex left over from staging on a live site. WordPress's "Discourage search engines" setting, for example, has added a noindex,nofollow meta tag since version 5.3. The leftover noindex covers how this happens on each platform.

How to check which rules a page sends

Check the header and the HTML separately, because a CDN, plugin or framework can add either.

1. Read the response headers:

   bash

   curl -sI https://example.com/reports/annual-2026.pdf | grep -i x-robots-tag

2. Read the meta tags in the raw HTML:

   bash

   curl -s https://example.com/pricing | grep -Eio '<meta[^>]*name="(robots|googlebot)"[^>]*>'

3. Find data-nosnippet blocks and confirm each sits on a span, div or section:

   bash

   curl -s https://example.com/pricing | grep -o '<[a-z]* [^>]*data-nosnippet[^>]*>'

4. Repeat with a crawler user agent, for example curl -sI -A "OAI-SearchBot" <url>, since some servers and CDNs vary responses by bot. A spoofed user agent does not come from the bot's IP ranges, so treat the result as a hint, not proof.

5. Confirm robots.txt allows the URL, or none of the above is visible to the crawler.

6. Check Google's view with URL Inspection in Search Console, which reports whether indexing is allowed and whether the page was blocked by robots.txt.

If a rule comes from JavaScript after load, crawlers that do not run JavaScript will not see it, and Google notes that its own rendering is not guaranteed. Google also says not to add or remove data-nosnippet on existing elements with JavaScript. Put robots rules in the server-sent HTML or the header.

Frequently asked questions

Does nosnippet keep my page out of AI Overviews?

Google says nosnippet prevents the page's content from being used as a direct input for AI Overviews and AI Mode. It also removes your text snippet from ordinary Google results, so it is a trade, not a free AI opt-out.

Should I use the meta tag or the X-Robots-Tag header?

They carry the same rules. Use the meta tag for HTML pages you edit in a CMS, and the header for PDFs, images and other non-HTML files, or when you want one server rule to cover a whole path.

Can I block AI Overviews but keep normal Google snippets?

Not with a documented control. Google applies nosnippet, max-snippet and data-nosnippet to both, and Google-Extended does not cover AI Overviews or AI Mode.

Does noindex stop AI models training on my content?

Not in general. Training opt-outs are separate robots.txt user-agent tokens such as GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. The documented exceptions use noarchive, not noindex: Amazon defines it as "do not use the page for model training" for Amazonbot, and Bing's 2023 post says noarchive content is not used to train Microsoft's models.

Will ClaudeBot or PerplexityBot respect nosnippet?

Neither Anthropic nor Perplexity documents support for nosnippet or max-snippet at the time of writing. Use robots.txt for those crawlers, which both operators document.

Next step

Rules set by a plugin, a CDN or a forgotten staging setting are easy to miss. The free single-page check runs on one live URL with no account, and its published checks include the X-Robots-Tag header, meta robots and the robots.txt rule for each named AI crawler, so you can see which controls apply before you change anything.