robots.txt vs meta robots: blocking crawling vs blocking use

robots.txt and the robots meta tag solve different problems. A Disallow rule in robots.txt tells a crawler not to fetch a URL; a noindex rule in a robots meta tag or X-Robots-Tag header tells a search engine not to list a page it has fetched. Because a crawler can only read noindex after fetching the page, putting both on the same URL backfires: the Disallow hides the noindex, and the URL can still show up in results.

This guide explains the difference, the URL-only listing it causes, a decision table for which to use, and how the same split applies to AI crawlers. Facts come from Google Search Central and the AI operators' own documentation, read on 8 November 2026.

What robots.txt does and what noindex does

The two mechanisms sit at different steps of the same pipeline: fetch, then index, then show.

Google states the boundary plainly in its robots.txt introduction: robots.txt is "not a mechanism for keeping a web page out of Google". Its main job is managing crawler traffic. To keep a page out of results, Google points you to noindex or password protection.

The robots meta tag documentation states the other side: robots meta tags and X-Robots-Tag headers are discovered when a URL is crawled, so if robots.txt blocks the crawl, those rules are ignored.

Why a robots.txt block can hide your noindex

Here is the configuration that causes most of the confusion. Someone wants /internal-report/ out of search, so they add both controls:

User-agent: *
Disallow: /internal-report/

<!-- /internal-report/index.html -->
<meta name="robots" content="noindex">

What a compliant crawler does:

1. It reads robots.txt and sees /internal-report/ is disallowed.
2. It never requests the page, so it never sees the noindex.
3. If other pages or sites link to /internal-report/, the search engine still knows the URL exists.

Google's noindex guide describes exactly this: if the page is blocked by robots.txt, the crawler will never see the noindex rule, and the page can still appear in search results, for example when other pages link to it.

What a URL-only listing looks like

When Google lists a URL it was not allowed to crawl, the result has no description, because Google never read the content. The robots.txt introduction says the URL "can still appear in search results, but the search result won't have a description". So the page you wanted hidden is still visible, just without a description.

WordPress hit this problem at scale. Up to version 5.2, its "Discourage search engines from indexing this site" setting served Disallow: / in robots.txt. In 5.3 it switched to a noindex,nofollow robots meta tag, because, as the core team explained, a robots.txt block can still let a linked URL appear in results. Webflow gives the same advice in its indexing help: do not use robots.txt to remove previously indexed pages; use the page-level setting that adds noindex.

noindex in robots.txt does not work

Some older robots.txt files contain a line like Noindex: /private/. Google's noindex guide says specifying noindex in robots.txt is not supported. RFC 9309 defines only User-agent, Allow and Disallow rules (it mentions other records such as Sitemap only as optional extras), so other crawlers have no standard reason to honour a Noindex line either. Move the rule into the page or the response header.

Where to put noindex: meta tag or X-Robots-Tag header

For HTML pages, a meta tag in the <head> is simplest:

<meta name="robots" content="noindex">

For PDFs, images and other files that have no <head>, send the rule as a response header. A PDF response carrying it looks like this:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

On nginx, a location block can add it to every PDF. Google's robots meta documentation shows the same pattern, with Apache equivalents:

location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex" always;
}

always sends the header on error responses too. An add_header inside a location replaces any add_header lines inherited from the server block, so repeat headers you still need there.

A few details from Google's robots meta documentation that matter in practice:

• none is equivalent to noindex, nofollow.
• When rules conflict, the more restrictive rule applies. A noindex from a plugin beats an index from your theme.
• The name attribute can target a crawler. Google supports googlebot and googlebot-news and ignores other crawler names; robots applies to all crawlers that support the tag.

When to use robots.txt and when to use noindex

Pick the control that matches the outcome you want, not the one that sounds strongest.

Two rules of thumb:

• Never combine Disallow and noindex on the same URL if your goal is removal. Choose one based on the table.
• Neither is security. robots.txt is public and lists the paths you want hidden. Anything that must stay private needs a login.

How to remove an indexed page properly

1. Remove any robots.txt rule that blocks the URL.
2. Add <meta name="robots" content="noindex"> or the X-Robots-Tag header, and keep returning 200.
3. Ask for a recrawl. In Google Search Console, the URL Inspection tool can request it; Google's noindex guide says a page still showing is probably one it has not recrawled since you added the rule.
4. Leave the noindex in place. If you disallow the URL afterwards, crawlers stop seeing the rule, and a linked URL can come back as a URL-only result.

How the same split applies to AI crawlers

The logic carries over to AI search, with one extra layer: AI operators separate crawling for search from using content for training.

AI search: the crawler still has to read the page

Google says its AI features, AI Overviews and AI Mode, are part of Search and are controlled by Googlebot's robots.txt rules plus noindex, nosnippet, data-nosnippet and max-snippet (AI features and your website). A page must be indexed and eligible for a snippet to appear as a supporting link. So the Disallow-hides-noindex problem applies to AI Overviews exactly as it applies to blue links. Google also says nosnippet keeps content from being used as a direct input for AI Overviews and AI Mode; noindex, nosnippet and X-Robots-Tag for AI search covers those controls in detail.

For ChatGPT search, OpenAI's crawler documentation says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. That is the AI version of a URL-only listing. At the time of writing, OpenAI's Publishers and Developers FAQ points to a noindex meta tag for keeping a page out of ChatGPT search and notes that the crawler must be allowed to crawl the page to read it. The FAQ is updated often, so check its current wording.

Not every operator documents page-level rules. Amazon says Amazonbot honours noindex and none, and defines noarchive as "do not use the page for model training". Anthropic's crawler article describes robots.txt controls for ClaudeBot, Claude-SearchBot and Claude-User but does not document robots meta tag handling. Where a rule is undocumented, do not assume it works.

AI training: a use control, not a listing control

Training opt-outs are robots.txt tokens. Disallowing GPTBot signals that content should not be used to train OpenAI's foundation models, and OpenAI says that setting is independent of OAI-SearchBot. Disallowing ClaudeBot signals a training opt-out to Anthropic. A noindex tag is not documented by either as a training opt-out.

Some tokens are pure use controls that never fetch anything. Google-Extended and Applebot-Extended have no crawler of their own: Googlebot and Applebot keep crawling, and the token only decides how the crawled content may be used: for Google-Extended, training Gemini models and grounding answers in Gemini Apps and Vertex AI; for Applebot-Extended, training Apple's foundation models. Google says Google-Extended does not affect inclusion in Google Search, and Apple says pages that disallow Applebot-Extended can still appear in Apple search results. That is the clearest case of "blocking use" rather than "blocking crawling".

This file leaves search crawlers such as Googlebot, OAI-SearchBot and Claude-SearchBot allowed through the * group while opting out of training for those four operators. Blocking Google-Extended also opts out of Gemini grounding, which is a separate choice from Google Search:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

The allow AI search, block AI training guide covers this pattern for each operator.

User-triggered fetchers

ChatGPT-User and Perplexity-User fetch a page because a person asked. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. Anthropic, by contrast, says its bots, Claude-User included, honour robots.txt. For the fetchers that skip it, neither control reliably stops the request. If a page must not be read, put it behind a login.

How to check which control a page is sending

Check all three layers, because a CDN, plugin or framework can add one you did not write. The commands below fetch, in order, the host's robots.txt, the X-Robots-Tag response header (sent with an OAI-SearchBot user agent), and any robots meta tags in the raw HTML:

curl -s https://example.com/robots.txt

curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  https://example.com/page | grep -i x-robots-tag

curl -s https://example.com/page | grep -Eio '<meta[^>]*name="(robots|googlebot)"[^>]*>'

Then read the results together:

• Disallowed and noindex: remove the Disallow if you want the noindex to work.
• Disallowed only: fine for crawl control, but the URL can still be listed without a description.
• noindex you did not expect: often a staging setting carried to production. See the leftover noindex.
• Meta tag only in rendered HTML: crawlers that do not run JavaScript will not see it. Check the raw response, as in seeing your page as an AI crawler with curl.

To confirm which robots.txt group each AI crawler actually follows, use the steps in testing robots.txt against each AI crawler. For Google, Search Console's URL Inspection shows whether a URL is blocked by robots.txt or excluded by noindex.

Frequently asked questions

Should I use robots.txt or noindex to keep a page out of Google?

Use noindex and leave the page crawlable. Google says robots.txt is not a mechanism for keeping a page out of Google, and a blocked URL can still appear without a description.

Why does my disallowed page still appear in search results?

The search engine knows the URL from links but was not allowed to fetch it, so it cannot read any noindex on the page. Remove the Disallow, add noindex and request a recrawl.

Can I put noindex in robots.txt?

No. Google does not support a noindex rule in robots.txt, and RFC 9309 does not define one. Use a robots meta tag or an X-Robots-Tag header.

Does noindex stop AI models from training on my content?

Not by itself for the major operators. OpenAI, Anthropic, Google and Apple document training opt-outs as robots.txt tokens: GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. Amazon is an exception: it defines the noarchive meta tag as "do not use the page for model training" for Amazonbot.

Does blocking Google-Extended remove my pages from AI Overviews?

No. Google says AI Overviews and AI Mode are part of Search and follow Googlebot rules and snippet controls. Google-Extended covers Gemini training and grounding, not Search inclusion.

Check your own pages

A conflict between robots.txt and noindex is invisible until you look at both on the same URL. Run the three commands above on the pages that matter most to you, or start with Crawlito's free check, which tests a single page with no account needed. If you need the whole site covered, the full audit runs deterministic checks across up to 500 pages and delivers a PDF with evidence and prioritized fixes.