Skip to content

X-Robots-Tag: noindex PDFs, images and other non-HTML files

How the X-Robots-Tag HTTP header controls indexing and snippets for PDFs and images, with nginx, Apache, Cloudflare and Vercel examples and how to test it.

By Crawlito · 6 min read

In short

  • X-Robots-Tag is an HTTP response header that accepts every rule the robots meta tag accepts, so it works for PDFs, images and other files that have no HTML head.
  • Google only sees the header when it crawls the URL, so a file blocked in robots.txt cannot be noindexed with X-Robots-Tag.
  • A rule can target one crawler with a prefix, such as googlebot: noindex, and when rules conflict Google applies the more restrictive one.
  • In nginx, add_header in a location block replaces headers inherited from the server block, and it only applies to some status codes unless you add always.
  • Google documents nosnippet, max-snippet and noindex as the controls that limit what appears in its AI features; other AI operators document their own controls or none.
On this page

To control how search engines index a PDF, image or any other file without an HTML <head>, send the rule in the X-Robots-Tag HTTP response header instead of a meta tag. For example, X-Robots-Tag: noindex on a PDF response keeps that PDF out of Google's results. The header accepts every rule the robots meta tag does, but it only works if the crawler is allowed to fetch the file, so do not also block the file in robots.txt.

This guide covers the syntax, server and CDN configuration, how to test that the header is really sent, and what it does and does not control in AI search. For the wider question of whether PDFs are a good way to publish content at all, see PDF documents and AI search.

What X-Robots-Tag does and when to use it

Google's robots meta tag documentation describes X-Robots-Tag as a header that can carry any rule valid in a robots meta tag. It is most useful where a meta tag cannot go:

  • PDFs and office documents: price lists, old brochures, internal forms that ended up public.
  • Images and video files: when you want an image file kept out of image results.
  • Feeds, JSON and plain text that should be fetchable but not listed.
  • Whole sections where editing every page template is harder than one server rule.

You can send several X-Robots-Tag headers in one response, or several rules in one header separated by commas. Google says the header name, crawler names and values are not case-sensitive.

http
HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, nofollow

Rules you can use

RuleEffect in Google Search
noindexDo not show the file or page in results
nofollowDo not follow links in it
noneSame as noindex, nofollow
nosnippetNo text snippet or video preview
max-snippet:[n]Snippet up to n characters; 0 acts like nosnippet
max-image-preview:[none, standard, large]Size of image preview
noimageindexDo not index images on the page
unavailable_after:[date]Stop showing the URL after a date
indexifembeddedAllow indexing when embedded in another page, only together with noindex

Targeting one crawler

Prefix a rule with a crawler name and a colon to aim it at that crawler only. Rules without a prefix apply to all crawlers:

http
X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: noindex, nofollow

When rules conflict, Google applies the more restrictive one. A response with both max-snippet:50 and nosnippet is treated as nosnippet.

Why robots.txt and X-Robots-Tag cannot be combined on one file

This is the most common mistake. If robots.txt disallows /downloads/, Google never fetches the PDF, so it never sees the header. Google says so directly: rules in the header are only discovered when the URL is crawled, and a URL blocked by robots.txt has its indexing rules ignored. A blocked file can still be listed from links elsewhere, usually as a bare URL without a description.

So pick one job per file:

  • To keep it out of results: allow crawling and send noindex.
  • To stop it being fetched at all: disallow it in robots.txt, and accept that its URL can still be known.

robots.txt vs meta robots goes through the same trade-off for HTML pages.

How to set X-Robots-Tag in nginx and Apache

These follow the examples in Google's documentation.

nginx

nginx
# Every PDF on the site
location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex, nofollow" always;
}

# Image files
location ~* \.(png|jpe?g|gif)$ {
    add_header X-Robots-Tag "noindex" always;
}

# One file
location = /files/old-price-list.pdf {
    add_header X-Robots-Tag "noindex" always;
}

Two nginx details catch people out, both described in the ngx_http_headers_module documentation:

  1. Inheritance. add_header directives are inherited from the level above only if the current level has none of its own. Adding X-Robots-Tag in a location block silently drops security headers you set in the server block. Repeat them, or move shared headers into an included file.
  2. Status codes. Without the always parameter, add_header only applies to a set of success and redirect codes such as 200, 301 and 304. That rarely matters for noindex, but it is why a header can seem to come and go while you test.

Apache

Apache needs mod_headers enabled:

apache
<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

<Files ~ "\.(png|jpe?g|gif)$">
  Header set X-Robots-Tag "noindex"
</Files>

In an .htaccess file, a <Files> block for a single file name must sit in the same directory as that file.

How to set X-Robots-Tag on a CDN or host

If you do not control the server config, add the header at the edge.

Cloudflare

Use a response header transform rule with the Set static operation. The rule language has a field for the file extension, http.request.uri.path.extension, which holds the lowercased extension without the dot:

http.request.uri.path.extension eq "pdf"

Set the header name to X-Robots-Tag and the value to noindex. Set replaces an existing header of the same name, while Add appends another one, per Cloudflare's response header modification docs.

Vercel

The headers property in vercel.json matches paths with a source pattern. Keeping files that should not be indexed in one folder makes the pattern simple:

json
{
  "headers": [
    {
      "source": "/internal-docs/(.*)",
      "headers": [{ "key": "X-Robots-Tag", "value": "noindex" }]
    }
  ]
}

The same folder approach works on most hosts with a headers file. It is also easier to audit than an extension rule, because you can see at a glance which files are covered.

Pointing a PDF at its HTML version

If a PDF duplicates an HTML page, you may prefer to keep it indexable but tell Google which version is preferred. Google's canonical documentation supports a rel="canonical" link in the HTTP header for non-HTML documents:

http
Link: <https://www.example.com/pricing>; rel="canonical"

How to check the header is really sent

Configuration files describe intent. The response is what crawlers see. Request the file and print only the headers:

bash
curl -sI https://www.example.com/files/old-price-list.pdf | grep -i x-robots-tag

If nothing prints, check three things in order:

  1. HEAD versus GET. Some servers and edge functions answer HEAD differently. Repeat with a GET that discards the body: curl -s -D - -o /dev/null URL. HTTP HEAD vs GET for crawler testing explains why the two can disagree.
  2. Redirects. Add -L to follow them, and check the header on the final URL, not the redirect.
  3. Cache. A CDN may still serve a cached copy without the new header. Purge the file and test again.

In Google Search Console, the URL Inspection tool shows whether indexing is allowed for a crawled URL, which confirms Google saw the rule. Removal is not instant: Google has to recrawl the file first.

For Google, the header controls Google's AI features the same way it controls classic results. Google's AI features page says that to limit the information shown from your pages in Search, you use nosnippet, data-nosnippet, max-snippet or noindex. A PDF with X-Robots-Tag: noindex is therefore out of AI Overviews and AI Mode as well, once Google has recrawled it. More on those controls in noindex and nosnippet for AI search.

Other operators are less specific:

  • OpenAI documents a noindex meta tag for keeping pages out of ChatGPT search in its publisher FAQ, and says the crawler must be able to fetch the page to read it. It does not separately document the X-Robots-Tag header, so do not rely on it alone for ChatGPT.
  • Microsoft Bing reads robots meta rules and the header, and has announced its own ai-training and ai-grounding preferences, which can be sent as X-Robots-Tag: Bingbot: ai-grounding:disallow. Its announcement explains how these replace the older noarchive and nocache values for those purposes.
  • Anthropic and Perplexity document robots.txt for their crawlers and do not document header rules at the time of writing.

If a file must not be read by anyone, a header is the wrong tool. Put it behind authentication.

Frequently asked questions

Can I use X-Robots-Tag on HTML pages too?

Yes. Google treats it the same as a robots meta tag. It is handy for applying noindex to a whole section from one server rule, but a meta tag is easier for editors to see and change.

Will X-Robots-Tag noindex remove a PDF that is already in Google?

Yes, after Google recrawls the file and sees the header. For something urgent, use Search Console's removal tool as a stopgap while the header takes effect.

Should I block PDFs in robots.txt as well?

No. If robots.txt blocks the file, Google cannot fetch it and never sees the noindex header, so the URL can stay listed.

Does X-Robots-Tag noindex stop AI training crawlers?

Not as a documented control. Training opt-outs are documented through robots.txt tokens such as GPTBot, ClaudeBot and Google-Extended. The header is an indexing and snippet control.

Find stray noindex headers

Headers are invisible in a browser, which is why a forgotten noindex from a staging config can sit on live files for months, as described in staging noindex leaks. Run the curl check above on a few important URLs, or run the free check on one page, with no account, to see how AI crawlers are treated there.

Filed under Access and robots.txt. Topics: meta-robots, noindex, pdf, http-status, google