X-Robots-Tag: noindex PDFs, images and other non-HTML files
How the X-Robots-Tag HTTP header controls indexing and snippets for PDFs and images, with nginx, Apache, Cloudflare and Vercel examples and how to test it.
By Crawlito · 6 min read
In short
- X-Robots-Tag is an HTTP response header that accepts every rule the robots meta tag accepts, so it works for PDFs, images and other files that have no HTML head.
- Google only sees the header when it crawls the URL, so a file blocked in robots.txt cannot be noindexed with X-Robots-Tag.
- A rule can target one crawler with a prefix, such as googlebot: noindex, and when rules conflict Google applies the more restrictive one.
- In nginx, add_header in a location block replaces headers inherited from the server block, and it only applies to some status codes unless you add always.
- Google documents nosnippet, max-snippet and noindex as the controls that limit what appears in its AI features; other AI operators document their own controls or none.
On this page
- What X-Robots-Tag does and when to use it
- Why robots.txt and X-Robots-Tag cannot be combined on one file
- How to set X-Robots-Tag in nginx and Apache
- How to set X-Robots-Tag on a CDN or host
- How to check the header is really sent
- What X-Robots-Tag means for AI search
- Frequently asked questions
- Find stray noindex headers
To control how search engines index a PDF, image or any other file without an HTML <head>, send the rule in the X-Robots-Tag HTTP response header instead of a meta tag. For example, X-Robots-Tag: noindex on a PDF response keeps that PDF out of Google's results. The header accepts every rule the robots meta tag does, but it only works if the crawler is allowed to fetch the file, so do not also block the file in robots.txt.
This guide covers the syntax, server and CDN configuration, how to test that the header is really sent, and what it does and does not control in AI search. For the wider question of whether PDFs are a good way to publish content at all, see PDF documents and AI search.
What X-Robots-Tag does and when to use it
Google's robots meta tag documentation describes X-Robots-Tag as a header that can carry any rule valid in a robots meta tag. It is most useful where a meta tag cannot go:
- PDFs and office documents: price lists, old brochures, internal forms that ended up public.
- Images and video files: when you want an image file kept out of image results.
- Feeds, JSON and plain text that should be fetchable but not listed.
- Whole sections where editing every page template is harder than one server rule.
You can send several X-Robots-Tag headers in one response, or several rules in one header separated by commas. Google says the header name, crawler names and values are not case-sensitive.
HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, nofollowRules you can use
| Rule | Effect in Google Search |
|---|---|
noindex | Do not show the file or page in results |
nofollow | Do not follow links in it |
none | Same as noindex, nofollow |
nosnippet | No text snippet or video preview |
max-snippet:[n] | Snippet up to n characters; 0 acts like nosnippet |
max-image-preview:[none, standard, large] | Size of image preview |
noimageindex | Do not index images on the page |
unavailable_after:[date] | Stop showing the URL after a date |
indexifembedded | Allow indexing when embedded in another page, only together with noindex |
Targeting one crawler
Prefix a rule with a crawler name and a colon to aim it at that crawler only. Rules without a prefix apply to all crawlers:
X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: noindex, nofollowWhen rules conflict, Google applies the more restrictive one. A response with both max-snippet:50 and nosnippet is treated as nosnippet.
Why robots.txt and X-Robots-Tag cannot be combined on one file
This is the most common mistake. If robots.txt disallows /downloads/, Google never fetches the PDF, so it never sees the header. Google says so directly: rules in the header are only discovered when the URL is crawled, and a URL blocked by robots.txt has its indexing rules ignored. A blocked file can still be listed from links elsewhere, usually as a bare URL without a description.
So pick one job per file:
- To keep it out of results: allow crawling and send
noindex. - To stop it being fetched at all: disallow it in robots.txt, and accept that its URL can still be known.
robots.txt vs meta robots goes through the same trade-off for HTML pages.
How to set X-Robots-Tag in nginx and Apache
These follow the examples in Google's documentation.
nginx
# Every PDF on the site
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow" always;
}
# Image files
location ~* \.(png|jpe?g|gif)$ {
add_header X-Robots-Tag "noindex" always;
}
# One file
location = /files/old-price-list.pdf {
add_header X-Robots-Tag "noindex" always;
}Two nginx details catch people out, both described in the ngx_http_headers_module documentation:
- Inheritance.
add_headerdirectives are inherited from the level above only if the current level has none of its own. AddingX-Robots-Tagin alocationblock silently drops security headers you set in theserverblock. Repeat them, or move shared headers into an included file. - Status codes. Without the
alwaysparameter,add_headeronly applies to a set of success and redirect codes such as 200, 301 and 304. That rarely matters fornoindex, but it is why a header can seem to come and go while you test.
Apache
Apache needs mod_headers enabled:
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>
<Files ~ "\.(png|jpe?g|gif)$">
Header set X-Robots-Tag "noindex"
</Files>In an .htaccess file, a <Files> block for a single file name must sit in the same directory as that file.
How to set X-Robots-Tag on a CDN or host
If you do not control the server config, add the header at the edge.
Cloudflare
Use a response header transform rule with the Set static operation. The rule language has a field for the file extension, http.request.uri.path.extension, which holds the lowercased extension without the dot:
http.request.uri.path.extension eq "pdf"Set the header name to X-Robots-Tag and the value to noindex. Set replaces an existing header of the same name, while Add appends another one, per Cloudflare's response header modification docs.
Vercel
The headers property in vercel.json matches paths with a source pattern. Keeping files that should not be indexed in one folder makes the pattern simple:
{
"headers": [
{
"source": "/internal-docs/(.*)",
"headers": [{ "key": "X-Robots-Tag", "value": "noindex" }]
}
]
}The same folder approach works on most hosts with a headers file. It is also easier to audit than an extension rule, because you can see at a glance which files are covered.
Pointing a PDF at its HTML version
If a PDF duplicates an HTML page, you may prefer to keep it indexable but tell Google which version is preferred. Google's canonical documentation supports a rel="canonical" link in the HTTP header for non-HTML documents:
Link: <https://www.example.com/pricing>; rel="canonical"How to check the header is really sent
Configuration files describe intent. The response is what crawlers see. Request the file and print only the headers:
curl -sI https://www.example.com/files/old-price-list.pdf | grep -i x-robots-tagIf nothing prints, check three things in order:
- HEAD versus GET. Some servers and edge functions answer
HEADdifferently. Repeat with a GET that discards the body:curl -s -D - -o /dev/null URL. HTTP HEAD vs GET for crawler testing explains why the two can disagree. - Redirects. Add
-Lto follow them, and check the header on the final URL, not the redirect. - Cache. A CDN may still serve a cached copy without the new header. Purge the file and test again.
In Google Search Console, the URL Inspection tool shows whether indexing is allowed for a crawled URL, which confirms Google saw the rule. Removal is not instant: Google has to recrawl the file first.
What X-Robots-Tag means for AI search
For Google, the header controls Google's AI features the same way it controls classic results. Google's AI features page says that to limit the information shown from your pages in Search, you use nosnippet, data-nosnippet, max-snippet or noindex. A PDF with X-Robots-Tag: noindex is therefore out of AI Overviews and AI Mode as well, once Google has recrawled it. More on those controls in noindex and nosnippet for AI search.
Other operators are less specific:
- OpenAI documents a
noindexmeta tag for keeping pages out of ChatGPT search in its publisher FAQ, and says the crawler must be able to fetch the page to read it. It does not separately document theX-Robots-Tagheader, so do not rely on it alone for ChatGPT. - Microsoft Bing reads robots meta rules and the header, and has announced its own
ai-trainingandai-groundingpreferences, which can be sent asX-Robots-Tag: Bingbot: ai-grounding:disallow. Its announcement explains how these replace the oldernoarchiveandnocachevalues for those purposes. - Anthropic and Perplexity document robots.txt for their crawlers and do not document header rules at the time of writing.
If a file must not be read by anyone, a header is the wrong tool. Put it behind authentication.
Frequently asked questions
Can I use X-Robots-Tag on HTML pages too?
Yes. Google treats it the same as a robots meta tag. It is handy for applying noindex to a whole section from one server rule, but a meta tag is easier for editors to see and change.
Will X-Robots-Tag noindex remove a PDF that is already in Google?
Yes, after Google recrawls the file and sees the header. For something urgent, use Search Console's removal tool as a stopgap while the header takes effect.
Should I block PDFs in robots.txt as well?
No. If robots.txt blocks the file, Google cannot fetch it and never sees the noindex header, so the URL can stay listed.
Does X-Robots-Tag noindex stop AI training crawlers?
Not as a documented control. Training opt-outs are documented through robots.txt tokens such as GPTBot, ClaudeBot and Google-Extended. The header is an indexing and snippet control.
Find stray noindex headers
Headers are invisible in a browser, which is why a forgotten noindex from a staging config can sit on live files for months, as described in staging noindex leaks. Run the curl check above on a few important URLs, or run the free check on one page, with no account, to see how AI crawlers are treated there.
Filed under Access and robots.txt. Topics: meta-robots, noindex, pdf, http-status, google