noindex detected in X-Robots-Tag HTTP header: find the source
Search Console says noindex detected in the X-Robots-Tag HTTP header but your HTML is clean. Isolate the CDN, origin, user agent or cache layer adding it.
By Crawlito · 9 min read
In short
- The message means the response Google fetched carried an X-Robots-Tag header with noindex, so the page source can be completely clean and the rule still applies.
- URL Inspection shows the reason under Indexing allowed? and the date of the crawl behind it, and the live test fetches the page again with Google-InspectionTool rather than Googlebot.
- curl --resolve sends the request for your real hostname to an address you choose, so you can ask the origin server directly and see whether the CDN adds the header.
- Change one thing per request (user agent, path, host, cache state, HEAD or GET) and the layer that adds the header shows itself.
- On feeds, API endpoints, private PDFs and preview hosts the header is often deliberate, and Google's Page indexing help says to remove it only if you want the page indexed.
On this page
- What the message does and does not tell you
- Step 1: read what Google recorded
- Step 2: reproduce Google's request at the public edge
- Step 3: split the CDN from the origin with curl --resolve
- Step 4: change one variable at a time
- When the header is correct and needs no fix
- After the fix: confirm it before asking Google
- Frequently asked questions
- Check your live headers
"noindex detected in 'X-Robots-Tag' http header" in Google Search Console means the response Google fetched for that URL carried an X-Robots-Tag header containing noindex, so Google left the page out of its index. The rule is in the HTTP headers, not the HTML, which is why viewing the page source shows nothing. To find it, compare what Google reports with what your CDN and your origin server each return, changing one variable at a time until the header appears.
This post is the isolation procedure. For header syntax and how to set the header on purpose, see X-Robots-Tag for PDFs and other files. For the usual staging and host defaults that cause it, see staging noindex leaks.
What the message does and does not tell you
Google's robots meta tag documentation says any rule valid in a robots meta tag can be sent as an X-Robots-Tag header, that several headers can be combined in one response, and that header name, user agent name and values are not case sensitive. So all of these produce the same message:
X-Robots-Tag: noindex
x-robots-tag: noindex, nofollow
X-Robots-Tag: none
X-Robots-Tag: googlebot: noindexThe last line matters for debugging. A rule prefixed with googlebot: applies only to Googlebot, so a tool that checks for a plain noindex may miss it. Google also says that when rules conflict the more restrictive one applies, so a noindex in the header still applies when the HTML meta tag says index or all.
What the message does not tell you is which layer added the header, or whether the crawl it describes is recent. The URL Inspection help page separates the indexed result, which comes from the most recently indexed version of the page, from the live test, which fetches the URL in real time. A page fixed last week can still show the old indexed result until Google recrawls it.
Step 1: read what Google recorded
Start with Google's side, because everything else is a reproduction of it.
- Inspect the URL in Search Console and expand the Page indexing details. Indexing allowed? shows the reason indexing was disallowed, Last crawl shows when the crawl behind it happened and Crawled as shows the crawler type. Compare that date with your recent deploys and CDN changes.
- Click Test live URL. The live test fetches the page now, but with Google-InspectionTool, whose user agent strings do not contain the word Googlebot, according to Google's common crawlers page.
- If View crawled page or View tested page opens, read the HTTP headers and write down the
x-robots-tagline and any cache or CDN headers next to it, such ascf-cache-status,server,viaorx-cache. Google's help documents this response data as available only when the status is "URL is on Google" (indexed result) or "URL is available to Google" (live test), with or without issues, so for a page blocked bynoindexyou may not get it. In that case your own requests in step 2 are the evidence.
The two results lead to different paths:
| Indexed result | Live test | What it means |
|---|---|---|
| noindex detected | noindex detected | The header is still being served. Go to step 2. |
| noindex detected | Indexing allowed | It was fixed, it appears only some of the time, or it depends on the user agent. Check cache and user agent rules before requesting indexing. |
| Indexing allowed | noindex detected | Something changed recently. Check recent deploys and CDN rules. |
The middle row needs care. A rule that adds the header only when the user agent contains "Googlebot" will hit the real crawl and miss the live test, so a clean live test is not proof of a fix. Google-InspectionTool explains how the two crawlers differ.
Step 2: reproduce Google's request at the public edge
Now ask the same question from your own machine. Use a GET request that throws the body away, so you see the headers a real page load gets:
UA='Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
curl -s -o /dev/null -D - -A "$UA" https://www.example.com/page/ \
| grep -i -E '^(HTTP|x-robots-tag|cf-cache-status|server|via|x-cache)'-D - prints the received headers, -o /dev/null discards the body and -A sets the user agent. Replace W.X.Y.Z with a current Chrome version: Google's crawler overview says that part of the string is a placeholder for the Chrome version the crawler uses. The full list is in Googlebot user agent strings.
Run it again with your browser's user agent, and once with no -A at all, which sends curl's default curl/VERSION user agent. Three results tell you a lot:
- Header with every user agent: a site-wide or path rule. Go to step 3 to find out whether the CDN or the origin adds it.
- Header only with the Googlebot user agent: something is branching on the user agent. Look at bot rules, Workers or edge functions, and server code that checks for crawlers.
- No header with any user agent, but Google reports one: the difference is something you cannot copy with a user agent alone, such as Google's IP addresses, its location or a cached copy Google hit. Google says its crawler's default IP addresses appear to be in the USA and that it sends requests without an
Accept-Languageheader, per its locale-adaptive pages guide.
A spoofed Googlebot user agent from your laptop is not Googlebot. A CDN that verifies crawlers by IP may block or challenge it, which shows up as a 403 or a challenge page rather than a missing header.
Step 3: split the CDN from the origin with curl --resolve
This is the test that settles most cases. The curl manual describes --resolve as providing a custom address for a specific host and port pair instead of the normally resolved one, "a sort of /etc/hosts alternative" on the command line. The URL keeps your real hostname, but the connection goes straight to the IP address you choose and skips the CDN.
# 203.0.113.10 is your origin server's IP, not the CDN's
curl -s -o /dev/null -D - -A "$UA" \
--resolve www.example.com:443:203.0.113.10 \
https://www.example.com/page/ | grep -i -E '^(HTTP|x-robots-tag)'Compare the two runs:
| Through the CDN | Direct to origin | The header comes from |
|---|---|---|
| Present | Present | The origin: web server config, application, CMS or plugin |
| Present | Absent | The CDN: a response header rule, Worker, edge function or a cached copy |
| Absent | Present | The origin, and the CDN strips or overrides it for some requests |
Three practical snags:
- The origin only accepts CDN traffic. If the firewall drops your IP, run the same command on the server itself with
--resolve www.example.com:443:127.0.0.1. - The certificate does not match. Origins behind a CDN often use a certificate only the CDN trusts. Add
-k, which the curl manual says skips the certificate verification step, for this test only. - The port differs. The manual says the port should be the one used for the protocol, so use
:443with anhttps://URL and:80with anhttp://URL.
Once you know the layer, the search gets small. At the origin, look for X-Robots-Tag in the nginx or Apache config, .htaccess, framework header config and SEO plugin settings. At the CDN, look at response header rules, Workers and edge functions first.
Step 4: change one variable at a time
If the header still comes and goes, test each condition separately. Keep the layer fixed (CDN or origin) and vary one thing per request.
| Variable | How to test it | What a difference points to |
|---|---|---|
| Path | Compare /page/, /page, a sibling page and the homepage | A rule scoped to a folder, file type or pattern |
| Host | Compare www, the bare domain and any other hostname serving the page | A host-specific server block or a preview hostname |
| Cache state | Read cf-cache-status or x-cache, then add a throwaway query string such as ?nocache=1 | A cached response stored while the header was on |
| Method | Compare curl -I (a HEAD request) with the GET command above | Code that answers HEAD and GET differently |
| Response body | Check that the request got the real page, not a maintenance, login or challenge page served with a 200 | A fallback page that carries its own headers |
The cache row needs care. Cloudflare's cache response documentation says HIT means the resource was found in Cloudflare's cache, and STALE, UPDATING and REVALIDATED also describe a cached copy being served, while MISS and EXPIRED were served from the origin and DYNAMIC means the response was not eligible for cache. A header that appears only on cached responses was most likely stored before someone removed it. Purge that URL and test again. A query string may be cached as a separate copy or ignored, depending on your cache settings, so treat a clean result with a query string as a hint, not proof.
When the header is correct and needs no fix
Not every report is a mistake. Google's Page indexing report help describes the status as Google finding a noindex directive and says to remove it if you do want the page indexed. If you do not, leave it. These are common, deliberate cases:
- Feeds such as
/feed/or/rss.xml: you want them fetched for discovery, not listed as search results. - API endpoints and JSON that a crawler found through a link or script.
- PDFs and other files you keep public but do not want in results, set as in the X-Robots-Tag guide.
- Preview and staging hosts. Cloudflare's preview deployments page says every Pages preview deployment includes
X-Robots-Tag: noindexby default. If the reported URL is on a preview hostname, the header is working as designed. Cloudflare Pages noindex headers covers that platform in detail.
Before changing anything, check the exact hostname and path in the report. A URL on a preview or deployment host, or one you never meant to publish, is a reason to find out where Google discovered it, not to remove the header.
After the fix: confirm it before asking Google
- Repeat the CDN and origin commands on several URLs of the same type, with both the Googlebot and a browser user agent. The header should be gone from every run.
- Purge the CDN cache for the affected paths, then check that
cf-cache-statusorx-cacheshows a fresh response without the header. - Run a live test in URL Inspection, read the headers in View tested page if it is available, and request indexing for the key URLs. Google says a request does not guarantee indexing.
- If many URLs were affected, use Validate fix in the Page indexing report. Google's help says validation typically takes up to about two weeks. Why validation fails explains what ends a run early.
Google's block indexing guide says Google has to crawl a page to see its meta tags and HTTP headers, and that it may take months for Googlebot to revisit some pages, so the report only clears after a recrawl.
The same header matters for Google's AI features. Google's AI features documentation says a page must be indexed and eligible to be shown in Search with a snippet to be a supporting link in AI Overviews or AI Mode. Removing a stray noindex takes away that barrier; it does not guarantee the page will be shown.
Frequently asked questions
Why does view source show no noindex when Search Console reports one?
View source shows the HTML only. X-Robots-Tag is an HTTP response header, so you need the Network panel in browser developer tools, the headers in URL Inspection when they are available, or curl -D - to see it.
Why is the live test clean when the indexed result says noindex?
Either the fix happened after Google's last crawl, or the header depends on the request. The live test uses Google-InspectionTool, whose user agent does not contain "Googlebot", so a rule that targets Googlebot by user agent can skip it.
Does googlebot: noindex in the header affect other crawlers?
Google's documentation says a rule with a user agent prefix applies to that crawler, and rules without a prefix are valid for all crawlers. Other search engines document their own support separately.
How long does the report take to clear after a fix?
It clears only after Google recrawls the URL, which Google says can take months for some pages. Request indexing for key URLs, and expect a Validate fix run to take up to about two weeks.
Should I remove X-Robots-Tag noindex from my RSS feed?
Only if you want the feed itself to appear in search results, which is rarely useful. A noindex rule does not block crawling, so Google can still fetch the feed; asking it not to follow links is the separate nofollow rule.
Check your live headers
The fastest next step is to see what a crawler gets from the page in question. The free check fetches one public page, with no account, and reports its status, X-Robots-Tag and meta robots. If the header shows up on some page types and not others, an audit checks up to 500 pages so you can see which templates or paths carry it.
Filed under Access and robots.txt. Topics: noindex, meta-robots, search-console, curl, troubleshooting