Indexed, though blocked by robots.txt: what it means and the fix
Indexed, though blocked by robots.txt means Google listed a URL it may not crawl. How to find the cause, pick the right fix and get Validate fix to pass.
By Crawlito · 9 min read
In short
- Indexed, though blocked by robots.txt is a warning in the Page indexing report: Google indexed the URL from links elsewhere without being allowed to crawl it, so any snippet is probably very limited.
- There are two opposite fixes: remove the Disallow if you want the page in Google, or allow crawling and serve noindex, 404 or 410 if you want it gone.
- Keeping the Disallow while adding noindex does not work, because Googlebot never fetches the page and never sees the noindex.
- URL Inspection shows a Referring page Google possibly used to discover the URL and any Sitemaps that list it, which point you to the link behind the warning.
- Validate fix rechecks only the URLs known to have the issue and, according to Google, typically takes up to about two weeks.
On this page
- Where the warning appears and what Google says about it
- Decide first: do you want this URL in Google?
- Find what links to the blocked URL
- Find the Disallow line that matches
- Fix 1: you want the page indexed
- Fix 2: you want the page out of Google
- How Validate fix works for this warning
- Frequently asked questions
- Check the page behind the warning
"Indexed, though blocked by robots.txt" means Google has put a URL in its index even though your robots.txt does not let Googlebot crawl it. Google learned about the URL from links, kept it, and can show it in results, but with little or no description because it never read the page. The fix depends on one question: do you want that URL in Google? If yes, remove the rule that blocks it. If no, let Google crawl it and serve noindex, 404 or 410, because robots.txt cannot remove anything.
This post is the workflow for the status label itself: where it appears, when to leave it alone, how to find the link and the rule behind it, which fix applies, and how to validate. The underlying mechanism, why a Disallow hides a noindex, is covered in robots.txt vs meta robots. Facts come from Google's Page indexing report help and Search Central documentation, read on 15 November 2026.
Where the warning appears and what Google says about it
The Page indexing report in Search Console sorts URLs into Indexed and Not indexed. Reasons for not indexing go in the "Why pages aren't indexed" table. "Indexed, though blocked by robots.txt" is not there: Google lists it as a warning, and at the time of writing the help page says warnings appear in the "Improve page experience" table on the report's summary page. Google describes warnings as issues that don't prevent a page from being indexed, so the URLs listed here are in the index.
Google's entry for the status makes three points:
- The page was indexed despite the robots.txt block. Google always respects robots.txt, but that does not necessarily prevent indexing when other pages link to the URL: Google can index it using information from those links.
- Any snippet shown for it in results will probably be very limited.
- robots.txt is not the correct mechanism to keep a page out of the index. To block it, remove the robots.txt rule and use
noindex. To include it, change robots.txt so it is no longer blocked.
Do not confuse it with its neighbour, which Google's help page calls "URL blocked by robots.txt". It sits under "Why pages aren't indexed" and covers blocked URLs that are not in the index. That status is often expected and harmless. Finding and fixing the rule behind it is covered in Blocked by robots.txt in Search Console.
| Status | Indexed? | What Google read | Usual meaning |
|---|---|---|---|
| Indexed, though blocked by robots.txt | Yes | Links to the URL, not the page | A blocked URL is linked from somewhere |
| URL blocked by robots.txt | No | Nothing | Your rule is doing its job, or blocking a page by mistake |
| URL marked 'noindex' | No | The page, including its noindex | The removal you asked for has worked |
Decide first: do you want this URL in Google?
Export the example URLs from the issue page (the table shows up to 1,000 rows) and sort them into two piles before you touch robots.txt. The two fixes are opposites, so applying one to the whole list breaks half of it.
| URL type | Usually want it in Google? | Action |
|---|---|---|
| Product, article, category or landing page | Yes | Remove or narrow the Disallow |
| Page blocked by an old or copied rule | Yes | Remove the rule |
| Thank-you, account, staging or test page | No | Allow crawling, serve noindex or require login |
| Deleted or retired page | No | Allow crawling, return 404 or 410 |
| Cart, checkout or session URL | No | Often fine to leave, see below |
| Internal search results and filter parameters | No | Often fine to leave, see below |
When to leave the warning alone
Many sites block /cart, /search or ?sort= URLs on purpose, to stop Googlebot spending requests on endless combinations. If a handful of those get linked from somewhere and indexed, the warning is the price of that crawl control. Leaving it is reasonable when all of these hold:
- The URLs are low-value variants, not pages you want found.
- The count is small and not growing month to month.
- They do not show up for searches that matter to you, such as your brand name.
If the count keeps climbing, or a URL appears in results where people see it, treat it as the "do not want it" case. For the wider trade-offs, see internal search results pages.
Find what links to the blocked URL
The block is only half the cause. The other half is a link, and until it is gone Google keeps rediscovering the URL. Inspect one example URL in Search Console with URL Inspection and read the page indexing details:
- Crawl allowed? confirms the robots.txt block.
- Referring page is a page Google possibly used to discover the URL. It is often an internal link, a template link or a URL in someone else's content.
- Sitemaps lists sitemaps submitted in Search Console or referenced in robots.txt that contain the URL. A blocked URL in your own sitemap is a contradiction you control: either you want it crawled or you do not.
Then search your own site for the pattern. A quick check on a sitemap and a template:
# Does the sitemap list any blocked cart or search URLs?
curl -s https://example.com/sitemap.xml | grep -Eo '<loc>[^<]*(/cart|/search|\?sort=)[^<]*</loc>'
# Does the homepage link to them?
curl -s https://example.com/ | grep -Eo 'href="[^"]*(/cart|/search|\?sort=)[^"]*"'External links you cannot remove are the reason robots.txt alone never fixes this warning. Only a page-level answer from your server, noindex, 404 or 410, overrides them, and Googlebot has to be allowed to fetch the URL to receive it.
Find the Disallow line that matches
Before you edit, identify the exact rule. The full method (which group Googlebot follows, then which rule inside it) is in Blocked by robots.txt in Search Console. The short version from Google's robots.txt specification: the rule with the longest matching path wins, and on a conflict Google uses the least restrictive rule. Length is counted on the rule as written, wildcards included, which catches people out with parameter rules:
User-agent: *
Disallow: /shop/
Allow: /shop/products/
Disallow: /*?sort=Here /shop/basket is blocked by Disallow: /shop/, but /shop/products/blue-mug?sort=price is allowed: Allow: /shop/products/ (15 characters) is longer than Disallow: /*?sort= (8 characters). If you meant to block sort URLs everywhere, a short wildcard rule will not do it under a longer Allow.
URL Inspection's Crawl allowed? field tells you whether Google treats a URL as blocked, not which line blocks it. To test rules locally the way Google parses them, run Google's open-source robots.txt parser; testing robots.txt against each crawler shows the commands.
Fix 1: you want the page indexed
Remove or narrow the rule so Googlebot can fetch the page.
Before: blocks the whole shop, products included.
User-agent: *
Disallow: /shop/After: blocks only the basket and checkout.
User-agent: *
Disallow: /shop/basket
Disallow: /shop/checkoutThen:
- Confirm the live file with
curl -s https://example.com/robots.txt. Google generally caches robots.txt for up to 24 hours. - In Search Console's robots.txt report, open the menu next to the file and choose Request a recrawl. Google suggests this after unblocking important URLs.
- Run a live test in URL Inspection on one affected URL. Crawl allowed? should now say yes.
- Make sure the page returns
200and carries nonoindex, or you will swap this warning for another status.
Fix 2: you want the page out of Google
Here the instinct to add a stronger block is wrong. The order matters:
- Remove the Disallow for those URLs, so Googlebot can fetch them.
- Give each URL a page-level answer:
noindexif the page should stay live for users,404or410if it no longer exists. Google's noindex guide says the page must not be blocked by robots.txt for the rule to be seen. - Remove the URLs from your sitemaps and internal links, so you stop pointing Google at them.
- Let Google recrawl them. Validate fix (below) queues the affected URLs for recrawling. Google's noindex guide warns that, depending on a page's importance, Googlebot may take months to revisit it.
For parameter URLs that have no template to edit, send the rule as a header. In nginx:
# Let Googlebot fetch filter URLs, but keep them out of the index
location /shop/ {
# Note: add_header here stops add_header lines from outer levels being inherited
if ($arg_sort) {
add_header X-Robots-Tag "noindex" always;
}
# ...your existing handling for /shop/ stays here
}
# Retired section: tell crawlers it is gone for good
location /old-campaign/ {
return 410;
}The response a crawler should now get for a filter URL:
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
X-Robots-Tag: noindexShould you put the Disallow back once the URLs drop out? Usually not. If links to them remain, a new Disallow hides the noindex again and the URLs can return as "Indexed, though blocked by robots.txt". Re-add it only for crawl-load reasons, and accept the warning may come back. If a URL must disappear from results today while you wait, a temporary removal hides it for a while but is not a fix on its own; see temporary removals and the permanent fix.
How Validate fix works for this warning
Once every URL in the list has one of the two fixes, open the issue in the Page indexing report and click Validate fix. Google's help page says:
- Only URLs with known instances of the issue are queued for recrawling, not your whole site.
- It typically takes up to about two weeks, sometimes much longer.
- The request moves through states such as Started, Looking good, Passed and Failed, and each URL is marked Pending, Passed, Failed or Other.
- Do not click Validate fix again until validation has succeeded or failed.
What "fixed" looks like differs by path. With fix 1, URLs leave the warning and become plain indexed pages. With fix 2, they move to a Not indexed reason such as "URL marked 'noindex'" or "Not found (404)" (the names Google's help page uses), which is the outcome you wanted. Validation fails when checked URLs still have the issue, which usually means a URL was in the wrong pile or a second rule still matches. Why Search Console validation fails covers the common causes.
The same logic reaches beyond classic results. Google's AI features guide says robots.txt rules for Googlebot are the crawl control for AI Overviews and AI Mode, and that noindex and snippet rules control what appears there. OpenAI's crawler documentation says a site that opts out of OAI-SearchBot can still appear in ChatGPT as a navigational link. A blocked page is not necessarily a removed page.
Frequently asked questions
Is "Indexed, though blocked by robots.txt" bad for SEO?
It is a warning, not a reason a page was left out: Google describes warnings as issues that don't prevent a page from being indexed. It matters when the URL is a page you want found, because it is listed with a very limited snippet, or one you want hidden, because it is still listed.
Why did Google index a page I blocked in robots.txt?
robots.txt stops crawling, not indexing. Google found links to the URL, from your site, your sitemap or other sites, and indexed it from that information without fetching the page.
Can I fix it by adding noindex and keeping the Disallow?
No. Googlebot never fetches a disallowed page, so it never sees the noindex. Remove the Disallow first, then serve noindex, 404 or 410.
Should I fix this warning for cart and internal search URLs?
Not necessarily. If you blocked them to save crawl requests and only a few low-value URLs are listed, leaving the warning is a reasonable trade. Act if the count keeps growing or the URLs show up for searches that matter.
How long does it take for the warning to clear?
Google says Validate fix typically takes up to about two weeks, and sometimes much longer. Its noindex guide adds that, depending on a page's importance, Googlebot may take months to revisit it.
Check the page behind the warning
Before you choose a fix, confirm what a crawler actually gets from one affected URL: the robots.txt rule, the status code and any robots meta tag or header. The free check reports those for a single page with no account. If the list spans hundreds of URLs across templates, an audit reviews robots.txt and page-level rules together across the site.
Filed under Access and robots.txt. Topics: robots-txt, search-console, google, noindex, troubleshooting