Skip to content

Blocked by robots.txt in Search Console: when to fix it

Blocked by robots.txt in Search Console is often intended. How to sort the list, find the exact rule Googlebot follows, fix the ones that matter and revalidate.

By Crawlito · 9 min read

In short

  • URL blocked by robots.txt is a Not indexed reason in Search Console's Page indexing report: Googlebot was not allowed to crawl the URL, so it did not read the page.
  • The status is often intended, for carts, admin areas and internal search, and only needs fixing when the URL is one you want in Google.
  • Setting the report's sitemap filter to All submitted pages shows blocked URLs you also listed in a sitemap, which is a contradiction you should resolve one way or the other.
  • Googlebot follows only its most specific matching group, and inside it the longest matching rule wins; on a tie between Allow and Disallow, Google uses the less restrictive rule.
  • The robots.txt report keeps a 30-day history of fetches whose content or result changed, so you can see when a blocking rule went live and request a recrawl after fixing it.
On this page

"URL blocked by robots.txt" in Google Search Console means Googlebot found the URL but your robots.txt did not let it crawl it, so Google never read the page. That is often exactly what you wanted: carts, account pages and internal search results should be blocked. It only needs fixing when the list contains pages you want in Google, such as URLs from your sitemap, whole content sections or the files a page needs to render. The work is in sorting the list, finding the one rule behind each unwanted block, and changing it.

This post is the Search Console workflow. If a blocked URL is still in the index, you are looking at a different status, covered in Indexed, though blocked by robots.txt. For testing the file against other crawlers, see how to test robots.txt against each AI crawler. Facts below come from Google's Page indexing report help, its robots.txt report help and the robots.txt specification, read on 15 November 2026.

What the status means, and what it does not

In the Page indexing report, URLs are split into Indexed and Not indexed. "URL blocked by robots.txt" is one of the Not indexed reasons, next to statuses such as "URL marked 'noindex'" and "Discovered - currently not indexed". Google's description is short: the page was blocked by your site's robots.txt file.

Three points follow from how Google treats robots.txt:

  • It is a crawl block, not an index block. Google's robots.txt introduction says Google cannot index the content of a disallowed page but may still index the URL. If that happens, Search Console reports it as "Indexed, though blocked by robots.txt", a warning, instead.
  • It is not security. RFC 9309 says robots.txt rules "are not a form of access authorization". Anything private needs a login, not a Disallow.
  • It matters for AI features too. Google's AI features documentation says a page must be indexed and eligible for a snippet to appear as a supporting link in AI Overviews or AI Mode. A page Googlebot cannot crawl has no content for that.

Sort the list: intended blocks and accidental ones

Open the status from the list of Not indexed reasons and export or copy the example URLs. Google notes the table is limited to 1,000 rows, so on a large site you see a sample, not everything. Then group the URLs by path pattern. Most lists collapse into a handful of patterns, and each pattern falls on one side of this table:

Usually intendedUsually accidental
/cart/, /checkout/, /account/URLs that are also in your sitemap
/wp-admin/, /admin/, /loginA whole content section such as /blog/ or /products/
Internal search results, such as /search?q=Category or product pages caught by a broad wildcard
Filter and sort parameters, such as ?sort= or ?color=Pages on a host that copied a staging Disallow: /
Thank-you and confirmation pagesScript and style files the page needs to render

If every pattern sits in the left column, you are done: the report is confirming your rules work. There is nothing to validate and no reason to remove the rules. Blocking internal search results and faceted filter URLs is a deliberate way to keep Googlebot's attention on real pages.

Check the sitemap contradiction first

The fastest way to find accidental blocks is the sitemap filter at the top of the Page indexing report. Google offers three choices: All known pages (the default), All submitted pages and Unsubmitted pages only. Switch to All submitted pages and look at "URL blocked by robots.txt" again.

Anything left is a URL you told Google you want crawled, through a sitemap, while your robots.txt tells it not to crawl. Google's sitemap guide says to include the URLs "that you want to see in Google's search results". A blocked URL cannot meet that goal, so one of the two signals is wrong:

  • If the page should be in Google, the robots.txt rule is wrong. Change it.
  • If the page should not be in Google, the sitemap is wrong. Remove the URL from the sitemap generator, not by hand, or it will come back.

Those URLs are the top of your fix list.

Do not forget rendering resources

The Page indexing report lists pages, so a blocked stylesheet or script rarely shows up there. It still matters: the example file in Google's robots.txt introduction keeps CSS and JavaScript crawlable because Google needs them for rendering. Look for asset paths in your file, such as /wp-content/, /_next/, /assets/ or rules ending in .js$ or .css$. The separate problem of blocked CSS and JavaScript covers what breaks and how to allow them.

Confirm the block in URL Inspection

Take one URL from each accidental pattern and inspect it. In the Page indexing section of the URL Inspection result, the Crawl allowed? field shows whether your site lets Google crawl the page or blocks it with a robots.txt rule. If it says No, the robots.txt rule is the cause.

Two other fields help with the sorting:

  • Sitemaps lists the sitemaps submitted in Search Console, or named in robots.txt, that point to the URL. It confirms the sitemap contradiction for that one URL.
  • Referring page shows a page Google may have used to discover the URL. For a URL that should never have been found, such as an odd parameter combination, this is where the link comes from.

Click Test live URL as well. The live test has its own Crawl allowed? field, in its Availability section, that reflects whether your site will let Google crawl the page now, so it tells you whether a rule you already changed lets the page through. The live test does not check sitemaps or referring pages. What URL Inspection does not show is which line blocked the URL, so finding the line is a manual job, and the next section does it.

Find the rule that blocks the URL

Google applies two steps, and reading the file top to bottom gets both wrong.

Step 1: pick the group. Googlebot follows one group only: the one whose user agent most specifically matches it. A User-agent: Googlebot group beats User-agent: *, and when Googlebot has its own group, the * rules no longer apply to it at all. Groups naming the same crawler are merged; a specific group and * are never combined. The robots.txt groups guide explains this in detail.

Step 2: pick the rule inside that group. The rule with the longest matching path wins, wherever it sits in the file. If an Allow and a Disallow match with the same length, Google uses the less restrictive rule, which is the Allow. RFC 9309 says the same.

Here is a file that produces a confusing report:

User-agent: *
Disallow: /search
Disallow: /*?

User-agent: Googlebot
Disallow: /admin/
Disallow: /*.pdf$
Allow: /blog/
Disallow: /blog/drafts

And how Googlebot reads it for five URLs:

URLGroup Googlebot usesWinning ruleResult
/search?q=shoesGooglebotNone matchAllowed (the * rules do not apply)
/blog/postGooglebotAllow: /blog/Allowed
/blog/drafts-and-notesGooglebotDisallow: /blog/drafts (longer)Blocked
/blog/guide.pdfGooglebotDisallow: /*.pdf$ (longer than /blog/)Blocked
/products/shoe?color=redGooglebotNone matchAllowed

Two lessons come out of this. First, the * group's search and parameter blocks do nothing for Googlebot once it has its own group, so Googlebot is free to crawl internal search results you meant to block. Second, a path rule without a trailing slash, /blog/drafts, also blocks /blog/drafts-and-notes, which is a common way to block more than intended. Wildcard length is counted on the rule as written: Google's own example says disallow: /*.htm beats allow: /page for /page.htm because the rule path is longer. Paths are case-sensitive, so /Blog/ and /blog/ are different rules.

To check your conclusion mechanically, run the URL and the Googlebot token through Google's open-source parser, as shown in the testing guide linked above.

Check the version Google has, then fix and recrawl

Before editing, confirm Google is reading the file you think it is. The robots.txt report lists the robots.txt files Google found (in a Domain property, for its top 20 hosts), with Fetch status, Checked on, size and an issue count. It works only for Domain properties and URL-prefix properties without a path.

Click a file to see the last version Google fetched, with any errors and warnings highlighted. Then click Versions: the report lists every fetch from the last 30 days whose content or result differed from the one before. That history answers a question URL Inspection cannot: when did the blocking rule go live? If the "URL blocked by robots.txt" count jumped on a date that matches a new version, you have found the deploy that caused it.

Also look at the issue count. Google ignores lines it cannot parse, and errors stop a rule from being used. A malformed line can mean a rule you rely on is not applied.

Then fix and tell Google:

  1. Change the rule. Narrow a broad Disallow, add a trailing slash, or add a longer Allow for the section you want crawled.
  2. Remove any URLs you have decided to keep blocked from your sitemap.
  3. Fetch the live file and confirm the change is there: curl -s https://www.example.com/robots.txt.
  4. In the robots.txt report, open the menu next to the file and choose Request a recrawl. Google says this is usually unnecessary, because it recrawls robots.txt often, and that it does not guarantee an immediate recrawl of the unblocked URLs.
  5. Run Test live URL in URL Inspection on a fixed URL. Crawl allowed? should now say yes.
  6. Back in the Page indexing report, open "URL blocked by robots.txt" and click Validate fix. Google says validation typically takes up to about two weeks.

Google generally caches robots.txt for up to 24 hours, and longer if it cannot refresh the file because of timeouts or 5xx errors, so a change is not visible straight away. Other crawlers have their own timing, collected in how long robots.txt changes take for AI crawlers.

When to leave the status alone

Validation only makes sense for blocks you removed. If the remaining URLs are carts, admin pages, internal search and filter combinations, leave them in the report. A large count is not a problem in itself; Google is telling you what it found and did not crawl.

Two cases deserve a second look even when the block is intended:

  • You want a page out of Google entirely. A Disallow will not do that, because Googlebot never sees a noindex on a page it cannot fetch. Allow crawling and serve noindex, as explained in robots.txt vs meta robots.
  • The block hides something private. Move it behind authentication. Listing a path in robots.txt also tells anyone who reads the file that it exists.

Frequently asked questions

Is "URL blocked by robots.txt" an error?

No. Google lists it as a reason a page is not indexed, and for carts, admin and internal search it is the expected result. Fix it only when the URL is one you want in Google.

Why are URLs from my sitemap blocked by robots.txt?

Your sitemap and robots.txt disagree. Use the All submitted pages filter to list them, then either change the rule or remove those URLs from the sitemap.

Can Search Console show which robots.txt line blocks a URL?

URL Inspection shows that robots.txt blocks the URL in its Crawl allowed? field, but not which line. Work it out from the group Googlebot follows and the longest matching rule, or run Google's open-source parser.

How long after fixing robots.txt does the status clear?

Google generally caches robots.txt for up to 24 hours, and you can request a recrawl in the robots.txt report. Validate fix then typically takes up to about two weeks, according to Google.

Does a robots.txt block in Google also block AI crawlers?

Only if the rule applies to them. Each crawler follows the group that names it, or * if none does, so a User-agent: Googlebot group says nothing to OAI-SearchBot or ClaudeBot.

Check the rule for every crawler at once

Search Console answers the question for Googlebot. The free check reads one page and reports the robots.txt rules that apply to each named AI crawler, with no account needed.

Filed under Access and robots.txt. Topics: robots-txt, search-console, google, troubleshooting