Skip to content

Blocked due to access forbidden (403): fix Googlebot blocks

Search Console says 'Blocked due to access forbidden (403)'? Find the firewall, CDN, plugin or server rule that refuses Googlebot, verify it and fix it safely.

By Crawlito · 10 min read

In short

  • Google's Search Console documentation says Googlebot never sends credentials, so a 403 means your server answered Googlebot with a refusal it should not have, and the page will not be indexed.
  • Google treats 401 and 403 like other 4xx errors: the content is ignored and a URL that was already indexed is removed from the index.
  • Search Console does not tell you which layer returned the 403, so check your CDN or WAF security events first, then your plugins and web server logs.
  • Allow Googlebot by verified identity, not by user agent alone: Google documents a reverse DNS check and published IP range files for this.
  • Validation in Search Console typically takes up to about two weeks, can take much longer, and Google does not promise that a fixed page will be indexed.
On this page

"Blocked due to access forbidden (403)" means Googlebot asked for your page and your server answered 403 Forbidden. Google's Search Console documentation says Googlebot never provides credentials, so the 403 is a refusal your site should not be giving it, and the page will not be indexed. The cause is usually a layer in front of your content: a firewall or CDN rule, a security plugin, or a web server rule. This guide shows how to confirm the 403, find the layer, allow Googlebot safely and ask Google to check again.

A short caveat first. Fixing the block lets Google fetch the page. Whether Google then indexes it, and where it appears, depends on things outside this guide.

What does the Search Console 403 status mean?

In the page indexing report help, Google defines it this way: HTTP 403 means the user agent provided credentials but was not granted access. It adds that Googlebot never provides credentials, so your server is returning this error incorrectly. Its recommended fixes are to let non-signed-in users access the page, or to explicitly allow Googlebot requests without authentication, ideally after verifying its identity.

Two related statuses are easy to mix up:

Status in Search ConsoleWhat Google says it isWhere to look
Blocked due to access forbidden (403)The server refused GooglebotFirewall, CDN, plugin, server rules
Blocked due to unauthorized request (401)The page asked Googlebot for authorizationLogin or basic auth on the page
Server error (5xx)Googlebot could not access the URL, the request timed out, or the site was busyServer health; Google also says to check that firewall or DoS protection is not blocking Googlebot

The consequence is spelled out in Google's HTTP status code documentation: Google's crawlers treat 401 and 403 like other 4xx errors except 429. The content is ignored, a URL that was already indexed is removed from the index, and the crawling frequency gradually decreases. Google also says not to use 401 or 403 to limit the crawl rate. If you want to slow a crawler down, see rate limiting AI crawlers, which covers 429 and 503. The same status-code rules are summarised for AI crawlers in HTTP status codes and redirects.

Why does the page work in your browser but return 403 to Googlebot?

Google does not publish a list of causes, so what follows is the usual explanation rather than a documented one. Your rules treat a human browser and Googlebot differently, and nobody notices because everyone on your team tests with a browser. The common triggers are:

  • A bot protection or rate limit rule that matches Googlebot's user agent, its request rate, or its IP range.
  • A country or hosting-network block. Googlebot's published ranges are Google IPs, so rules that block by network can catch it. Geo blocking crawlers covers how these rules misfire.
  • A security plugin or host-level firewall that blocks "suspicious" user agents.
  • A password, basic auth or login left on a section after a staging launch or migration.
  • A server rule, such as a deny directive or a rewrite, that matches the paths Googlebot requests.

Cloudflare and other WAFs can also answer with a challenge page rather than a plain block. WAF challenge pages and AI crawlers shows how to tell the two apart.

How to confirm the 403 with the URL Inspection live test

Start in Search Console, because that shows what Google itself receives.

  1. Open the URL Inspection tool and paste the affected URL.
  2. Select Test live URL. Google's URL Inspection documentation says this fetches the page in real time and that the page must be reachable without a login.
  3. Look at Page fetch, which shows whether Google could get the page from your server. A failure on the live test points to a problem that exists now, not only in the old crawl. Google's page does not say which label a 403 gets here. The indexed result reflects the last crawl, not the live page.
  4. If the live test fetches the page, use View tested page to read the HTML and HTTP headers Google received. Google makes this available only when the live result is "URL is available to Google" or "available, but has issues".

Treat a passing live test as a good sign, not proof. Google notes that server errors can be transient, so a live test can pass even when a crawl failed, and the test does not guarantee the page shows up in Search. A single passing test cannot rule out a rule that depends on request rate or on a pattern over many requests.

Next, size the problem. The Crawl Stats report groups Googlebot's responses by type and marks 401/407, 5xx and other 4xx as bad. It is available only for root-level properties, meaning a Domain property or a URL-prefix property at the site root. Google's page does not name 403, so check the 401/407 and other 4xx rows. If the same pattern appears across many URLs, suspect a site-wide rule. If it affects one section, suspect a path-based rule or a login.

How to find the rule that returns the 403

Search Console cannot tell you which layer answered. Work from the outside in, because an outer layer hides the inner ones.

Check your CDN or WAF security events first

If the site sits behind Cloudflare, Cloudflare's security events let you investigate requests that its security products acted on or flagged. At the zone level, open Analytics, then the Events tab, and filter by user agent, IP address or action. Look for events on your site from Googlebot's user agent and note which service and rule produced the action. The Cloudflare AI crawler settings guide covers the other Cloudflare settings that can block crawlers.

On other CDNs and WAFs, use the equivalent firewall or bot log and filter on the Googlebot user agent. Note that a request blocked at the CDN never reaches your origin, so your server logs will show nothing.

Then read your origin access logs

If the CDN shows no block, the origin is refusing. Find Googlebot requests that got a 403:

bash
awk '$9 == 403' access.log | grep -i googlebot | tail -20

This assumes the common combined log format, where the status code is the ninth field. Adjust it if your format differs. The same approach for counting by crawler is in AI crawlers in server logs. If the 403 lines are there, check the web server's error log for the same timestamps, then your application and plugins.

Reproduce it with curl

You can test whether a user-agent rule is involved by sending the Googlebot token yourself:

bash
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/page
curl -s -o /dev/null -w "%{http_code}\n" -A "Googlebot" https://example.com/page

If the second command returns 403 and the first returns 200, a rule matches the user agent. If both return 200, the rule probably depends on something curl does not copy, such as the IP address or request rate. Google's own live test is the better evidence in that case. Googlebot has Smartphone and Desktop subtypes, and most crawl requests come from Smartphone, so match on the Googlebot token rather than one full string.

Isolate the layer

If you still cannot tell, switch off one layer at a time in a safe window and rerun the live test: the security plugin, the host's firewall, then the CDN's bot features. Re-enable each one afterwards.

How to check that the 403 is not an impostor

Before you write an allow rule, make sure the traffic you are allowing is real. Google's Googlebot overview says other crawlers often spoof the user agent, so verify a request before blocking or allowing it. Verify real AI crawlers covers the other operators too. For Google there are two documented methods, described in verifying Googlebot.

Reverse DNS, for single lookups. Run a reverse lookup on the IP from your log. The hostname must be in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that hostname and confirm it returns the original IP.

bash
host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

Google gives crawl-***-***-***-***.googlebot.com and geo-crawl-***-***-***-***.geo.googlebot.com as the expected hostname patterns for common crawlers like Googlebot. It calls reverse DNS sufficient for most use cases.

Published IP ranges, for automation. For large-scale checks Google says to use an automatic solution that matches the IP against its published lists. Googlebot is in common-crawlers.json, in CIDR format.

bash
python3 -c 'import ipaddress,json,sys,urllib.request as u; ip=ipaddress.ip_address(sys.argv[1]); d=json.load(u.urlopen("https://developers.google.com/static/crawling/ipranges/common-crawlers.json")); print(any(ip in ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in d["prefixes"] if ("ipv4Prefix" in p) == (ip.version == 4)))' 66.249.66.1

Google's page does not say how often the file changes or how often to fetch it, so fetch it on a schedule instead of pasting the ranges into a config once. A real Googlebot that gets a 403 shows up in your logs with an IP that passes these checks. A request that fails them is not Google's crawler by Google's own methods, so refusing it is fine.

How to allow Googlebot without opening the site to fakes

Allowing by user agent alone is the shortcut that invites fakes, because anyone can send that string. Allow on verified identity instead.

  • On Cloudflare, cf.client.bot is a boolean field that Cloudflare defines as true when the request came from a known good bot or crawler. It covers any known good bot, not only Googlebot, so add a user agent condition if you want a rule scoped to Googlebot. If a custom rule or rate limit is the cause, a Skip rule can bypass rate limiting rules, Managed Rules and Super Bot Fight Mode rules for those requests, per Cloudflare's Skip action documentation. Cloudflare says Bot Fight Mode on the Free plan cannot be skipped.
  • On other WAFs and CDNs, write the allow rule to match the Googlebot user agent and the ranges in common-crawlers.json, and exempt that combination from challenge, country and rate rules.
  • In a security plugin or web server, add the same exemption for verified Googlebot IPs, or remove the rule that matches Googlebot's user agent if it blocks more than it protects.
  • If a login or password is the cause, remove it from pages you want indexed. Google's guidance is to let non-signed-in users access the page, or allow Googlebot without authentication after verifying its identity.

Keep the rule narrow. Exempt verified Googlebot from the rule that blocked it, not from every protection you have.

After the change, rerun Test live URL on one affected page and check that Page fetch succeeds.

How to validate the fix and what to expect

When the live test passes on a sample of affected URLs, go to the page indexing report and open the issue. Google's guidance is to fix all instances first, then click Validate fix, and not click it again until validation succeeds or fails. Validation typically takes up to about two weeks, and Google says it can take much longer. If it fails, open the issue details to find the URL that caused it, fix that page, confirm the fix on the URLs still pending and restart validation.

Google does not publish how quickly an individual URL is recrawled after you remove a block. You can also use Request indexing from URL Inspection for a few important pages, though Google says it is limited per day and does not guarantee indexing. For many pages, an XML sitemap is the recommended route.

If a page was indexed before the block started, Google's status code documentation says a 403 removes a URL from the index. Whether and when it returns after the fix is not documented, so check again after validation rather than assuming.

Frequently asked questions

Is a 403 in Search Console the same as being blocked in robots.txt?

No. robots.txt tells Googlebot which URLs it may crawl, while a 403 is the server refusing a request Googlebot was allowed to make. Search Console reports them as different statuses. A 403 on /robots.txt itself is a separate case, covered in HTTP status codes and redirects.

Why does Search Console show 403 when my site loads fine?

Because you are a different visitor. Googlebot comes from Google's IP ranges with its own user agent and request pattern, and your firewall, plugin or server can treat that combination differently from your browser. Use the live test and your security logs, not your browser, as evidence.

Should I block Googlebot with a 403 to reduce server load?

No. Google says not to use 401 or 403 to limit crawl rate, and a 403 on a URL causes it to be treated as unavailable and removed if it was indexed. Google says 5xx and 429 responses make its crawlers temporarily slow down, so use those if you need to throttle, and keep 403 for traffic you really mean to refuse.

How long until the 403 status disappears?

Google says validation typically takes up to about two weeks, though it can take much longer. Recrawl timing for individual URLs is not published.

Can I just allow every request that claims to be Googlebot?

Not safely. Google says other crawlers often spoof the Googlebot user agent. Verify with reverse DNS or the published IP ranges before you allow or block.

Check what a crawler sees on your site

The free check reads one page the way AI crawlers do, with no account needed: robots.txt rules per crawler, the robots meta tag and headers, and what is in the raw HTML. It runs from Crawlito's servers, not Google's, so Search Console's live test and your CDN's security log remain the evidence for a Googlebot block. If you suspect rules are catching crawlers across a whole site, an audit covers up to 500 pages with evidence and prioritized fixes.

Filed under CDN and bot protection. Topics: bot-protection, google, http-status, waf, troubleshooting, server-logs