Skip to content

Blocked due to unauthorized request (401): causes and fixes

Search Console says 'Blocked due to unauthorized request (401)'? Find the layer asking for a login, then open the page or drop private URLs from sitemaps.

By Crawlito · 9 min read

In short

  • Search Console reports 'Blocked due to unauthorized request (401)' when a page asked Googlebot for authorization, and Google does not index URLs that return a 4xx status.
  • Under RFC 9110 a 401 must carry a WWW-Authenticate header with at least one challenge, and that header is the quickest clue to which layer asked for the login.
  • Googlebot never provides credentials, according to Google's page indexing help, and the crawler documentation from OpenAI, Anthropic and Perplexity describes no way to give their bots a login either.
  • Vercel's Standard Protection covers every deployment except production domains, so links, sitemaps or canonicals that point at a preview or generated deployment URL send crawlers to a login.
  • For private or staging pages a 401 is the right answer: leave the protection on and take those URLs out of sitemaps, canonicals and internal links instead.
On this page

"Blocked due to unauthorized request (401)" means Googlebot asked for a URL and your server answered that it needs a login first. Google's page indexing report help says the page was blocked to Googlebot by a request for authorization, and Googlebot never sends credentials, so it cannot get past it. The fix depends on the page. If it should be public, remove the login that sits in front of it. If it is meant to be private, the 401 is correct, and the real fix is to stop pointing crawlers at it.

This guide covers what a 401 is at the HTTP level, how to find the layer that sends it, and how to handle both cases. If Search Console shows 403 instead, the causes are usually firewalls and bot rules, covered in blocked due to access forbidden (403).

What a 401 response means

RFC 9110 defines 401 Unauthorized as a request that was not applied because it lacks valid authentication credentials for the target resource. It also sets one hard rule: a server that sends a 401 must include a WWW-Authenticate header with at least one challenge. A typical response from HTTP basic auth looks like this:

http
HTTP/2 401
www-authenticate: Basic realm="Staging"
content-type: text/html

The difference from 403 matters when you debug. A 401 says "tell me who you are". A 403 says "I understood the request and I refuse it", and the RFC notes that a 403 can be for reasons unrelated to credentials. In practice, a 401 usually comes from something configured to require a login: basic auth on a web server, a host's password or deployment protection, an authentication proxy, or the application itself.

Signal401 Unauthorized403 Forbidden
What the server is sayingCredentials are missing or invalidThe request is refused
WWW-Authenticate headerRequired by RFC 9110Not required
Usual sourceBasic auth, password or deployment protection, auth proxy, app loginWAF, bot rule, IP or country block, server deny rule
Search Console statusBlocked due to unauthorized request (401)Blocked due to access forbidden (403)

How Google and AI crawlers handle a 401

Google's HTTP status code documentation treats every 4xx except 429 the same way. Google does not index URLs that return a 4xx, URLs that were already indexed are removed, and crawling frequency for them gradually decreases. Google also says not to use 401 or 403 to limit crawl rate: 4xx codes other than 429 have no effect on it.

Googlebot will not try to log in. The same help page states that Googlebot never provides credentials, and its suggested fixes for a 401, if you want the page indexed, are to remove the authorization requirement for the page, or to allow Googlebot after verifying its identity.

AI crawlers are in the same position. At the time of writing, the crawler documentation from OpenAI, Anthropic and Perplexity does not describe sending credentials or any way to give their crawlers and fetchers an account. Expect OAI-SearchBot, Claude-SearchBot, PerplexityBot and user-triggered fetchers such as ChatGPT-User to get the same 401 your logged-out visitors get, and so never see the page's content. Browser agents that act for a signed-in person are a separate case and outside this guide. Login walls and AI crawlers covers what they see behind redirects and login forms.

One special case: a 401 on /robots.txt. Google's robots.txt specification says its crawlers treat all 4xx errors except 429 as if no valid robots.txt file existed, so Google assumes there are no crawl restrictions. Your Disallow rules stop applying to Google, while the pages themselves may still return 401.

How to confirm the 401 and find the layer

Start with the quick test Google itself suggests: open the URL in a private or incognito window. If you see a browser password prompt or a login page, any visitor without a session does too. Your normal browser often hides the problem because it holds a session cookie.

Then fetch the page without any cookies and look at the status and the challenge:

bash
curl -sI https://www.example.com/pricing/ | grep -i -E '^(HTTP|www-authenticate|server|location)'

The WWW-Authenticate header and the realm in it usually identify the layer:

  • Basic realm="..." with a realm you recognise, such as "Staging" or "Restricted", points to basic auth in nginx, Apache, an .htaccess file or a host panel.
  • A server header that names your CDN or host, together with a challenge you did not write, points to a protection setting at that provider.
  • A Bearer challenge, or a 401 with a JSON body, points to your application or an API gateway in front of it.
  • A redirect (location:) to a login page is not a 401 at all on that URL. Follow it with curl -sIL to see where the chain ends.

Repeat the check on the exact hostname in the report, with and without www, and on a deep URL. Protection is often set per host or per path, so one clean homepage proves little. To check whether a rule treats crawlers differently, repeat the request with -A and a crawler user agent string.

In Search Console, open the report's example URLs in URL Inspection and run Test live URL on one. If the live test still fails, the protection is on now, not only during the last crawl.

Fix 1: public pages caught by a login

These are the cases where the page should be public and the 401 is a leftover or a scope mistake.

Basic auth left on after launch

A common cause is a staging password that stayed on the production server, or one that was meant for one section and was set at too high a level. In nginx, auth_basic is inherited by inner levels, and the nginx documentation says the special value off cancels an inherited auth_basic. To keep a private area protected while the rest of the site is public, set it only where it belongs:

nginx
server {
    server_name www.example.com;

    # No auth_basic at server level: public pages stay public.

    location /internal/ {
        auth_basic           "Internal";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }
}

If the directive sits at server level for historical reasons, auth_basic off; in the public locations also works, but moving it to the private location is easier to reason about later. In Apache, look for AuthType Basic and Require valid-user in the virtual host or in any .htaccess file up the directory tree. After the change, reload the server, purge the CDN cache and rerun the curl check. The other leftovers that travel from staging to production, such as noindex headers and Disallow: /, are in staging settings that hide live sites.

Deployment protection on preview or generated URLs

Hosts that protect previews can produce a 401 report even when your production domain is open, because Googlebot found a protected URL somewhere. Vercel is a clear example. Its Deployment Protection documentation describes these scopes at the time of writing:

Vercel scopeWhat it protects
Standard ProtectionAll deployments except production domains
All DeploymentsEvery URL, including production domains
Only Production DeploymentsProduction only, through Trusted IPs, on Enterprise

The settings page also lists two legacy scopes, (Legacy) Standard Protection and (Legacy) Pre-Production Deployments, which some older projects still use. Vercel notes that when you enable Standard Protection the production generated deployment URL becomes restricted, and it tells you to update fetch requests that use VERCEL_URL or VERCEL_BRANCH_URL, because those addresses are no longer publicly accessible. The same reasoning applies to anything a crawler reads: if your canonical tags, sitemap, Open Graph URLs or absolute internal links are built from those variables, they point crawlers at a protected address. Build them from your production domain instead. If All Deployments is selected, the production domain is protected too, and the fix is to switch to Standard Protection for a public site.

Vercel's documentation describes Vercel Authentication as a redirect to a Vercel login and does not state the exact status code a crawler receives, so check your own URLs with curl rather than assuming. Do not hand crawlers the x-vercel-protection-bypass value: Vercel documents Protection Bypass for Automation for automated tests, CI/CD pipelines, monitoring and webhooks, and the bypass value skips Password Protection, Vercel Authentication and Trusted IPs checks, as well as bot protection challenges, for anyone who has it. Vercel's bot checks are a separate layer, covered in Vercel Security Checkpoint and crawlers.

A proxy or application rule

Authentication proxies, identity-aware access gateways and app middleware can all answer 401 on paths they were never meant to cover. Typical slips are a pattern that matches more than intended (protecting /app* also catches /apps-directory/), a rule written for one hostname that also applies to the main site, or middleware that runs on every route because its matcher was left at the default. Narrow the rule to the private paths, then confirm with curl on both a public and a private URL.

Fix 2: private pages where 401 is correct

If the URL is a staging site, an admin area, a customer portal or an internal tool, keep the protection. Google lists password protection as the way to keep confidential content out of Search in its guide to controlling what you share, and says content that already appears will eventually be removed. A login also stops user-triggered fetchers that may not follow robots.txt: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.

What you change is where crawlers learn about those URLs:

  1. Sitemaps. Google's sitemap guide says to include the URLs you want to see in Google's search results. Remove protected URLs, and check that no sitemap index lists a sitemap from a staging host.
  2. Internal links. Search your templates, navigation and content for links to the staging host or the private paths. A "Log in" link in the header is fine; a footer link to a protected staging page is not.
  3. Canonicals and hreflang. Make sure no public page names a protected URL as its canonical or alternate.
  4. Leave robots.txt out of it. A Disallow line advertises the private path and is not access control. The login already keeps crawlers out.

Once nothing public points at those URLs, the 401 entries in Search Console become a record of correctly protected pages rather than a problem. Google's help frames the 401 fix as something to do if you want the page indexed, and notes that non-indexed URLs can be fine. You do not need to click Validate fix for URLs you mean to keep private.

How to validate the fix

For pages you opened up, run Test live URL on a sample of affected URLs and check that the page fetch now succeeds. Then open the issue in the page indexing report and select Validate fix. Google says validation typically takes up to about two weeks and can take longer, and it fails if a certain threshold of pages still shows the issue, which Search Console validation failed walks through. If the report also lists server errors for the same section, treat them separately with server error (5xx) in Search Console, because 5xx responses are handled differently from a 401.

A successful fetch removes the technical barrier. Whether and when Google indexes the page again, or an AI assistant uses it, depends on things outside the status code.

Frequently asked questions

Is a 401 worse than a 403 for SEO?

Google treats both the same way: like other 4xx errors except 429, the page is not indexed and an indexed URL is removed. The difference is diagnostic, because a 401 points to a login and a 403 usually to a firewall or bot rule.

Can I let only Googlebot past my basic auth?

Google's help says you can allow Googlebot after verifying its identity, but anything you then show Googlebot is effectively public in Search. If the content should be public, remove the login for everyone; if it is private, keep it private from crawlers too.

Why does Search Console show a 401 on a staging URL I never submitted?

Googlebot found the URL somewhere: a link, a sitemap, a canonical tag or an absolute URL built from a deployment variable. Find that reference and remove it, and keep the staging protection on.

Does a 401 on robots.txt block crawling?

No. Google treats a 4xx on robots.txt, other than 429, as if the file did not exist and assumes no crawl restrictions, so your Disallow rules stop applying.

Do AI crawlers ever log in to read protected pages?

OpenAI, Anthropic and Perplexity do not document any way for their crawlers or fetchers to send credentials, so treat anything behind a 401 as unreadable to them.

Check what an anonymous crawler gets

Once the login is gone from your public pages, run the free check on one of them. It checks that single page and needs no account. If preview domains, proxies and several hosts are involved, an audit covers up to 500 pages and returns a PDF with evidence and prioritized fixes.

Filed under Access and robots.txt. Topics: http-status, search-console, google, troubleshooting