Skip to content

Claude can't access a URL: web fetch errors and their causes

Claude can't access a URL? Use Anthropic's web fetch error codes to tell site-side failures from request-side ones, with a check and a fix for each cause.

By Crawlito · 9 min read

In short

  • Anthropic's web fetch tool returns failures as error codes inside a normal 200 API response, so the code, not the HTTP status of the API call, tells you what went wrong.
  • url_not_allowed covers domain filters and Anthropic-side restrictions including robots.txt, while url_not_accessible means the fetch itself failed with an HTTP error.
  • url_not_in_prior_context is never the site's fault: web fetch only opens URLs that already appeared in a user message, a client tool result or an earlier search or fetch result.
  • Anthropic says web fetch does not support pages rendered with JavaScript and only handles text, HTML and PDF, so a fetch can succeed and still return little of your content.
  • Fetch results are cached, so a page you just fixed can still come back in its old state until the cache refreshes or a developer sets use_cache to false.
On this page

When Claude says it can't access a URL, the failure is on one of two sides. Either your site refused or broke the request (an HTTP error, a robots.txt rule, a format Claude cannot read, content that only appears after JavaScript runs), or the request never reached your site because of how it was made (the URL was not in the conversation, a domain filter, a usage limit). Anthropic's web fetch tool documents an error code for each of these, and the code tells you which side to fix.

This post sorts those codes into site-side and request-side causes and gives a check for each site-side one. It does not repeat the robots.txt rules for Anthropic's three bots: those are in ClaudeBot, Claude-SearchBot and Claude-User explained. For the same kind of diagnosis on OpenAI's side, see why ChatGPT can't see your website.

How Claude's web fetch tool reports a failure

The web fetch tool is a server tool in the Claude API: Anthropic fetches the page during the request and puts the content into the conversation. When a fetch fails, the API still returns HTTP 200. The error sits in the response body, and Claude receives it and continues the turn, which is usually when you see it say that it could not open the page.

json
{
  "type": "web_fetch_tool_result",
  "tool_use_id": "srvtoolu_a93jad",
  "content": {
    "type": "web_fetch_tool_result_error",
    "error_code": "url_not_accessible"
  }
}

If you are a developer, log error_code from every web_fetch_tool_result_error block. If you are a site owner who only has Claude's reply, ask the person who saw the error to try again with the exact URL, then work through the site-side checks below. Anthropic does not publish error codes for the Claude apps, so the API list is the closest public map of what can go wrong.

Which web fetch errors are on your site's side

At the time of writing, Anthropic documents ten error codes. Five of them can point at your site:

Error codeAnthropic's descriptionLikely site-side cause
url_not_accessibleFailed to fetch content (HTTP error)An error status such as a 403 from a firewall, a 404 or a 5xx
url_not_allowedBlocked by domain filtering (including the organization's settings) or Anthropic-side restrictions, such as private addresses, robots.txt, and URLs that appear to contain a credential the caller did not provideYour robots.txt disallows the fetch
unsupported_content_typeContent type not supported (only text, HTML and PDF)The URL serves an image, a video or another non-text file
content_too_largeA fetched PDF would push the request over the PDF page or size limitsA very large PDF (PDFs and images already in the conversation count toward the same limits)
too_many_requestsRate limit exceededUsually Anthropic's limit, not yours, but check your own rate limiting too

Two of these need care. url_not_allowed mixes request-side causes (a developer's domain filter, a private IP address) with one site-side cause (robots.txt), so check robots.txt before you assume it is the caller's setting. And too_many_requests is documented as a rate limit without saying whose, and Anthropic does not say which code a 429 from your own server produces. If you rate limit bots, check your logs for 429s anyway.

What about a fetch that works but returns nothing useful

The worst case is not an error at all. Anthropic says the tool "currently does not support websites dynamically rendered with JavaScript". If your main text is injected by client-side scripts, the fetch succeeds and Claude gets the empty shell: a title, a navigation bar, maybe a loading message. Claude may then say it cannot find the information on the page, which sounds like an access problem but is a rendering one. Do AI crawlers run JavaScript covers this in depth.

Which errors are on the request side

The other codes describe how the fetch was asked for. Nothing you change on your server fixes them.

Error codeWhat it meansWho fixes it
url_not_in_prior_contextThe URL had not appeared earlier in the conversation, or only in a source excluded by url_sourcesThe user pastes the full URL
invalid_tool_inputMalformed URL or a non-HTTP(S) schemeThe user or developer
url_too_longURL longer than 250 charactersShorten the URL, for example by dropping tracking parameters
max_uses_exceededThe max_uses limit for the turn was reached; failed fetches count tooThe developer
unavailableAn internal error at AnthropicRetry later

url_not_in_prior_context is the one that confuses people most. To limit data exfiltration, Anthropic only lets web fetch open URLs that already appeared in the conversation: in a user message, in a client-side tool result, or in an earlier web search or web fetch result. A URL that Claude writes itself, or that sits only in the system prompt, cannot be fetched. So if a user says "read the pricing page on example.com" and Claude guesses https://example.com/pricing, the fetch fails even when the page is perfectly reachable. When web search is also enabled, Claude can search first and then fetch a URL from the results.

The tool also refuses a URL that appears to contain a credential, such as an API key or a password, unless that credential appears in the system prompt or in the text of a user message. A credential that appears only in a tool result does not count. So a share link with a token in the query string works when a user pastes it, but the same link found in a search result or another tool's output can fail with url_not_allowed.

How to check each site-side cause

Work through these in order. Each takes a minute with curl.

1. Check the status code the page returns

bash
curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" https://www.example.com/page
curl -s -o /dev/null -w "%{http_code}\n" -A "Claude-User" https://www.example.com/page

A 200 for both is what you want. A 403 or 503 only on the second request means a rule is keying on the user agent. Anthropic does not document how web fetch handles redirects, so if the page redirects, make sure the final URL returns a 200 with your content, not a 404 or a login page. Anthropic's web fetch page does not name the user agent the tool sends, so treat the Claude-User test as a way to find user-agent rules, not as an exact copy of the real request.

2. Check for a challenge page

A firewall that serves a JavaScript challenge or CAPTCHA often returns 403 or 503 with a challenge body. Anthropic does not document how web fetch treats challenge pages, but a tool that does not run JavaScript cannot pass a JavaScript challenge, so expect an error or the challenge text instead of your page. Look for challenge markers in the body:

bash
curl -s -A "Claude-User" https://www.example.com/page | grep -iE "captcha|challenge|cf-chl|just a moment" | head

WAF challenge pages and AI crawlers explains how to allow verified bots past them.

3. Check robots.txt

Anthropic lists robots.txt among the restrictions behind url_not_allowed. Its crawler support article says that disabling Claude-User "prevents our system from retrieving your content in response to a user query". Fetch your file and look for a group that names Claude-User, or a User-agent: * group that disallows the path with no Claude-User group to override it:

bash
curl -s https://www.example.com/robots.txt | grep -iA5 -E "claude-user|user-agent: \*"

Also check the status code of robots.txt itself. Anthropic does not document how web fetch treats a robots.txt that errors, but RFC 9309 tells crawlers to assume complete disallow when the file returns a 5xx, while a 4xx lets them access anything.

4. Check the content type

bash
curl -sI https://www.example.com/page | grep -i "^content-type"

Anthropic lists only text, HTML and PDF as supported, without naming every media type it accepts. text/html and application/pdf are the safe answers. A page served as application/octet-stream or an image type does not match any of the three. Content-Type and encoding errors covers headers that are subtly wrong.

5. Check that the main text is in the raw HTML

bash
curl -s https://www.example.com/page | grep -c "a sentence from your main content"

A count of 0 means the text only appears after JavaScript runs, and web fetch will not see it. The fix is server-side rendering, static generation or prerendering for that route.

Why Claude sees an old version of your page

Anthropic says the web fetch tool caches results, and that "the content returned might not always reflect the latest version available at the URL". The cache is managed automatically and Anthropic does not publish how long entries live. So if you fixed a 403 or published new text ten minutes ago, Claude can still see the old response.

Developers can bypass the cache. With tool version web_fetch_20260309 or later, setting "use_cache": false fetches fresh content:

json
{
  "type": "web_fetch_20260309",
  "name": "web_fetch",
  "use_cache": false
}

Anthropic advises turning the cache off only when the user explicitly asks for fresh content or the source changes rapidly, because bypassing it increases latency. Anthropic documents no way for a site owner to clear the cache. Each fetch result carries a retrieved_at timestamp, which tells a developer whether they are looking at a stale copy.

Settings developers often miss

If you build on the API and fetches fail for sites that load fine in a browser, check these before blaming the site:

  • Domain filters match on the domain only for web fetch. An allowed_domains entry with a path, such as example.com/blog, never matches a web fetch URL. Subdomains are included automatically, so example.com covers docs.example.com.
  • Organization settings apply too. An admin can turn web fetch off or set allowed and blocked domains in the Claude Console. With web fetch off, the request fails with a 400 invalid_request_error, not an error code inside the result.
  • max_content_tokens truncates. A long page can be cut off, which looks like missing content rather than a failure. It applies to text, not PDFs.
  • Some platforms support only the basic version. Microsoft Foundry deployments hosted on Azure support only web_fetch_20250910, without dynamic filtering, response inclusion or cache bypass.

For pages that need a real browser to render, click or fill forms, Anthropic points developers to its browser use tool instead. That is a client tool: the developer's application drives the browser, so the site sees the developer's network, not an Anthropic fetcher. Anthropic's own hosted browser, which sends a Claude-Agent token, is a different product, covered in Claude-Agent and Anthropic's hosted browser.

Frequently asked questions

Does Claude's web fetch tool follow robots.txt?

Anthropic lists robots.txt among the Anthropic-side restrictions that produce a url_not_allowed error. Its crawler article says disabling Claude-User prevents retrieval of your content in response to a user query.

What user agent does Claude's web fetch send?

The web fetch documentation does not name one. Anthropic's crawler article ties user-directed retrieval to Claude-User, so match on that token in logs, but do not assume a full user-agent string that Anthropic has not published.

Why does Claude say it can't access a URL that works in my browser?

Your browser runs JavaScript and can pass bot challenges, while Anthropic says web fetch does not support JavaScript-rendered sites. Or the URL was never in the conversation, which gives url_not_in_prior_context however reachable the page is.

Can Claude read PDFs from my site?

Yes. Anthropic says web fetch supports text, HTML and PDF, and returns a PDF as a document. A PDF that would exceed the PDF page or size limits fails with content_too_large.

How do I make Claude see my updated page?

Fix the cause first, then wait for the cache to refresh, since Anthropic does not publish its duration. A developer can force a fresh fetch with use_cache set to false on web_fetch_20260309 or later.

Check what Claude's fetchers receive

Most site-side failures come down to a status code, a robots.txt group, a challenge page or HTML without the content. The free check runs one page through Crawlito's published checks, with no account needed. If many pages fail, or you need evidence to hand a developer, an audit covers up to 500 pages and comes as a PDF with the evidence and prioritized fixes.

Filed under AI crawlers. Topics: anthropic, claudebot, troubleshooting, ai-crawlers, robots-txt