Claude can't access a URL: web fetch errors and their causes
Claude can't access a URL? Use Anthropic's web fetch error codes to tell site-side failures from request-side ones, with a check and a fix for each cause.
By Crawlito · 9 min read
In short
- Anthropic's web fetch tool returns failures as error codes inside a normal 200 API response, so the code, not the HTTP status of the API call, tells you what went wrong.
- url_not_allowed covers domain filters and Anthropic-side restrictions including robots.txt, while url_not_accessible means the fetch itself failed with an HTTP error.
- url_not_in_prior_context is never the site's fault: web fetch only opens URLs that already appeared in a user message, a client tool result or an earlier search or fetch result.
- Anthropic says web fetch does not support pages rendered with JavaScript and only handles text, HTML and PDF, so a fetch can succeed and still return little of your content.
- Fetch results are cached, so a page you just fixed can still come back in its old state until the cache refreshes or a developer sets use_cache to false.
On this page
When Claude says it can't access a URL, the failure is on one of two sides. Either your site refused or broke the request (an HTTP error, a robots.txt rule, a format Claude cannot read, content that only appears after JavaScript runs), or the request never reached your site because of how it was made (the URL was not in the conversation, a domain filter, a usage limit). Anthropic's web fetch tool documents an error code for each of these, and the code tells you which side to fix.
This post sorts those codes into site-side and request-side causes and gives a check for each site-side one. It does not repeat the robots.txt rules for Anthropic's three bots: those are in ClaudeBot, Claude-SearchBot and Claude-User explained. For the same kind of diagnosis on OpenAI's side, see why ChatGPT can't see your website.
How Claude's web fetch tool reports a failure
The web fetch tool is a server tool in the Claude API: Anthropic fetches the page during the request and puts the content into the conversation. When a fetch fails, the API still returns HTTP 200. The error sits in the response body, and Claude receives it and continues the turn, which is usually when you see it say that it could not open the page.
{
"type": "web_fetch_tool_result",
"tool_use_id": "srvtoolu_a93jad",
"content": {
"type": "web_fetch_tool_result_error",
"error_code": "url_not_accessible"
}
}If you are a developer, log error_code from every web_fetch_tool_result_error block. If you are a site owner who only has Claude's reply, ask the person who saw the error to try again with the exact URL, then work through the site-side checks below. Anthropic does not publish error codes for the Claude apps, so the API list is the closest public map of what can go wrong.
Which web fetch errors are on your site's side
At the time of writing, Anthropic documents ten error codes. Five of them can point at your site:
| Error code | Anthropic's description | Likely site-side cause |
|---|---|---|
url_not_accessible | Failed to fetch content (HTTP error) | An error status such as a 403 from a firewall, a 404 or a 5xx |
url_not_allowed | Blocked by domain filtering (including the organization's settings) or Anthropic-side restrictions, such as private addresses, robots.txt, and URLs that appear to contain a credential the caller did not provide | Your robots.txt disallows the fetch |
unsupported_content_type | Content type not supported (only text, HTML and PDF) | The URL serves an image, a video or another non-text file |
content_too_large | A fetched PDF would push the request over the PDF page or size limits | A very large PDF (PDFs and images already in the conversation count toward the same limits) |
too_many_requests | Rate limit exceeded | Usually Anthropic's limit, not yours, but check your own rate limiting too |
Two of these need care. url_not_allowed mixes request-side causes (a developer's domain filter, a private IP address) with one site-side cause (robots.txt), so check robots.txt before you assume it is the caller's setting. And too_many_requests is documented as a rate limit without saying whose, and Anthropic does not say which code a 429 from your own server produces. If you rate limit bots, check your logs for 429s anyway.
What about a fetch that works but returns nothing useful
The worst case is not an error at all. Anthropic says the tool "currently does not support websites dynamically rendered with JavaScript". If your main text is injected by client-side scripts, the fetch succeeds and Claude gets the empty shell: a title, a navigation bar, maybe a loading message. Claude may then say it cannot find the information on the page, which sounds like an access problem but is a rendering one. Do AI crawlers run JavaScript covers this in depth.
Which errors are on the request side
The other codes describe how the fetch was asked for. Nothing you change on your server fixes them.
| Error code | What it means | Who fixes it |
|---|---|---|
url_not_in_prior_context | The URL had not appeared earlier in the conversation, or only in a source excluded by url_sources | The user pastes the full URL |
invalid_tool_input | Malformed URL or a non-HTTP(S) scheme | The user or developer |
url_too_long | URL longer than 250 characters | Shorten the URL, for example by dropping tracking parameters |
max_uses_exceeded | The max_uses limit for the turn was reached; failed fetches count too | The developer |
unavailable | An internal error at Anthropic | Retry later |
url_not_in_prior_context is the one that confuses people most. To limit data exfiltration, Anthropic only lets web fetch open URLs that already appeared in the conversation: in a user message, in a client-side tool result, or in an earlier web search or web fetch result. A URL that Claude writes itself, or that sits only in the system prompt, cannot be fetched. So if a user says "read the pricing page on example.com" and Claude guesses https://example.com/pricing, the fetch fails even when the page is perfectly reachable. When web search is also enabled, Claude can search first and then fetch a URL from the results.
The tool also refuses a URL that appears to contain a credential, such as an API key or a password, unless that credential appears in the system prompt or in the text of a user message. A credential that appears only in a tool result does not count. So a share link with a token in the query string works when a user pastes it, but the same link found in a search result or another tool's output can fail with url_not_allowed.
How to check each site-side cause
Work through these in order. Each takes a minute with curl.
1. Check the status code the page returns
curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" https://www.example.com/page
curl -s -o /dev/null -w "%{http_code}\n" -A "Claude-User" https://www.example.com/pageA 200 for both is what you want. A 403 or 503 only on the second request means a rule is keying on the user agent. Anthropic does not document how web fetch handles redirects, so if the page redirects, make sure the final URL returns a 200 with your content, not a 404 or a login page. Anthropic's web fetch page does not name the user agent the tool sends, so treat the Claude-User test as a way to find user-agent rules, not as an exact copy of the real request.
2. Check for a challenge page
A firewall that serves a JavaScript challenge or CAPTCHA often returns 403 or 503 with a challenge body. Anthropic does not document how web fetch treats challenge pages, but a tool that does not run JavaScript cannot pass a JavaScript challenge, so expect an error or the challenge text instead of your page. Look for challenge markers in the body:
curl -s -A "Claude-User" https://www.example.com/page | grep -iE "captcha|challenge|cf-chl|just a moment" | headWAF challenge pages and AI crawlers explains how to allow verified bots past them.
3. Check robots.txt
Anthropic lists robots.txt among the restrictions behind url_not_allowed. Its crawler support article says that disabling Claude-User "prevents our system from retrieving your content in response to a user query". Fetch your file and look for a group that names Claude-User, or a User-agent: * group that disallows the path with no Claude-User group to override it:
curl -s https://www.example.com/robots.txt | grep -iA5 -E "claude-user|user-agent: \*"Also check the status code of robots.txt itself. Anthropic does not document how web fetch treats a robots.txt that errors, but RFC 9309 tells crawlers to assume complete disallow when the file returns a 5xx, while a 4xx lets them access anything.
4. Check the content type
curl -sI https://www.example.com/page | grep -i "^content-type"Anthropic lists only text, HTML and PDF as supported, without naming every media type it accepts. text/html and application/pdf are the safe answers. A page served as application/octet-stream or an image type does not match any of the three. Content-Type and encoding errors covers headers that are subtly wrong.
5. Check that the main text is in the raw HTML
curl -s https://www.example.com/page | grep -c "a sentence from your main content"A count of 0 means the text only appears after JavaScript runs, and web fetch will not see it. The fix is server-side rendering, static generation or prerendering for that route.
Why Claude sees an old version of your page
Anthropic says the web fetch tool caches results, and that "the content returned might not always reflect the latest version available at the URL". The cache is managed automatically and Anthropic does not publish how long entries live. So if you fixed a 403 or published new text ten minutes ago, Claude can still see the old response.
Developers can bypass the cache. With tool version web_fetch_20260309 or later, setting "use_cache": false fetches fresh content:
{
"type": "web_fetch_20260309",
"name": "web_fetch",
"use_cache": false
}Anthropic advises turning the cache off only when the user explicitly asks for fresh content or the source changes rapidly, because bypassing it increases latency. Anthropic documents no way for a site owner to clear the cache. Each fetch result carries a retrieved_at timestamp, which tells a developer whether they are looking at a stale copy.
Settings developers often miss
If you build on the API and fetches fail for sites that load fine in a browser, check these before blaming the site:
- Domain filters match on the domain only for web fetch. An
allowed_domainsentry with a path, such asexample.com/blog, never matches a web fetch URL. Subdomains are included automatically, soexample.comcoversdocs.example.com. - Organization settings apply too. An admin can turn web fetch off or set allowed and blocked domains in the Claude Console. With web fetch off, the request fails with a 400
invalid_request_error, not an error code inside the result. max_content_tokenstruncates. A long page can be cut off, which looks like missing content rather than a failure. It applies to text, not PDFs.- Some platforms support only the basic version. Microsoft Foundry deployments hosted on Azure support only
web_fetch_20250910, without dynamic filtering, response inclusion or cache bypass.
For pages that need a real browser to render, click or fill forms, Anthropic points developers to its browser use tool instead. That is a client tool: the developer's application drives the browser, so the site sees the developer's network, not an Anthropic fetcher. Anthropic's own hosted browser, which sends a Claude-Agent token, is a different product, covered in Claude-Agent and Anthropic's hosted browser.
Frequently asked questions
Does Claude's web fetch tool follow robots.txt?
Anthropic lists robots.txt among the Anthropic-side restrictions that produce a url_not_allowed error. Its crawler article says disabling Claude-User prevents retrieval of your content in response to a user query.
What user agent does Claude's web fetch send?
The web fetch documentation does not name one. Anthropic's crawler article ties user-directed retrieval to Claude-User, so match on that token in logs, but do not assume a full user-agent string that Anthropic has not published.
Why does Claude say it can't access a URL that works in my browser?
Your browser runs JavaScript and can pass bot challenges, while Anthropic says web fetch does not support JavaScript-rendered sites. Or the URL was never in the conversation, which gives url_not_in_prior_context however reachable the page is.
Can Claude read PDFs from my site?
Yes. Anthropic says web fetch supports text, HTML and PDF, and returns a PDF as a document. A PDF that would exceed the PDF page or size limits fails with content_too_large.
How do I make Claude see my updated page?
Fix the cause first, then wait for the cache to refresh, since Anthropic does not publish its duration. A developer can force a fresh fetch with use_cache set to false on web_fetch_20260309 or later.
Check what Claude's fetchers receive
Most site-side failures come down to a status code, a robots.txt group, a challenge page or HTML without the content. The free check runs one page through Crawlito's published checks, with no account needed. If many pages fail, or you need evidence to hand a developer, an audit covers up to 500 pages and comes as a PDF with the evidence and prioritized fixes.
Filed under AI crawlers. Topics: anthropic, claudebot, troubleshooting, ai-crawlers, robots-txt