Skip to content

Gemini can't access your URL: URL context limits and fetch rules

Gemini can't access a URL? Read the retrieval status, then check Google's documented URL context limits: public URLs only, no paywalls, 20 URLs, 34MB per URL.

By Crawlito · 7 min read

In short

  • Google's Gemini API URL context tool first tries an internal index cache and only falls back to a live fetch when the URL is not in that index, for example a very new page.
  • Each URL gets a retrieval status in the response metadata: URL_RETRIEVAL_STATUS_SUCCESS, _ERROR, _PAYWALL or _UNSAFE, so a failed fetch is visible even when the request itself succeeds.
  • Google documents a limit of 20 URLs per request and 34MB of content per URL, and the URLs must be publicly reachable: localhost, private networks and tunnels such as ngrok are not supported.
  • Paywalled pages, pages that need a login, YouTube videos, Google Workspace files, and video or audio files are unsupported; text formats, PNG, JPEG, BMP, WebP and PDF are supported.
  • At the time of writing, Google does not document the user agent, IP ranges or robots.txt handling for URL context live fetches.
On this page

When Gemini can't access a URL you gave it through the Gemini API's URL context tool, the response usually tells you why: every URL gets a retrieval status in the response metadata, such as URL_RETRIEVAL_STATUS_ERROR or URL_RETRIEVAL_STATUS_PAYWALL. Read that first. Then compare the page with the limits Google documents: the URL must be public, not behind a login or paywall, one of the supported content types, no larger than 34MB, and one of at most 20 URLs in the request. What Google does not document is how the live fetch identifies itself, so firewall and robots.txt questions need testing rather than guessing.

This post works from Google's URL context documentation and its API references. It covers the Gemini API tool that developers call. Google's page does not describe how the consumer Gemini app fetches links, so do not assume the same rules apply there.

How Gemini URL context retrieves a page

Google describes a two-step retrieval process:

  1. Index cache first. The tool tries to get the content from an internal index, which Google describes as a highly optimized cache.
  2. Live fetch as fallback. If the URL is not in that index, for example because the page is very new, the tool fetches the URL directly in real time.

Three practical consequences follow:

  • A success does not prove your server was contacted. If the URL was served from the cache, no request reached your site, so an empty access log is not evidence of a failure.
  • A cached copy may not be the version you just published. Google does not say how fresh the cache is or how it is refreshed. If Gemini describes an older version of the page, the cache is a likely explanation, but this is not documented.
  • Only the URLs you pass are read. Google says the model retrieves content only from the URLs you provide, not from links nested inside them. Pass the exact page, not a hub page that links to it.

Google also runs a content moderation check on each URL. A URL that fails it is reported as unsafe and is not used.

How to read the URL retrieval status

The status is the fastest way to tell a site problem from a request problem. In the generateContent API, each candidate carries urlContextMetadata with one urlMetadata entry per URL. A request looks like this (model names change often; gemini-3.6-flash is one of the models Google lists as supporting URL context at the time of writing):

bash
curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.6-flash:generateContent" \
  -H "x-goog-api-key: $GEMINI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "contents": [{"parts": [{"text": "Summarize https://example.com/pricing"}]}],
    "tools": [{"urlContext": {}}]
  }'

And the part of the response to inspect has this shape (the values here are an example):

json
{
  "urlContextMetadata": {
    "urlMetadata": [
      {
        "retrievedUrl": "https://example.com/pricing",
        "urlRetrievalStatus": "URL_RETRIEVAL_STATUS_ERROR"
      }
    ]
  }
}

The Python SDK uses snake case for the same fields: url_context_metadata, url_metadata, retrieved_url and url_retrieval_status. Google's generateContent API reference defines the status values:

StatusGoogle's descriptionWhat to check on your site
URL_RETRIEVAL_STATUS_SUCCESSRetrieval succeededNothing; if the answer looks wrong, the content may have come from the cache
URL_RETRIEVAL_STATUS_ERRORRetrieval failed due to an errorStatus code, firewall, content type, size, reachability
URL_RETRIEVAL_STATUS_PAYWALLThe content is behind a paywallWhether an anonymous visitor gets the full text
URL_RETRIEVAL_STATUS_UNSAFEThe content is unsafeFailed Google's moderation check; nothing in your server config changes this
URL_RETRIEVAL_STATUS_UNSPECIFIEDDefault value, unusedNot expected in practice

At the time of writing, the URL context page shows its examples with the newer Interactions API (/v1beta/interactions) rather than generateContent. There, each retrieval appears in a url_context_result step whose results carry a url and a status, and the Interactions API reference lists the same four outcomes in lower case: success, error, paywall and unsafe. Answer text carries url_citation annotations with the source URL and title. Whichever API you use, log the per-URL status: the request as a whole can complete normally while one URL failed.

Google does not say how it decides that content is behind a paywall, and the ERROR status does not carry a reason code. So the checks below are about removing the documented causes one by one.

The documented limits, and the site-side check for each

Documented limitHow to check itFix
Up to 20 URLs per requestCount the URLs in the promptSplit the work across requests
34MB maximum per URLcurl -sI and read content-length, or download and measureLink a smaller page or a summary instead of a large file
Must be publicly accessibleFetch from a network outside yoursPublish it on a public host
No localhost, private networks or tunnels such as ngrok or pinggyLook at the host nameDeploy to a public URL; Google lists tunnels as unsupported
No pages that need a login, no paywalled contentOpen in a private window with no cookiesMake a public version or summary
Supported content types onlyRead the content-type headerServe a supported type, or extract the text
Full URL including the protocolLook for a missing https://Pass the complete URL

Start with what an anonymous request from outside your network gets:

bash
curl -s -o /dev/null -D - -L https://example.com/pricing | grep -iE "^HTTP|content-type|content-length|location"

That shows the final status, the content type and the size, following redirects. A 401, 403 or a redirect to /login means the page is not public in the sense Google means. A 200 whose body is a sign-in form or a teaser paragraph is the same problem in disguise; login walls and paywalls covers how to spot both.

The development case is common enough to state plainly: a URL on localhost, 127.0.0.1, an office network or a tunnel will not work, whatever you allow on your side. Google lists those as unsupported.

Which content types URL context supports

Google lists the supported and unsupported types explicitly:

SupportedUnsupported
Text: text/html, application/json, text/plain, text/xml, text/css, text/javascript, text/csv, text/rtfPaywalled content
Images: image/png, image/jpeg, image/bmp, image/webpYouTube videos (Google points to its video understanding feature instead)
PDF: application/pdfGoogle Workspace files such as Docs and Sheets
Video and audio files

Two site-side mistakes trip this up. A server can send the wrong header, for example a PDF served as application/octet-stream or an HTML page labelled as something else. And a "document" link may point to a Google Doc or Sheet, which URL context does not read at all. Check the header your server actually sends; content-type and encoding errors shows how to find and fix a mislabelled response.

Note what the list does not say. Google does not document whether the live fetch runs JavaScript. If the text you care about only appears after scripts run, make sure it is in the HTML the server sends, so the answer does not depend on undocumented behaviour.

What Google does not document: user agent, IPs and robots.txt

At the time of writing, the URL context page says nothing about:

  • The user agent of the live fetch.
  • The IP ranges it comes from.
  • robots.txt: whether the live fetch reads or obeys it.

It is also not listed on Google's user-triggered fetchers page, which covers fetchers such as Google-Agent and Google-GeminiNotebook and says that, because a user requested the fetch, those fetchers generally ignore robots.txt. Google's crawler overview does not mention URL context either. So you cannot write a robots.txt group or a firewall rule that names the URL context fetch with any confidence.

Google-Extended does not settle it. Google says that token controls whether content Google crawls may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It does not mention the Gemini API URL context tool, so do not expect a Google-Extended rule to block or allow it.

What you can do is test:

  1. Make a fresh URL that cannot be in any cache, for example a new page with a unique path.
  2. Pass it to URL context and note the time and the retrieval status.
  3. Search your edge and origin logs for requests to that path in that minute. Record the user agent, IP and the status you returned.
  4. If your CDN or WAF blocked it, the request appears in the security events log, not your origin log. A challenge page also counts as a failure; see WAF challenge pages and AI crawlers.

Treat what you record as an observation from your own site on that day, not a documented identity. If you allow it in a firewall rule, revisit the rule when Google publishes documentation.

Frequently asked questions

Why does Gemini say it can't access a URL that opens fine in my browser?

Your browser may have cookies, a VPN or office network access that an outside fetch does not. Check the retrieval status, then fetch the page from outside your network without cookies and confirm the status code, content type and size.

Does robots.txt control Gemini URL context?

Google's URL context documentation does not mention robots.txt, and the tool is not listed among Google's documented crawlers or fetchers. There is no documented robots.txt token for it at the time of writing.

What does URL_RETRIEVAL_STATUS_PAYWALL mean?

Google's API reference says retrieval failed because the content is behind a paywall. Google does not document how it detects a paywall, so check what an anonymous visitor receives and offer a public summary if you want the page to be readable.

Why did Gemini describe an old version of my page?

URL context tries an internal index cache before fetching live, and Google does not say how fresh that cache is. A brand-new URL is the case Google gives for the live fetch fallback.

How many URLs can I pass to URL context?

Google documents up to 20 URLs per request, with a maximum of 34MB of content from each one.

Check what an outside fetch gets from your page

Most URL context failures that are fixable on your side come down to what an anonymous request receives: the status code, the content type, a login or paywall, or a firewall rule. The free check tests a single page from outside your network, with no account. It is not Google's fetcher, so your edge logs remain the evidence for how Google's own fetch is treated.

Filed under AI crawlers. Topics: google, ai-crawlers, troubleshooting, paywalls