Skip to content

Google-Agent: Google's AI browsing agent and robots.txt

Google-Agent is a user-triggered fetcher, not Googlebot. See what Google documents about robots.txt, its user agent strings and how to verify it.

By Crawlito · Updated 8 min read

In short

  • Google's documentation says Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions at a user's request; it is listed with Google's user-triggered fetchers, not its search crawlers.
  • Google says user-triggered fetchers generally ignore robots.txt rules because a user initiated the fetch, and its Google-Agent entry gives no robots.txt or blocking instructions of its own.
  • Google-Agent's IP ranges are published in user-triggered-agents.json, separate from the files for Googlebot and for other user-triggered fetchers.
  • Google says user agent strings can be spoofed, so verify by matching the source IP against the published ranges or by reverse DNS.
  • Google is experimenting with Web Bot Auth for some Google-Agent requests under the identity https://agent.bot.goog, and says not every request is signed.
On this page

Google-Agent is a user agent that Google's own agents use when they browse the web and take actions because a user asked them to. Google documents it as a user-triggered fetcher, in the same family as Google Site Verifier and Google Read Aloud, and not as a search crawler like Googlebot. Google says user-triggered fetchers generally ignore robots.txt rules, so a Disallow line is not a dependable way to stop it. Because the user agent string can be spoofed, check the source IP against Google's published list before you trust a request.

Google's crawling changelog records the addition as 20 March 2026, with a gradual rollout. This post covers what Google documents, what it leaves unsaid, and how to check what you see in your logs. The details may change, so confirm against Google's Google-Agent entry before you act on them.

What is Google-Agent?

Google's page says Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request. The page does not name the products behind it. In practice, that means a person asks a Google agent to do something, and the agent visits pages on their behalf, possibly to read them and possibly to act on them.

Two things follow from that wording:

  • A person is behind the visit. Like ChatGPT-User or Claude-User, this is not a bot wandering the web to build an index. Google's page does not describe Google-Agent as feeding search results or model training, so do not treat it as either.
  • It may do more than read. The description says "perform actions". The page does not say which actions, so you cannot assume it only sends GET requests. Look at the methods and paths in your own logs.

Google-Agent vs Googlebot: how they differ

Google groups its automated clients into three types in its crawler overview. Google-Agent sits in the third, and the robots.txt statement for that row comes from Google's user-triggered fetchers page.

TypeExampleWhat Google says about robots.txt
Common crawlersGooglebotThey always respect robots.txt rules for automatic crawls
Special-case crawlersAdsBotUsed for products with an agreement with the crawled site; AdsBot, for example, ignores the global * group with the publisher's permission
User-triggered fetchersGoogle Site Verifier, Google-AgentGenerally ignore robots.txt, because a user initiated the fetch

The practical differences:

  • Different token. Googlebot is Google's main crawler for Search. Google-Agent has its own user agent and its own entry in Google's list of user-triggered fetchers.
  • Different IP file. Googlebot's ranges are in common-crawlers.json. Google-Agent's are in user-triggered-agents.json.
  • Different trigger. Googlebot crawls on its own schedule. Google-Agent runs when someone asks an agent to go somewhere.

Google-Extended is a third thing again: a robots.txt-only token that never appears in logs. Do not look for it there, and do not expect a Google-Extended rule to affect Google-Agent. Google does not document any link between them.

Does robots.txt apply to Google-Agent and other user-triggered fetchers?

Google's user-triggered fetchers page states that, because the fetch was requested by a user, these fetchers generally ignore robots.txt rules. Google-Agent is on that page's list, so that statement covers it. Google's Google-Agent entry does not give a separate robots.txt rule for it, and it does not describe a way to block or control it.

So, at the time of writing:

  • Do not rely on a robots.txt group named Google-Agent to keep it out. Google does not document that it reads one.
  • A Disallow: / under User-agent: * does not stop Google's user-triggered fetchers either, since Google says they generally ignore robots.txt.
  • Do not edit your Googlebot rules for this. Googlebot's rules govern Googlebot's automatic crawls and nothing in Google's Google-Agent entry links them to this agent.

This is not unusual. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Anthropic's documentation says its bots honor robots.txt directives, and it does not carve out an exemption for Claude-User. If you need to restrict a user-triggered fetcher, the control is on your server or edge network (a firewall rule, authentication), not in robots.txt. RFC 9309 itself says robots.txt rules are not a form of access authorization.

Google-Agent user agent strings (desktop and mobile)

Google publishes two strings. The Chrome version appears as the placeholder W.X.Y.Z, which Google replaces with the actual version number in real requests.

Desktop:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36

Mobile:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent)

Both look like a normal Chrome browser, so match on the token, not the whole string. To find it in an nginx or Apache access log:

bash
grep -i "Google-Agent" /var/log/nginx/access.log | awk '{print $1, $6, $7, $9}' | tail -20

That prints the IP, method, path and status of the last 20 matches. The IP column is what you need for the next step. For where logs live on other stacks, see finding AI crawler visits in your server logs.

How to verify Google-Agent with user-triggered-agents.json and reverse DNS

Google warns that the user agent string can be spoofed. Anyone can send Google-Agent from a script, so treat a log line as a claim until the IP checks out. Google documents two checks.

Check the IP against user-triggered-agents.json

Google-Agent uses the ranges in user-triggered-agents.json. The file has a creationTime and a prefixes list, where each entry holds one ipv4Prefix or ipv6Prefix in CIDR notation. This script tests one IP:

bash
python3 -c 'import ipaddress,json,sys,urllib.request; ip=ipaddress.ip_address(sys.argv[1]); d=json.load(urllib.request.urlopen("https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json")); print(any(ip in ipaddress.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"]) for p in d["prefixes"]))' 203.0.113.7

Replace the sample address with one from your log. True means the IP is inside a published range; False means it is not, and the request is not Google-Agent as Google documents it. Fetch the file fresh rather than copying ranges into a config, since Google can change it. Google also publishes user-triggered-fetchers.json and user-triggered-fetchers-google.json for other user-triggered fetchers, so a Google-Agent request should not be judged against those.

Check reverse DNS

Google's verification guidance is a reverse lookup, a check that the hostname ends in googlebot.com, google.com or googleusercontent.com, then a forward lookup that must return the original IP:

bash
host 203.0.113.7
host <hostname-from-the-first-result>

For user-triggered fetchers Google lists two hostname patterns: ***-***-***-***.gae.googleusercontent.com and google-proxy-***-***-***-***.google.com. At the time of writing, Google's pages disagree on which is which. The user-triggered fetchers page assigns the first to Google-owned fetchers and the second to user-owned ones. The verification guide says IPs in user-triggered-fetchers.json resolve to gae.googleusercontent.com, and fetchers controlled by Google use user-triggered-fetchers-google.json and resolve to google.com. Neither page says which pattern Google-Agent resolves to, so accept either, use the IP file as your main check and DNS as a second opinion. For the related fetchers, see Google's Gemini, NotebookLM and Read Aloud fetchers. More on this method in how to verify real AI crawlers.

Google's Web Bot Auth experiment (agent.bot.goog)

Web Bot Auth lets a bot sign its requests so you can verify them cryptographically, not just by IP. Google's Web Bot Auth page describes its implementation as experimental, built on a draft IETF specification that may change. Google's changelog dates the documentation to 4 May 2026.

What Google documents:

  • Signed requests are authenticated as https://agent.bot.goog.
  • Participating requests carry a Signature-Agent header set to g="https://agent.bot.goog", plus Signature and Signature-Input headers labeled g.
  • The public keys are at https://agent.bot.goog/.well-known/http-message-signatures-directory. Google says to cache them according to the Cache-Control header and to drop cached keys that disappear from the file.
  • Not all Google user agents use it, and only some Google-Agent requests are signed. Google recommends keeping IP and user agent checks alongside it.

Your options are to use a CDN or bot-management product that verifies Web Bot Auth for you (Google says some infrastructure services let you look up Google-Agent and allowlist it), to verify signatures yourself using RFC 9421, or to fall back on IP checks. For the protocol itself, see Web Bot Auth and signed crawlers.

What to do if a WAF blocks Google-Agent

Google does not document how Google-Agent responds to a block, so test instead of guessing. Whether you want it blocked or allowed is your call. If you want to allow it, make sure your firewall is not stopping it by accident.

  1. Look for it in your edge logs, not just origin logs. A request stopped at the CDN never reaches your server. Search your CDN's security events for the Google-Agent token and for the IP ranges in the JSON file.
  2. Check the status codes. A 403 or 429 for an IP that passes the check above suggests a bot rule or rate limit. A challenge page can also be served in place of your content; see WAF challenge pages and AI crawlers for how to spot one.
  3. Allow by IP, not by user agent alone. An allow rule that trusts the string Google-Agent lets spoofers through. Match the user agent token and the published IP ranges together, and refresh the ranges automatically.
  4. Block fakes, not the real one. A request that claims Google-Agent from an IP outside the file is not Google-Agent, and blocking it is safe.
  5. If you want it out, enforce that at your firewall or with authentication on the sensitive paths. Since the agent can take actions, put anything that changes state behind a login or other server-side protection whatever the user agent says.

Frequently asked questions

Is Google-Agent the same as Googlebot?

No. Googlebot is a common crawler that Google says always respects robots.txt for automatic crawls. Google-Agent is listed under user-triggered fetchers, with its own user agent, its own entry in Google's documentation and its own IP file.

Can I block Google-Agent with robots.txt?

Google says user-triggered fetchers generally ignore robots.txt rules, and its Google-Agent entry documents no robots.txt control. Use a firewall rule on verified IPs or authentication, not robots.txt.

Google's Google-Agent entry does not say it feeds search results, and Googlebot is a separate user agent. Check your own results after any firewall change instead of assuming. Rules that catch Googlebot by mistake can hurt your search crawling.

How do I know a Google-Agent request is real?

Match the source IP against user-triggered-agents.json, or run the reverse and forward DNS check Google describes. A matching user agent string alone proves nothing, and Google says it can be spoofed.

Does Google-Agent sign its requests?

Some of them, as an experiment. Google says only a subset of Google-Agent requests use Web Bot Auth, so you cannot rely on a signature being present.

Next step

If you want to know what AI crawlers see on your pages, the free check reads one URL the way they do, with no account. It runs from Crawlito's servers, not Google's, so it cannot tell you how Google-Agent is treated by your firewall: your edge logs hold that evidence. For a whole site, an audit covers up to 500 pages with evidence and prioritized fixes.

Filed under AI crawlers. Topics: google, ai-crawlers, robots-txt, web-bot-auth, server-logs