Skip to content

GoogleOther and Google-CloudVertexBot: what they crawl

GoogleOther is Google's generic crawler; Google-CloudVertexBot crawls only on a site owner's request. See what each does, what blocking changes and how to verify.

By Crawlito · 8 min read

In short

  • Google describes GoogleOther as a generic crawler that various product teams may use to fetch public content, for example one-off internal research and development crawls, and says its robots.txt rules do not affect any specific product.
  • Google says Google-CloudVertexBot crawls only at site owners' request while they build Vertex AI Agents, and that it has no effect on Google Search or other products.
  • Neither crawler is the documented Gemini control: Google-Extended is the robots.txt token for Gemini training and grounding, and it never appears in logs.
  • A robots.txt group for Googlebot does not apply to GoogleOther or Google-CloudVertexBot, because Google's crawlers follow the group that most specifically matches their own token.
  • Both crawlers can be verified like Googlebot: Google says they generally crawl from the ranges in common-crawlers.json, with reverse DNS on googlebot.com or geo.googlebot.com.
On this page

GoogleOther is a generic Google crawler that product teams can use to fetch publicly accessible content, and Google gives "one-off crawls for internal research and development" as an example. Google-CloudVertexBot is a different thing: it crawls only when a site owner asks for it, while building a Vertex AI Agent. Google says GoogleOther's rules don't affect any specific product and that Google-CloudVertexBot has no effect on Google Search or other products. By Google's description, neither one is part of Search, and neither is the documented switch for Gemini.

The Google facts in this post come from Google's common crawlers documentation, its crawler verification guide, its search updates changelog and Google Cloud's Vertex AI documentation, all read on 11 November 2026. Crawler documentation changes, so treat the details as current at the time of writing.

What is GoogleOther (and GoogleOther-Image and GoogleOther-Video)?

Google lists GoogleOther among its common crawlers, alongside Googlebot. It describes it as "the generic crawler that may be used by various product teams for fetching publicly accessible content from sites." The documentation gives one example use, one-off crawls for internal research and development, and says crawling preferences for GoogleOther don't affect any specific product.

There are three variants, each with its own robots.txt token:

TokenWhat Google says it isLog token to search for
GoogleOtherThe generic crawlerGoogleOther in a longer browser-style user agent
GoogleOther-ImageThe version optimized for fetching publicly accessible image URLsGoogleOther-Image/1.0
GoogleOther-VideoThe version optimized for fetching publicly accessible video URLsGoogleOther-Video/1.0

The two media variants were added in Google's changelog on 16 May 2024, described as versions of GoogleOther optimized for fetching image and video bytes. Google's documentation says all of these crawlers "always obey robots.txt rules when crawling automatically."

What Google does not say is which teams use GoogleOther, or what the fetched content is used for beyond that generic description. If you see it in your logs, the documented answer is: a Google crawler fetching public pages for an unspecified product or research purpose, not Google Search indexing.

What is Google-CloudVertexBot and who triggers it?

Google-CloudVertexBot is a site-owner-requested crawler. Google's changelog added it on 20 August 2024, and the documentation says its robots.txt preferences "affect crawls requested by the site owners' for building" Vertex AI Agents. Google adds that it has no effect on Google Search or other products.

Google Cloud's own documentation shows what "requested" means in practice. When someone creates a website data store with advanced website indexing, Google Cloud says the domain must be verified: "If you don't verify your domain ownership, your website can't be indexed using advanced website indexing." Verification is by the domain owner, or through a domain-association request that the owner approves in Search Console. The documentation also says that if you can't verify ownership, your Google customer engineer or account team can submit a domain-verification exemption request.

So a visit from Google-CloudVertexBot generally means that you, someone on your team, or a vendor working with your domain asked Google Cloud to crawl and index your site. The Google Cloud pages say that page crawling uses Google-CloudVertexBot, and that the user-agent needs access to your content. They don't describe the crawler visiting sites at random, and Google's documentation does not say that it does.

Two practical details from the same pages:

  • If you submit a sitemap, Google Cloud says it fetches the sitemap file with Googlebot, so Googlebot must be allowed as well. If your robots.txt permits Google-CloudVertexBot but blocks Googlebot, pages are still crawled, but the sitemap cannot be fetched.
  • A recrawl that robots.txt blocks fails with a "Crawling was blocked by the site's robots.txt" error, which the documentation treats as not transient. The fix it gives is to unblock the URL and make sure the Google-CloudVertexBot user agent is permitted.

Does blocking GoogleOther affect Google Search or Gemini?

Google Search: no, by Google's own description. Search indexing is Googlebot's job. Google says GoogleOther preferences don't affect any specific product and that Google-CloudVertexBot has no effect on Search.

Gemini: neither is the documented control. The control Google documents for Gemini is Google-Extended, a token in robots.txt that governs whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google says Google-Extended doesn't have its own user agent string and doesn't affect inclusion in Google Search. Our Google-Extended guide covers it in full.

Here is the honest limit of what is known. Google's documentation says GoogleOther may be used by "various product teams" and doesn't list those products. So you cannot read a GoogleOther block as a Gemini opt-out, because Google doesn't document it as one, and Google doesn't say what GoogleOther fetches are used for either way. If your goal is keeping content out of Gemini training, Google-Extended is the documented route. If your goal is stopping generic Google fetches that you can't tie to a product, a GoogleOther rule is a reasonable choice, with the understanding that Google says it won't change anything specific.

GoogleOther vs Google-Extended vs Googlebot

These names get mixed up because all three begin with "Google" and all appear in AI-crawler lists. They do different jobs.

GooglebotGoogleOtherGoogle-CloudVertexBotGoogle-Extended
Is it a crawler?YesYes (plus Image and Video variants)YesNo, a robots.txt token only
Appears in logs?YesYesYesNo
Documented purposeBuilding Google's search indexesGeneric fetching by various product teamsCrawls requested by site owners building Vertex AI AgentsControls Gemini training and grounding use
Affects Search?YesGoogle says no specific product is affectedGoogle says noGoogle says no
Obeys robots.txt?YesYesYesIt is a robots.txt control

The row that matters most for ranking questions is the first: Googlebot is the only one here that builds the Search index. The training vs search vs user crawlers guide puts these Google agents next to their OpenAI, Anthropic and Perplexity equivalents, and the AI crawler user agents list lists each token in one place.

How to block or allow each in robots.txt

Each of these tokens is matched on its own. Google says its crawlers use the group with the most specific user agent that matches the crawler, and that specific groups and the * group are not combined. Two consequences follow:

  • A Googlebot group does not apply to GoogleOther or Google-CloudVertexBot. Their tokens are different.
  • If no group names them, they follow the * group. A blanket User-agent: * with Disallow: / blocks them, along with Googlebot if Googlebot has no group of its own. See robots.txt user-agent groups for how that plays out.

Block GoogleOther and its variants

Google lists each variant as its own token, so name each one you want covered:

User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
Disallow: /

Googlebot has no named group in this file, so it keeps its existing access.

Allow Google-CloudVertexBot for a Vertex AI data store

If you use a blanket block under * but want a Vertex AI Agent to index your site, name the crawler. This is the form Google Cloud shows, and you should add a Googlebot allowance if you use a sitemap:

User-agent: Google-CloudVertexBot
Allow: /

Block Google-CloudVertexBot

If no one at your company has requested a Vertex AI crawl and you want to be sure it stays that way:

User-agent: Google-CloudVertexBot
Disallow: /

Google Cloud documents that a recrawl blocked by robots.txt fails with the blocked-by-robots.txt error, so tell anyone who runs a Vertex AI data store on your domain before you add it. Google's examples also show partial rules, such as allow: /archive/1Q84 followed by disallow: /archive/, for any of these tokens.

Google says it generally caches robots.txt for up to 24 hours, and the cache can last longer if it can't refresh the file, so a change may not apply immediately. Confirm in your logs rather than assuming. The test robots.txt guide shows how to test a file against a named token.

How to verify Google crawler requests in your logs

Anyone can type "GoogleOther" into a user agent, so verify before you act on a log line. Start by finding the requests, matching on the token rather than the whole string, since the browser-style version numbers change:

bash
grep -E "GoogleOther|Google-CloudVertexBot" access.log | awk '{print $1, $7, $9}' | tail -20

GoogleOther also matches GoogleOther-Image and GoogleOther-Video. Then check the source IP in one of two ways.

Reverse and forward DNS. Google's manual method: run a reverse lookup on the IP, confirm the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and confirm it returns the original IP. For the common crawlers, the page gives the reverse DNS pattern as crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com.

bash
host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

That example IP is the one Google uses in its guides. The first lookup should return the googlebot.com name, and the second should return the IP you started with.

The published IP ranges. For many requests, match the IP against Google's common-crawlers.json, which Google says these crawlers generally use. The file lists CIDR prefixes:

bash
python3 -c 'import ipaddress,json,sys,urllib.request; ip=ipaddress.ip_address(sys.argv[1]); d=json.load(urllib.request.urlopen("https://developers.google.com/static/crawling/ipranges/common-crawlers.json")); print(any(ip in ipaddress.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"]) for p in d["prefixes"]))' 66.249.66.1

It prints True if the IP is inside a listed range. The file changes, so fetch it fresh instead of copying ranges into a firewall. Google keeps separate files for special-case crawlers and user-triggered fetchers, so a request from one of those will not match this file. The wider approach is in how to verify a real AI crawler, and finding AI crawler visits in your server logs covers the log side.

Frequently asked questions

Is GoogleOther the same as Googlebot?

No. Googlebot builds Google's search indexes. GoogleOther is a separate generic crawler with its own robots.txt token, and Google says its preferences don't affect any specific product.

Should I block GoogleOther?

Google gives no product reason to, and says its rules affect no specific product. Block it if you want to turn away generic Google fetches, but use Google-Extended if your aim is to opt out of Gemini training and grounding.

Why is Google-CloudVertexBot visiting my site?

Google says it crawls at site owners' request while they build Vertex AI Agents. Check whether you, a colleague or a vendor set up a Vertex AI data store that includes your domain. Google Cloud's documentation requires domain verification for advanced website indexing.

Does GoogleOther obey robots.txt?

Yes, according to Google's documentation, which says its common crawlers always obey robots.txt rules when crawling automatically. Verify the requests first, because a fake user agent will not follow your rules.

Will a Googlebot rule block GoogleOther?

No. Google's crawlers follow the group that most specifically matches their own token, and a Googlebot group does not match GoogleOther. Name GoogleOther in its own group.

Check what your robots.txt says to Google

If you are not sure which of these rules your live site gives to Google's crawlers, the free check reads one page the way AI crawlers do, with no account needed. It looks at the robots.txt rules, the robots meta tag and headers, and whether the main text is in the raw HTML. For a whole site, an audit runs the checks across up to 500 pages with evidence and prioritized fixes.

Filed under AI crawlers. Topics: google, googleother, ai-crawlers, robots-txt, server-logs