Google-Extended: what it controls and what it doesn't
Google-Extended is a robots.txt token, not a crawler. It covers Gemini training and grounding, not Google Search rankings or AI Overviews. Here is how to use it.
By Crawlito · 8 min read
In short
Google-Extended is a robots.txt product token, not a crawler: Google crawls with its existing user agents, so you will never see Google-Extended in your server logs.
Disallowing Google-Extended tells Google not to use your content to train future Gemini models or for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.
Google says Google-Extended does not affect whether a site is included in Google Search and is not used as a ranking signal.
Google-Extended does not control AI Overviews or AI Mode: those are part of Search and follow Googlebot rules plus noindex, nosnippet, data-nosnippet and max-snippet.
Blocking Googlebot to opt out of AI stops Google crawling your pages for Search as a whole, and Applebot follows Googlebot rules when robots.txt does not name Applebot.
Google-Extended is a robots.txt token that lets you tell Google not to use your content for Gemini: training future Gemini models, and grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. It is not a separate crawler, it has no effect on whether your pages are in Google Search or how they rank, and it does not keep your content out of AI Overviews or AI Mode. Those are part of Search, so they follow your Googlebot rules and snippet controls instead.
Google describes Google-Extended as a "standalone product token" that publishers can use to manage whether content Google crawls may be used:
to train future generations of the Gemini models that power Gemini Apps and the Vertex AI API for Gemini, and
for grounding, which Google defines as providing content from the Google Search index to the model at prompt time, in Gemini Apps and in Grounding with Google Search on Vertex AI.
The important word is token. Google says Google-Extended "doesn't have a separate HTTP request user agent string." Crawling is done with Google's existing user agent strings, and the token is used only in robots.txt, as a control over how the content Google has crawled may be used. Applebot-Extended works the same way for Apple; the AI crawler user agents list shows which other vendors use usage-only tokens.
A short history from Google's Search Central changelog: the token was added on 28 September 2023, Google clarified on 8 February 2024 that it does not affect Google Search, and the description was updated on 25 April 2025.
What Google-Extended controls, and what it doesn't
This is the table most people need. It sets out which control applies to which Google surface, according to Google's own documentation.
Google surface
Controlled by Google-Extended?
What controls it
Training future Gemini models (Gemini Apps, Vertex AI API for Gemini)
Yes
User-agent: Google-Extended in robots.txt
Grounding in Gemini Apps and Grounding with Google Search on Vertex AI
Yes
User-agent: Google-Extended in robots.txt
Inclusion in Google Search
No
Googlebot rules in robots.txt, noindex
Ranking in Google Search
No
Google says Google-Extended is not a ranking signal
Google's AI features page puts it plainly: AI is built into Search, so robots.txt directives for Googlebot are how you manage crawling for Search, AI features included. Google-Extended is mentioned there only as the way to limit AI training and grounding in some of Google's other systems.
Why AI Overviews are not covered
The most repeated claim about Google-Extended is that blocking it keeps your content out of AI Overviews. It does not. To appear as a supporting link in AI Overviews or AI Mode, a page only needs to be indexed and eligible to be shown in Google Search with a snippet, and Google says there are no additional technical requirements. If you want to limit how your content is used there, the controls are the same ones that govern ordinary snippets:
nosnippet stops the page's content from being used as a direct input for AI Overviews and AI Mode, and removes text snippets from normal results too.
max-snippet:[number] limits how much content may be used as a direct input.
data-nosnippet excludes part of a page, and works only on span, div and section elements.
The common mistake: blocking Googlebot to opt out of AI
Some site owners, wanting nothing to do with Google's AI, block Googlebot. That works only in the worst sense. This file:
User-agent: Googlebot
Disallow: /
stops Google crawling the site for Search. Google can no longer read your pages, so they lose their content in normal results (a blocked URL can still be listed without a description if other pages link to it), and they stop being eligible for AI Overviews and AI Mode, which need a page that is indexed and can be shown with a snippet. It is also broader than you might expect: Apple's Applebot documentation says that if robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions. A Googlebot block can quietly stop Applebot too, and Applebot's crawl data powers search features in Spotlight, Siri and Safari.
The same applies to a blanket rule under *. If no group names Googlebot, Googlebot follows User-agent: *, so Disallow: / there takes you out of Search, not just out of AI features.
If what you want is "stay in Google Search, keep my content out of Gemini training and grounding," the correct file names Google-Extended and leaves Googlebot alone:
Googlebot has no named group here, so it follows the * group and keeps crawling everything except /admin/. Google-Extended has its own group and is disallowed from the whole site. Remember that a named group replaces the * group for that token: rules are not merged, as the robots.txt groups guide explains.
How to use Google-Extended in robots.txt
Google-Extended accepts normal path rules, so you can opt out only part of a site. Google's own example allows one page inside a disallowed directory:
If your goal is a consistent training opt-out across vendors, Google-Extended usually sits next to the other training controls. Each vendor documents its own token, and the search crawlers stay allowed:
Google-Extended covers grounding as well as training. Grounding, in Google's words, is providing content from the Google Search index to the model at prompt time, in Gemini Apps and in Grounding with Google Search on Vertex AI. Google lists grounding inside the token's scope, so disallowing Google-Extended is a signal against that use too. Do not assume the token is a training-only switch. Decide whether you want your content available to Gemini answers before you add it.
What about the new AIPREF signal?
On 30 October 2026 Google added a note that it intends to support the AIPREF shared training-preference signal, written in robots.txt as user-agent: __usage_ai-training, in early 2027. At the time of writing that is a stated plan, not a live control. Keep using Google-Extended today.
Check for rules you did not write
Platforms and CDNs sometimes add Google-Extended for you. Cloudflare's managed robots.txt adds a Disallow: / group for Google-Extended, among other AI crawlers, when Bot Preference Sync is turned on and at least one of your Search, Agent or Training policies is set to Block, Block on pages with ads, or Disallow. Squarespace's "Block known artificial intelligence crawlers" checkbox, which is off by default, also lists Google-Extended, along with GoogleOther. A Google-Extended rule does not affect Google Search, but you should know it is there.
How to check your Google-Extended setup
Because Google-Extended never appears as a user agent, you cannot confirm it from logs. You check the rules themselves.
Read the live file, not the one in your repository. CDNs and platforms can rewrite it:
Look for a Google-Extended group, any Googlebot group, and a blanket Disallow: / under *.
Confirm Googlebot is still allowed. If you see User-agent: Googlebot followed by Disallow: /, that is a Search block, not an AI opt-out.
Test the token against a URL. Google's open-source robots.txt parser builds a robots_main tool (with bazel build :robots_main) that takes a file, a user-agent token and a URL:
Each run prints ALLOWED or DISALLOWED. You want Google-Extended disallowed where you intended and Googlebot allowed.
Check what Google fetched. The Search Console robots.txt report shows the robots.txt files Google found for the top 20 hosts on your site, when it last crawled them and any errors. It cannot test individual URLs, which is why step 3 matters.
Remember that robots.txt applies per host, protocol and port. A rule on www.example.com does nothing for shop.example.com. For the wider testing workflow across every AI crawler, see how to test your robots.txt against each AI crawler.
What you will see in your logs
Only Google's real user agents, such as Googlebot. To confirm a Googlebot request is genuine, Google says to run a reverse DNS lookup on the IP, check the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP:
In Google's example, the first lookup returns crawl-66-249-66-1.googlebot.com and the second returns 66.249.66.1, the IP you started with. The details are in Google's verification guide.
Frequently asked questions
Does blocking Google-Extended hurt my Google rankings?
No. Google says Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal.
Does blocking Google-Extended keep my content out of AI Overviews?
No. AI Overviews and AI Mode are part of Search. They follow your Googlebot robots.txt rules and the noindex, nosnippet, data-nosnippet and max-snippet controls, not Google-Extended.
Why can't I find Google-Extended in my server logs?
Because it is not a crawler. Google says it has no separate user agent string: Google crawls with its existing user agents and reads the Google-Extended token from robots.txt only to decide how that content may be used.
Is blocking Google-Extended the same as blocking GoogleOther?
No. GoogleOther is a real generic crawler that various Google product teams may use, for example for one-off internal research crawls, and Google says rules addressed to it don't affect any specific product. Google-Extended is a usage token for Gemini training and grounding.
Will a Google-Extended rule remove content already used for training?
Google's documentation describes the token as controlling use for training future Gemini models. It does not say the token removes content from models that were already trained.
Next step
If you are not sure whether your live robots.txt opts out of Gemini, blocks Googlebot by accident or carries rules a CDN added, run the free single-page check on your site. It needs no account, and it checks the live site rather than the copy in your repository.
Filed under AI crawlers. Topics: · google-extended, google, robots-txt, ai-overviews, ai-crawlers
Google says AI Overviews and AI Mode need nothing beyond normal Search eligibility. Here is what that means, which controls apply and how to check each.
How nosnippet, max-snippet, data-nosnippet and noindex control AI Overviews and AI Mode, what other AI tools document, and how to set X-Robots-Tag for PDFs.