Googlebot-Image, Googlebot-Video and Googlebot-News explained
What the Googlebot-Image, Googlebot-Video and Googlebot-News robots.txt tokens control, how they fall back to Googlebot rules, and how to test them.
By Crawlito · 8 min read
In short
- Google says rules for the Googlebot-Image token affect Google Images, Discover, Video mode and every Search feature that shows images, logos or favicons.
- Googlebot-Image, Googlebot-Video and Googlebot-News each list Googlebot as a second robots.txt token, and Google says a crawler needs to match only one token for a rule to apply.
- With no group naming Googlebot-Image, a Googlebot group applies to it; only when neither exists does it fall back to the * group.
- Googlebot-News has no user agent string of its own: Google says crawling is done with various Googlebot user agent strings, so you will not find it in your logs.
- Blocking Googlebot-Image keeps images out of Google Images, while blocking Googlebot removes them from all Google searches, according to Google's image removal guide.
On this page
Googlebot-Image, Googlebot-Video and Googlebot-News are robots.txt tokens that let you set rules for Google's image, video and news surfaces without touching the rules for web search. A group for Googlebot-Image controls whether your images appear in Google Images, Discover, Video mode and the Search features that show images, logos and favicons. If no group names one of these tokens, it follows your Googlebot group, and only falls back to * when there is no Googlebot group either.
That fallback is the part most people get wrong, so this post starts with what each token controls, then shows exactly where the fallback comes from in Google's documentation, what blocking each token does, and how to test your file. For the general rules of group selection (one group per crawler, no merging with *), read robots.txt user-agent groups explained first; this post does not repeat them.
What each Google token controls
Google lists all three on its common crawlers page, which was last updated on 30 October 2026 at the time of writing. Each entry gives a user agent string, the tokens you can use in robots.txt and the products affected.
| Crawler | User agent in HTTP requests | robots.txt tokens | What Google says the rules affect |
|---|---|---|---|
| Googlebot Image | Googlebot-Image/1.0 | Googlebot-Image, Googlebot | Google Images, Discover, Video mode, and all Search features where images, logos and favicons are presented |
| Googlebot Video | Googlebot-Video/1.0 | Googlebot-Video, Googlebot | Video-related Google Search features and other products dependent on videos |
| Googlebot News | None of its own | Googlebot-News, Googlebot | The Google News product, including news.google.com and the Google News app |
| Googlebot | Smartphone and desktop strings | Googlebot | Google Search, including Discover and all Search features, plus Google Images, Video mode and Google News |
Two details matter in practice:
- Googlebot-News is a token, not a visitor. Google's page says it "doesn't have a separate HTTP request user agent string" and that crawling is done with various Googlebot user agent strings. You can write rules for it, but you will never see
Googlebot-Newsin an access log. Our Googlebot user agent strings post covers what you will see. - The Googlebot token reaches everything. Rules for
Googlebotaffect Google Images, Video mode and Google News as well as web search. The narrower tokens exist so you can restrict one surface without restricting the rest.
How these tokens fall back to Googlebot rules
The fallback is not stated as a separate rule. It follows from two things Google documents.
First, the common crawlers page lists two tokens for each of these crawlers, the specific one and Googlebot, and says:
"Some crawlers have more than one user agent token; you need to match only one crawler token for a rule to apply."
Second, Google's robots.txt specification says only one group is valid for a crawler: the group "with the most specific user agent that matches the crawler's user agent". A group naming Googlebot matches Googlebot-Image through its second token and is more specific than *, so it wins when there is no Googlebot-Image group. Google's open-source robotstxt library says the same in its README, giving Googlebot-Image respecting the rules for User-agent: Googlebot as an example of logic a crawler might apply outside the parser.
Put together, the order for Googlebot-Image (and likewise Googlebot-Video and Googlebot-News) is:
- A group that names
Googlebot-Image. Only that group applies. - If none, a group that names
Googlebot. - If neither, the
*group. - If there is no
*group either, no rules apply, as RFC 9309 puts it.
Here is one file and the group each crawler ends up with:
User-agent: *
Disallow: /private/
User-agent: Googlebot
Disallow: /drafts/
User-agent: Googlebot-Image
Disallow: /photos/originals/| Crawler | Group it follows | /drafts/a.jpg | /photos/originals/b.jpg | /private/c.mp4 |
|---|---|---|---|---|
| Googlebot | Googlebot | Blocked | Allowed | Allowed |
| Googlebot-Image | Googlebot-Image | Allowed | Blocked | Allowed |
| Googlebot-Video | Googlebot (no own group) | Blocked | Allowed | Allowed |
| Googlebot-News | Googlebot (no own group) | Blocked | Allowed | Allowed |
Note the trap in the second row: because Googlebot-Image has its own group, it no longer reads the /drafts/ rule from the Googlebot group. If you want both, repeat the rule inside the Googlebot-Image group. And note the third column for all three: /private/ lives only in the * group, which none of these four crawlers read once a Googlebot group exists. Google crawlers that do not list Googlebot as a token, such as Storebot-Google, still use * here.
One more case from Google's own example table: Googlebot-News follows its own group even when it fetches images, because, in Google's words, it "doesn't crawl the images for Google Images".
What blocking each token does
Googlebot-Image
Google's image removal guide says that "by specifying Googlebot-Image as the User-agent, the images will be excluded from Google Images", and that to exclude images from all Google searches, including web Search, you specify Googlebot instead. The next time Google crawls the blocked image, it sees the rule and drops the image from its results. Google does not give a timeframe, only that robots.txt is slower than the Removals tool.
Because the common crawlers page also lists Discover, Video mode and the features that show logos and favicons, a broad Googlebot-Image block reaches further than the Images tab. A rule that covers your icon file can keep your favicon out of results: the favicon troubleshooting guide walks through that case. If the wrong picture shows next to your result instead, see wrong image in Google search results.
Googlebot-Video
Google describes the affected products only as video-related Search features and products that depend on videos. Its video best practices add two requirements that a careless block can break: the video thumbnail must be accessible to Googlebot and Googlebot Images, and Google needs to fetch the actual video bytes for features like video previews and key moments, so you should not block the video file URL with robots.txt. A video thumbnail is an image, so a Googlebot-Image rule on your thumbnail folder can affect your video results too.
Googlebot-News
Google's Publisher Center help on blocking access to content says that to block Google News only, you block Googlebot-News in robots.txt, and to block Google News and Google Search, you block Googlebot. The same page offers a meta tag for single articles, <meta name="Googlebot-News" content="noindex, nofollow">. For getting into Google News rather than out of it, see the Google News sitemap guide.
robots.txt examples for each surface
Keep images out of Google Images but keep web search unchanged. Remember to copy any Googlebot rules you still need into the new group:
User-agent: Googlebot
Disallow: /checkout/
User-agent: Googlebot-Image
Disallow: /checkout/
Disallow: /images/press-archive/Block one file type from image results, using Google's own pattern:
User-agent: Googlebot-Image
Disallow: /*.gif$Keep a section out of Google News while leaving it in Search:
User-agent: Googlebot-News
Disallow: /sponsored/Keep video files and thumbnails reachable while restricting other media:
User-agent: Googlebot-Image
Disallow: /uploads/
Allow: /uploads/video-thumbnails/
User-agent: Googlebot-Video
Allow: /The Allow line wins in the first group because it is the longer matching path. The Googlebot-Video group with only Allow: / is deliberate: without it, Googlebot-Video would follow your Googlebot group, and any Disallow there would apply to it. Google does not document which crawler fetches the video bytes, and the thumbnail must be reachable by Googlebot as well as Googlebot Images, so check that your Googlebot group does not block those paths either.
When robots.txt is the wrong tool
robots.txt stops crawling; it is not always the best way to stop indexing. Google's image removal guide offers a noindex X-Robots-Tag on the image response as an alternative with the same effect, and warns that Googlebot has to crawl the image URL to see that header, so do not combine it with a robots.txt block on the same file. For an urgent case it points to the Removals tool, noting that images may resurface once the removal expires unless you also block or remove them. The noimageindex robots tag on a page stops images embedded in that page from being indexed, but Google notes the same images can still be indexed through other pages.
How to test which group a Google crawler follows
- Fetch the live file the crawler sees, on the exact host that serves the images. Google's image removal guide says the robots.txt goes at the root of the site that hosts the image, so images on a CDN host such as
cdn.example.comare governed by that host's robots.txt, not your main site's.
curl -s https://cdn.example.com/robots.txt- List the groups and note whether
Googlebot-Image,Googlebot-Video,Googlebot-NewsandGoogleboteach appear. - Run a parser once per token. Google's robotstxt library builds a
robots_mainbinary, set up as in how to test robots.txt for AI crawlers. Its README says it does not apply the Googlebot fallback, so choose the token yourself: testGooglebot-Imageif your file names it, andGooglebotif it does not.
./bazel-bin/robots_main robots.txt Googlebot-Image https://cdn.example.com/uploads/logo.png
./bazel-bin/robots_main robots.txt Googlebot https://cdn.example.com/uploads/logo.png- Check your logs for
Googlebot-Image/1.0andGooglebot-Video/1.0requests after a change. There is nothing to look for with Googlebot-News, because it has no string of its own.
Frequently asked questions
Does blocking Googlebot-Image remove my pages from Google Search?
No. Google's image removal guide says a Googlebot-Image block excludes images from Google Images; to remove them from all Google searches you block Googlebot. Your pages stay in web search, though image-based features such as favicons and Discover images can be affected.
If my robots.txt has no Googlebot-Image group, which rules does it follow?
It follows your Googlebot group, because Google lists Googlebot as a second token for Googlebot-Image and says matching one token is enough. If there is no Googlebot group, it follows *.
Why can't I find Googlebot-News in my server logs?
Google says Googlebot-News has no separate user agent string and crawls with the regular Googlebot strings. The token only exists for robots.txt and meta tag rules.
Do these tokens affect Google AI Overviews or Gemini training?
Their entries name Google Images, Discover, Video mode, video features and Google News; AI Overviews is not named, though the Googlebot-Image entry covers "all features in Google Search" that present images. Use of your content for Gemini training and grounding is managed with the separate Google-Extended token, which Google says does not affect inclusion in Google Search.
Check what Google's crawlers can reach
A wrong group for one of these tokens rarely shows up as an error: images or a favicon just quietly disappear. After changing your file, run the free check on an important page; it needs no account. For a site with images on several hosts, a full-site audit covers up to 500 pages and comes with evidence and prioritized fixes.
Filed under AI crawlers. Topics: googlebot, google, robots-txt, images