Skip to content

Blogger custom robots.txt: add AI crawler rules on Blogspot

How to use Blogger custom robots.txt to allow or block AI crawlers on Blogspot without losing the default /search rule, the AdSense group or your sitemap.

By Crawlito · 8 min read

In short

  • Blogger's custom robots.txt replaces the default file completely, so copy the default Mediapartners-Google group, the /search rule and the Sitemap line into your version.
  • The default Blogspot robots.txt we fetched in November 2026 disallows /search and /share-widget for all crawlers and gives Mediapartners-Google, the AdSense crawler, an empty Disallow.
  • A named group such as User-agent: OAI-SearchBot ignores the * group, so repeat Disallow: /search in it if you want label and search pages kept out.
  • Google-Extended is still a robots.txt token on a Google-hosted blog: Google says it has no separate user agent and does not affect Google Search inclusion or ranking.
  • Blogger describes its custom robots header tags as tags served to search engines, with no crawler name, so use named robots.txt groups to treat one AI crawler differently from another.
On this page

Blogger lets you replace your blog's robots.txt under Settings > Crawlers and indexing > Enable custom robots.txt. Whatever you type there is served instead of the default file, so the safe method is to copy the default rules first, then add named groups for the AI crawlers you want to block or allow. Page-level controls live next to it under Enable custom robots header tags, but those apply to all crawlers at once, so for AI crawler rules robots.txt is the right tool.

This post covers what the default Blogspot file contains, how to add AI crawler groups without losing the sitemap or AdSense lines, what the header tags can and cannot do, and how Google-Extended works on a blog that Google itself hosts. For the general rules behind each line, read robots.txt for AI crawlers first.

Where Blogger's robots.txt settings are

Blogger's settings help page lists three items in the Crawlers and indexing section:

SettingWhat Blogger says it does
Enable custom robots.txt"This text will be used instead of the default robots.txt content to search engines."
Enable custom robots header tagsSets the robots header tags served to search engines
Google Search ConsoleA link to Search Console

Two other settings affect crawlers too. Under Privacy, Visible to search engines allows search engines to find your blog, so keep it on if you want any crawler to read your posts. Under Publishing, a custom domain changes the host your robots.txt lives on: Blogger does not allow naked domains, so the file sits at https://www.yourdomain.com/robots.txt or https://blog.yourdomain.com/robots.txt, not at the bare domain.

Google's own guide to creating robots.txt notes that on hosting services such as Blogger you may not be able to edit the file directly and may get a settings page instead. Blogger is the case where you do get a text box.

What the default Blogspot robots.txt contains

Blogger's help page does not publish the default file, so we fetched it from two Google-run Blogger blogs on 15 November 2026. Both returned the same rules, with only the sitemap host changed:

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Disallow: /share-widget
Allow: /

Sitemap: https://blogger.googleblog.com/sitemap.xml

Line by line:

  • User-agent: Mediapartners-Google with an empty Disallow: lets Google's AdSense crawler fetch everything. Google's special-case crawlers page says this crawler visits participating sites to show relevant ads and that it ignores the global * group, which is why it gets a group of its own.
  • Disallow: /search covers more than search results. On Blogger, label pages live under /search/label/, so this rule keeps both out of crawlers that follow the * group.
  • Disallow: /share-widget blocks one more Blogger path. Blogger does not document what it serves, so keep the line as it is.
  • Allow: / allows everything else. It changes nothing on its own, since unmatched paths are allowed anyway.
  • Sitemap: points at /sitemap.xml, which on the blogs we checked lists posts. Static pages were in a separate /sitemap-pages.xml that the default file does not mention.

The default file has no AI crawler groups. Every AI crawler that follows robots.txt falls through to * and may read all posts and pages.

How to add AI crawler rules without breaking the default

Turn on Enable custom robots.txt, then edit the text. Because your text replaces the default file instead of being added to it, any default line you leave out is gone. Start from the default, then add your groups.

A version that blocks AI training tokens and keeps AI search allowed looks like this:

User-agent: Mediapartners-Google
Disallow: 

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /search
Disallow: /share-widget
Allow: /

Sitemap: https://yourblog.blogspot.com/sitemap.xml
Sitemap: https://yourblog.blogspot.com/sitemap-pages.xml

Replace yourblog.blogspot.com with your real address, or your custom domain if you use one. The reasoning behind which tokens go in the training group is in allow AI search, block AI training. Three Blogger-specific points:

  1. Keep the Mediapartners-Google group if you run AdSense. It ignores *, so it needs its own group to stay allowed.
  2. Write the sitemap URL by hand. The Sitemap: line has to be a fully qualified URL, and Google says it does not check http, https, www or non-www variants. The Sitemap directive guide explains the rules. Adding sitemap-pages.xml is optional but makes your static pages easier to find.
  3. Repeat /search in any group you add for search crawlers. Under RFC 9309, a crawler that finds a group naming it uses only that group. If you add User-agent: OAI-SearchBot with Allow: / and nothing else, OAI-SearchBot can crawl every label and search URL on the blog. The robots.txt groups guide shows more cases.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /search
Disallow: /share-widget
Allow: /

You do not need this group for those crawlers to be allowed, since the * group already allows them. Add it only if you want your intent written down, and keep its rules in step with *.

User-agent matching is case-insensitive under RFC 9309, but path values are case-sensitive in Google's robots.txt rules (RFC 9309 says matching should be), so Disallow: /Search would not block /search.

What Blogger's custom robots header tags do

Enable custom robots header tags sets robots rules for your pages instead of for paths. Blogger's help page describes these as flags for "the robots header tags served to search engines" and links to Google's robots meta tag documentation for their meaning. It does not list the individual flags, so check the settings screen for the options and page types offered at the time of writing.

What the rules mean comes from Google's page:

RuleEffect at Google
allNo restrictions, the default
noindexKeeps the page out of search results
nofollowGoogle does not follow the page's links
noneSame as noindex, nofollow
nosnippetNo text snippet or video preview, including in AI Overviews and AI Mode
noimageindexImages on the page are not indexed
notranslateNo translation offered in results
unavailable_afterDrops the page from results after a date
noarchiveNot used by Google, because the cached link feature no longer exists

Two limits matter for AI crawlers:

  • They are not per crawler. Blogger describes them as tags served to search engines and does not mention naming a crawler, so noindex on posts applies to every crawler that honours it, including Googlebot. You cannot use them to keep GPTBot out while letting OAI-SearchBot in. Use named robots.txt groups for that.
  • They only work on pages that can be crawled. Google says that if robots.txt disallows a page, its indexing rules are never seen and are ignored. A noindex on label pages does nothing for a crawler that the /search rule already keeps out.

The one AI-related job they do well is limiting Google. Google says nosnippet stops content being used as a direct input for AI Overviews and AI Mode, but it also removes your ordinary search snippets, so think twice before setting it blog-wide.

Why Google-Extended still applies on a Google-hosted blog

A Blogspot blog is hosted by Google, which leads many owners to assume that Google's AI controls work differently there. Google's common crawlers page describes Google-Extended as a product token that publishers use to manage whether content Google crawls from their sites may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It makes no exception for Blogger, and the token works the same way in your Blogger robots.txt as anywhere else.

Facts worth knowing before you add it:

  • It is a token, not a crawler. Google says Google-Extended has no separate user agent string. Crawling is done by Google's existing crawlers, so you will not see a Google-Extended user agent in your logs.
  • It does not affect Google Search. Google says it does not change inclusion in Google Search and is not a ranking signal. AI Overviews and AI Mode are part of Search and are governed by Googlebot and snippet rules instead.
  • The crawler docs cover crawled content only. They do not say how Google treats Blogger content under its product terms, so if that matters to you, read the terms rather than relying on robots.txt.

Google also says it intends to support the AIPREF training preference signal, written as user-agent: __usage_ai-training, in early 2027. Until it does, Google-Extended is the documented control. Google-Extended explained covers the token in more depth.

How to test your Blogger robots.txt

  1. Fetch the live file from every address you use. If you have a custom domain, check both it and the blogspot address, because the file applies only to the host that serves it.

    bash
    for host in yourblog.blogspot.com www.yourdomain.com; do
      echo "== $host"; curl -sL "https://$host/robots.txt"
    done

    You should see your custom text, not the default. If you still see the default, confirm the setting is switched on and saved.

  2. Read it as each crawler would. Find the group naming the crawler, or fall back to *, then check a post URL and a /search/label/ URL against it. The robots.txt testing guide shows how to do this with a parser.

  3. Use the robots.txt report in Search Console to see the version Google fetched and any parse problems. Google says the report only works for files that are already accessible on your site, and only for domain-level properties, which includes a URL-prefix property with no path such as https://yourblog.blogspot.com/.

  4. Check a post for robots rules if you changed the header tags. Look at both the response headers and the HTML, and allow for single or double quotes, since Blogger themes often use single quotes:

    bash
    curl -sD - -o post.html https://yourblog.blogspot.com/2026/11/your-post.html | grep -i 'x-robots-tag'
    grep -iE "name=['\"]robots" post.html

    No output from either means no page-level restriction.

Allow time for crawlers to pick up changes. RFC 9309 says crawlers should not use a cached robots.txt for more than 24 hours unless the file cannot be reached. Google says it generally caches robots.txt for up to 24 hours and may adjust that based on the Cache-Control max-age header. The Blogspot files we fetched were served with cache-control: private, max-age=86400, which is also 24 hours.

Frequently asked questions

Does Blogger's custom robots.txt add to the default file?

No. Blogger says the custom text is used instead of the default, so copy the Mediapartners-Google group, the /search rule and the Sitemap line into your version.

Can I block GPTBot with Blogger's custom robots header tags?

No. Blogger describes the header tags as rules served to search engines, with no way documented to name one crawler. Add a User-agent: GPTBot group with Disallow: / to your custom robots.txt instead.

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls Gemini training and grounding use of crawled content.

Why does Blogger block /search in robots.txt?

The default rule keeps crawlers out of search result URLs and label pages, which on Blogger live under /search/label/. Your posts and static pages are not affected.

Do I need a separate robots.txt for my custom domain?

There is one setting per blog, and Google applies a robots.txt file only to the host that serves it. Blogger does not document how the blogspot address handles robots.txt once a custom domain is set, so fetch /robots.txt on both and confirm each shows, or redirects to, the rules you expect.

Check what your blog serves

After saving your custom robots.txt, run the free single-page check on one of your posts to see which AI crawlers your live robots.txt allows and whether any page-level robots rule applies. It checks one page and needs no account.

Filed under Platform guides. Topics: robots-txt, ai-crawlers, meta-robots, google-extended, blogger