Skip to content

Meta name googlebot, bingbot and applebot: one-crawler rules

Which meta name values and X-Robots-Tag prefixes Google, Bing and Apple read, how Google combines them with robots rules, and why GPTBot is not a meta name.

By Crawlito · 8 min read

In short

  • Google reads only two crawler names in the robots meta tag, googlebot and googlebot-news, and ignores any other name value.
  • Google combines rules from a generic robots tag and a googlebot tag by adding up the negative rules, so a googlebot tag can add restrictions but cannot lift ones set for all crawlers.
  • Apple documents name="applebot" in meta tags and the applebot: prefix in X-Robots-Tag; Bing's own examples use name="bingbot" and the Bingbot: prefix.
  • robots.txt product tokens such as GPTBot, OAI-SearchBot, Google-Extended and Applebot-Extended are not documented as meta tag names, and Google says it ignores names it does not support.
  • A crawler can only read a meta tag or X-Robots-Tag header on a page it is allowed to fetch, so a robots.txt Disallow hides every crawler-specific rule on that page.
On this page

To aim a robots rule at one search engine, replace robots in the name attribute with that engine's crawler name: <meta name="googlebot" content="noindex"> for Google, name="bingbot" for Bing and name="applebot" for Apple. The HTTP header equivalent puts the name before the rules, as in X-Robots-Tag: googlebot: noindex. Those three names are the ones each operator documents. Names copied from robots.txt, such as GPTBot or Google-Extended, are not documented as meta names, and Google says it ignores names it does not support.

This guide covers the crawler-specific forms only. For what noindex, nosnippet and max-snippet do in AI Overviews and other AI tools, read noindex, nosnippet and X-Robots-Tag for AI search. For why a robots.txt block stops these tags from working at all, read robots.txt vs meta robots. Facts below come from Google's robots meta tag documentation, Apple's About Applebot page and Microsoft's Bing blog posts, read on 15 November 2026.

Which crawler names each operator documents

Each operator chooses which name values its crawler reads. Here is what each one documents at the time of writing:

OperatorMeta tag nameX-Robots-Tag formSource
Googlegooglebot (all text results), googlebot-news (news results)X-Robots-Tag: googlebot: noindexGoogle robots meta tag docs
Microsoft BingbingbotX-Robots-Tag: Bingbot: ai-grounding:disallowBing blog posts, 2023 and 2026
AppleapplebotX-Robots-Tag: applebot: nosnippetAbout Applebot
All of the aboverobotsX-Robots-Tag: noindex (no prefix, as Google documents it)Google and Apple docs, Bing 2023 post

Some details matter when you write these tags:

  • Google is strict about names. Its documentation says it supports two user agent tokens in the robots meta tag and that "other values are ignored". A tag named googlebot-image or google-extended does nothing in Google Search.
  • Case does not matter to Google. Google says both the name and content attributes are case-insensitive, and so are the header name, user agent name and values in X-Robots-Tag.
  • Bing's examples vary in case. Microsoft's 2023 post writes name="bingbot" and its October 2026 post writes name="Bingbot". We could not read Bing's help page on supported meta tags for this post, so it does not list every value Bing accepts.
  • Apple lists its directives. Apple documents noindex, nosnippet, nofollow, none and all, with examples using both name="robots" and name="applebot". It says noindex keeps a page out of Spotlight and Siri Suggestions, and nosnippet stops Applebot generating a description or web answer for the page.

A plain robots tag addresses every crawler that reads it. Google adds a caveat: the generic tag applies to search engine crawlers, and to restrict non-search crawlers such as AdsBot-Google you might need rules aimed at that crawler.

How Google combines robots and googlebot tags

You can put several robots meta tags on one page. Google's rule for a page that has both a generic tag and a crawler-specific one is that "the search engine will use the sum of the negative rules". Its own example:

html
<meta name="robots" content="nofollow">
<meta name="googlebot" content="noindex">

Googlebot treats this page as noindex, nofollow. Other crawlers that read only the robots tag see nofollow.

The consequence people miss is that a googlebot tag cannot loosen a generic rule. This does not keep the page indexable in Google:

html
<!-- Does not work: Google adds the rules together -->
<meta name="robots" content="noindex">
<meta name="googlebot" content="index, follow">

Google reads noindex from the first tag and keeps it. The same goes for snippet rules: Google says that when rules conflict, "the more restrictive rule applies", so nosnippet beats max-snippet:50. If you want a page out of every engine except Google, there is no Google-only override. Name each engine you want to restrict instead and leave the generic tag out:

html
<meta name="bingbot" content="noindex">
<meta name="applebot" content="noindex">

Bing does not publish a general combination rule in the sources we could read. Its 2023 example pairs a generic noarchive with a Bing-only nocache so that Bing treats the content as nocache, and the same post says content carrying both values is treated as nocache. That is a statement about those two values, not a general rule, so do not assume Bing follows Google's sum-of-negatives logic for every directive.

googlebot-news for Google News only

Google describes googlebot as covering "all text results" and googlebot-news as covering news results, so you can set a rule for news results separately. Google's example of addressing crawlers individually sets notranslate for Googlebot and nosnippet for Googlebot-News only:

html
<meta name="googlebot" content="notranslate">
<meta name="googlebot-news" content="nosnippet">

Sending a crawler-specific X-Robots-Tag header

PDFs, images and other files have no <head>, so the header is the only place for their rules. Google says the header "may optionally specify a user agent before the rules", and that rules without a user agent apply to all crawlers. Its example sends one rule to Googlebot and another to a second crawler:

http
HTTP/1.1 200 OK
X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: noindex, nofollow

Put each crawler's rules on their own X-Robots-Tag line, as Google's example does; Google says multiple X-Robots-Tag headers in one response can be combined. Apple's documented form follows the same pattern:

http
X-Robots-Tag: applebot: nosnippet

On nginx, each add_header line produces a separate header. This asks Bing and Apple not to index PDFs while leaving them indexable in Google:

nginx
location ~* \.pdf$ {
    add_header X-Robots-Tag "Bingbot: noindex";
    add_header X-Robots-Tag "applebot: noindex";
}

Two caveats. Bing's posts show the Bingbot: header prefix only with its AI values, and we could not read Bing's help page to confirm it treats noindex the same way, so check the result in Bing Webmaster Tools. Apple's header example uses nosnippet; it lists noindex as a supported directive and says the header carries indexing directives, but it does not show applebot: noindex itself.

Server-by-server setup for Apache, nginx and Next.js is in X-Robots-Tag for PDFs and other files. If a header appears that you did not write, a CDN or plugin may be adding it.

The common mistake: robots.txt tokens used as meta names

robots.txt and the robots meta tag use different vocabularies. robots.txt addresses crawlers by product token on its User-agent: line, and several AI tokens exist only there. People copy them into meta tags, and the tags do nothing that any operator documents:

html
<!-- None of these names is documented by its operator -->
<meta name="GPTBot" content="noindex">
<meta name="Google-Extended" content="noindex">
<meta name="Applebot-Extended" content="noindex">

Here is why each one fails:

  • Google-Extended is, in Google's words, "a standalone product token" that "doesn't have a separate HTTP request user agent string". It is read in robots.txt. As a meta name, it falls under Google's rule that unsupported values are ignored.
  • Applebot-Extended "does not crawl webpages", according to Apple. It only decides how content crawled by Applebot may be used for training Apple's foundation models, and Apple describes it only as a robots.txt rule. The Applebot-Extended guide covers the robots.txt form.
  • GPTBot and OAI-SearchBot are controlled through robots.txt in OpenAI's crawler documentation. That page does not mention robots meta tags or X-Robots-Tag at all.

If your goal is to refuse AI training, the documented controls are robots.txt groups for those tokens. Bing is the exception among the three search engines here: it now has page-level AI values set with its own crawler name, such as <meta name="Bingbot" content="ai-training:disallow">, which Microsoft says it will honour as it crawls starting 1 February 2027. Those values and their deadlines are covered in Bing's AI training and grounding preferences.

Common one-engine recipes

GoalTag
Keep a page out of Google, leave it to others<meta name="googlebot" content="noindex">
No snippet in Google News results only<meta name="googlebot-news" content="nosnippet">
No description or web answer in Apple features<meta name="applebot" content="nosnippet">
Keep a page out of Spotlight and Siri Suggestions<meta name="applebot" content="noindex">
Bing-specific AI grounding preference<meta name="Bingbot" content="ai-grounding:disallow">

Before you use any of these, weigh what you lose. A noindex for one engine removes the page from that engine's results. For Apple, noindex keeps the page out of Spotlight and Siri Suggestions, and Apple says it will not use nosnippet content as context for its AI-generated answers. For Google, the AI features documentation says a page must be indexed and eligible for a snippet to appear as a supporting link in AI Overviews or AI Mode, so a googlebot rule affects both.

How to check which crawler-specific rules a page sends

Look at the raw response, because a tag added by JavaScript may not be seen by every crawler. These commands list every robots-style meta tag and every X-Robots-Tag header, whatever crawler they name:

bash
curl -s https://example.com/page \
  | grep -Eio '<meta[^>]*name="(robots|googlebot|googlebot-news|bingbot|applebot)"[^>]*>'

curl -sI https://example.com/report.pdf | grep -i '^x-robots-tag'

Then read the results together:

  1. List every tag Google reads. Add up the negative rules from robots and googlebot. That sum is what Googlebot applies.
  2. Look for names nobody reads. A google-extended, gptbot or applebot-extended meta tag signals that someone meant a robots.txt rule. Move the intent to robots.txt.
  3. Confirm the page is crawlable. Google says these settings "can be read and followed only if crawlers are allowed to access the pages". If robots.txt disallows the URL for that crawler, the crawler-specific tag is never seen.
  4. Confirm in the engine's own tool. Search Console's URL Inspection shows whether Google found a noindex. For an unexpected one, see noindex detected in X-Robots-Tag.

Frequently asked questions

Can I use meta name="googlebot" to keep a page indexed in Google when robots says noindex?

No. Google adds up the negative rules from the generic and the googlebot tag, so the noindex from the robots tag still applies. Drop the generic tag and name each crawler you want to restrict instead.

Does Google read meta name="Google-Extended"?

Not according to Google's documentation. Google supports only googlebot and googlebot-news as names in the robots meta tag and ignores other values. Google-Extended is a robots.txt product token.

Does Applebot follow my googlebot meta tag?

Apple does not say so for meta tags. Apple's rule that Applebot follows Googlebot instructions when Applebot is not mentioned sits in its robots.txt section, and its meta tag examples use robots and applebot. Use one of those names for Apple.

Is meta name="bingbot" case-sensitive?

Microsoft's posts use both bingbot and Bingbot, and Bing does not state a case rule in the posts we read. Copy the form in the Bing post you are following.

Can I send different X-Robots-Tag rules to Google and Bing?

Yes. Send one X-Robots-Tag line per crawler, each starting with the crawler name, such as googlebot: or Bingbot: (the form in Bing's own examples). Google says rules without a user agent apply to all crawlers.

Check what your pages tell each crawler

Crawler-specific tags are easy to get wrong because a misspelled or unsupported name fails silently. As a next step, run the free check on an important page, with no account needed. For a whole site, an audit covers up to 500 pages with human review and a PDF of evidence and prioritized fixes.

Filed under Access and robots.txt. Topics: meta-robots, googlebot, bingbot, applebot, robots-txt