robots.txt user-agent groups: why a named group overrides *

A robots.txt file is a set of groups, and each crawler obeys exactly one of them: the group that names it, or the User-agent: * group if nothing names it. The two are never merged. So the moment you add a group for GPTBot, OAI-SearchBot or any other crawler, that crawler stops reading your * rules, and anything you blocked only under * is open to it again.

That is how a "small" robots.txt edit can change far more than intended. Below is how group selection and rule precedence work according to RFC 9309, the Robots Exclusion Protocol standard, with worked examples you can check against your own file.

What a robots.txt group is

A group starts with one or more User-agent lines and is followed by Allow and Disallow rules. It ends when the next User-agent line appears after a rule. RFC 9309 writes it like this in its grammar:

group = startgroupline *(startgroupline / emptyline) *(rule / emptyline)

In plain terms:

• One or more user-agent lines open a group. You can stack several crawlers on top of one set of rules.
• Rules belong to the group above them. A Disallow line has no meaning on its own.
• Blank lines are allowed inside a group under the RFC grammar. The original 1994 robots.txt document described records as separated by blank lines, so an older parser may end a group at a blank line. Keeping a group's lines together without gaps is still the safer habit.
• Other lines, such as Sitemap, are not part of any group. The RFC says such records must not end a group, and Google's robots.txt specification shows a sitemap line between two user-agent lines leaving them in the same group.

The value of a User-agent line is a product token: a short name such as GPTBot, OAI-SearchBot, ClaudeBot or Bingbot, not the full user-agent string you see in logs. RFC 9309 says crawlers must match it case-insensitively, so gptbot and GPTBot are the same group. Paths are the opposite: matching should be case-sensitive, so /Admin/ and /admin/ are different rules. Our AI crawler user agents list gives the token for each crawler.

How a crawler picks its group

RFC 9309, Section 2.2.1, sets three rules:

1. The crawler looks for a group whose user-agent line matches its product token. If it finds one, it obeys that group.
2. If several groups match its token, their rules must be combined into one group.
3. If no group matches, it obeys the * group. If there is no * group either, no rules apply and everything is allowed.

Notice what is missing: there is no step where the named group and the * group are added together. Google's specification states it directly: "User agent specific groups and global groups (*) are not combined." Bing's robots.txt guide says the same about Bingbot: once it finds a group addressed to itself, it ignores the generic section, so you must repeat any general rules inside the Bingbot group.

Here is how that plays out for one file:

User-agent: *
Disallow: /account/
Disallow: /checkout/

User-agent: GPTBot
Disallow: /premium/

User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

GPTBot is the surprise. The owner wanted to keep it out of /premium/ on top of the general rules, but the GPTBot group replaced the general rules instead of adding to them.

Crawlers with their own fallbacks

Some operators document extra behaviour on top of the RFC. Two that matter in practice:

• Applebot falls back to Googlebot. Apple says that if robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. A file that disallows Googlebot and says nothing about Applebot blocks Applebot too.
• Google crawlers pick the most specific matching group. Google's specification says only one group is valid for a particular crawler, chosen by the most specific user agent that matches. Its example has Googlebot-News using its own group while other Google crawlers without a matching group use *.

The accident: adding a named group opens paths you blocked for everyone

The brief version of the problem is this: a named group starts from nothing. Here is a typical file before an edit:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /internal-search
Disallow: /staging/

Someone wants to make sure ChatGPT search can reach the site, reads that OAI-SearchBot should be allowed, and adds:

User-agent: OAI-SearchBot
Allow: /

OAI-SearchBot now obeys only that group. /admin/, /cart/, /internal-search and /staging/ are all allowed for it. The same happens with an even smaller edit:

User-agent: GPTBot

A group with a user-agent line and no rules is valid. RFC 9309's own example includes one (quxbot) and explains that a crawler with an empty group may access everything. If you meant to block GPTBot and the Disallow: / line got lost in a copy and paste, you have done the opposite.

How to fix it

Repeat every general rule inside each named group, then add the crawler-specific rules:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /internal-search
Disallow: /staging/

User-agent: OAI-SearchBot
User-agent: PerplexityBot
Disallow: /admin/
Disallow: /cart/
Disallow: /internal-search
Disallow: /staging/

User-agent: GPTBot
Disallow: /admin/
Disallow: /cart/
Disallow: /internal-search
Disallow: /staging/
Disallow: /premium/

Two habits keep this manageable:

• Ask whether you need a named group at all. OAI-SearchBot already follows * if nothing names it. If * allows what you want it to reach, a named group adds nothing but a second copy to maintain.
• Stack crawlers that share the same policy into one group with several User-agent lines, as above, so there is one set of rules to keep in sync instead of five.

If your goal is to keep search crawlers in while opting out of training, How to allow AI search but block AI training has complete files for that split, and OpenAI's own example in its crawler documentation is exactly that: allow OAI-SearchBot, disallow GPTBot.

Allow vs Disallow: the longest match wins

Once a crawler has its group, it compares the URL path against every rule in that group. RFC 9309, Section 2.2.2, says:

• The most specific match must be used, and the most specific match is the one with the most octets (roughly, the longest matching rule path).
• If an Allow and a Disallow rule are equivalent, the Allow rule should be used.
• If no rule matches, the URL is allowed. /robots.txt itself is always allowed.

Order in the file does not matter. A Disallow: / at the top does not beat a longer Allow further down. The RFC also defines two special characters: * matches zero or more of any character, and $ marks the end of the pattern.

These examples come from Google's specification, which applies the same longest-match logic and resolves ties in favour of the less restrictive rule:

The practical use is the classic "block everything except one section" group:

User-agent: GPTBot
Disallow: /
Allow: /blog/

/blog/post matches both rules; Allow: /blog/ is longer, so it wins. /pricing matches only Disallow: /, so it is blocked. Remember the earlier lesson, though: if your * group blocked /blog/drafts/, this GPTBot group does not, unless you add that line here too.

One more detail: a Disallow: line with no path blocks nothing. RFC 9309's grammar allows a rule with an empty path, Google says its crawlers ignore a rule without a path, and the original 1994 document said an empty Disallow value means all URLs can be retrieved. Disallow: and Disallow: / look similar and mean opposite things.

How to check which group a crawler obeys

Read your live file first, not the one in your repository or CMS. Platforms, plugins and CDNs can rewrite it:

curl -s https://www.example.com/robots.txt

Then, for each crawler you care about:

1. Search the file for its product token, case-insensitively. If it appears, that group (combined with any other group naming the same token) is the only one it reads.
2. If it does not appear, it reads *. For Applebot, check for a Googlebot group before falling back to *.
3. For each important URL, find every rule in that group that matches the path and pick the longest. On a tie, Allow wins.

To test a URL mechanically, Google's open-source parser, google/robotstxt, takes a file, a product token and a URL:

bazel run robots_main -- ~/robots.txt GPTBot https://www.example.com/account/settings

It prints one line per check, in this form:

user-agent 'GPTBot' with URI 'https://www.example.com/account/settings': ALLOWED

Run it once per crawler token for the same URL and compare the answers. Be careful with Python's built-in urllib.robotparser and check which version you run. In Python 3.12 and earlier it returns the first rule that matches in file order rather than the longest one, so it can disagree with real crawlers. The code in the 3.13 and later branches of CPython uses the longest match instead. Our guide to testing robots.txt against each AI crawler covers more tools and a repeatable routine.

Changes are not instant. RFC 9309 says crawlers should not use a cached copy for more than 24 hours, and OpenAI says its search systems can take about 24 hours to adjust after a robots.txt update.

Frequently asked questions

Does a GPTBot group inherit the rules under User-agent: *?

No. Under RFC 9309 a crawler that finds a group naming it obeys only that group. Any rule you want GPTBot to follow has to be written inside the GPTBot group.

What happens if the same crawler is named in two groups?

The groups are combined. RFC 9309 says the rules of all groups matching the crawler's product token must be merged into one group, so a second User-agent: GPTBot block further down the file adds to the first rather than replacing it.

Does rule order matter inside a group?

No. The rule with the longest matching path wins wherever it sits in the group, and an Allow should win over a Disallow of equal length. Order only matters to older parsers that are not RFC 9309 compliant.

Is User-agent matching case-sensitive?

The product token is matched case-insensitively, so GPTBot and gptbot select the same group. RFC 9309 says paths in Allow and Disallow rules should be matched case-sensitively, so /Blog/ does not cover /blog/.

Can I list several crawlers in one group?

Yes. Put several User-agent lines one after another, then the rules. Every crawler named there obeys the same rules, which is the easiest way to keep policies for similar crawlers in sync.

Check what your file actually allows

Group mistakes are easy to miss when you read a robots.txt top to bottom, because the file looks stricter than it is. Work through the steps above for each crawler you care about, and use the robots.txt mistakes guide to look for the other errors that tend to travel with this one. If you want a second opinion on one page, run the free check: it covers a single page and needs no account.