robots.txt: allow AI crawlers in one folder, block the rest
How to let GPTBot, ClaudeBot and other AI crawlers read only /blog/ and /docs/, or block only /members/, with tested robots.txt rules and the group trap.
By Crawlito · 8 min read
In short
- To open one folder and close the rest, put Disallow: / and Allow: /blog/ in the same group: under RFC 9309 the longest matching rule wins, so /blog/ pages are allowed and everything else is not.
- To close one folder only, a single Disallow: /members/ line is enough, because anything with no matching rule is allowed.
- A crawler obeys the group that names it and ignores User-agent: *, so every rule you still want, such as /members/, has to be repeated inside each named group.
- Paths are case-sensitive prefixes: /blog/ does not match /blog, and /Blog/ is a different path from /blog/. Use /$ for the homepage and $ to anchor a file extension.
- robots.txt is a request, not access control: RFC 9309 says it is not a form of access authorization, so private content needs a login.
On this page
- Three goals, three patterns
- Allow one folder and block the rest
- Block one folder and allow the rest
- Repeat the rules inside each named group
- Trailing slashes, wildcards and $ end anchors
- Worked tests for a sample URL list
- Check the live file and see who still reaches blocked paths
- Frequently asked questions
- Check what your site serves
To let AI crawlers read one part of your site and nothing else, put a Disallow: / rule and an Allow: /blog/ rule in the same group for the crawler. The longest matching rule wins, so /blog/ pages are allowed and every other path is blocked. To block only one section, such as /members/, you need just one Disallow: /members/ line. The part that goes wrong is groups: a crawler with its own named group stops reading your User-agent: * rules, so any folder you still want closed has to be repeated.
Everything below is checked against RFC 9309, Google's robots.txt specification and the crawler documentation from OpenAI and Anthropic. The worked tests use a small matcher written to follow the RFC rules, and it reproduces the examples in Google's specification.
Three goals, three patterns
Most partial-access requests fall into one of these.
| Goal | Rules in the crawler's group | Why it works |
|---|---|---|
| Open one folder, block the rest | Disallow: / plus Allow: /blog/ | The longer Allow beats the one-character Disallow for URLs under /blog/ |
| Block one folder, allow the rest | Disallow: /members/ | Unmatched URLs are allowed |
| Both: open a folder but close a subfolder | Disallow: / plus Allow: /docs/ plus Disallow: /docs/internal/ | The longest match decides at each level |
Before you write any of them, decide which crawlers they are for. Training and search are separate tokens. OpenAI says GPTBot (training) and OAI-SearchBot (search) settings work independently. Anthropic lists ClaudeBot for training, Claude-SearchBot for search and Claude-User for user requests. Our AI crawler user agents list has the token for each one, and how to allow AI search but block AI training covers the whole-site version of that split.
Allow one folder and block the rest
This group lets GPTBot and ClaudeBot read /blog/ and /docs/ and nothing else:
User-agent: GPTBot
User-agent: ClaudeBot
Allow: /blog/
Allow: /docs/
Disallow: /Two User-agent lines in a row share the rules below them. For a URL, the crawler collects every rule in its group that matches, picks the one with the most characters, and follows it. RFC 9309 calls this the most specific match, defined as the match with the most octets. For /blog/post-1, Allow: /blog/ (6 characters) beats Disallow: / (1 character). For /pricing, only Disallow: / matches. The RFC decides by length, not by position, so line order does not change the result in a parser that follows it.
If the two rules ever have the same length, the RFC says the Allow rule should be used, and Google says it applies the least restrictive rule. Behavior on ties is therefore not guaranteed for every parser, so avoid writing rules that tie.
Two details catch people here:
- The homepage is blocked.
/matches onlyDisallow: /. If you want the homepage readable, addAllow: /$. Google's specification says/$matches only the root. - robots.txt itself is always readable. RFC 9309 says the
/robots.txtURI is implicitly allowed, soDisallow: /does not stop a crawler fetching your rules.
At the time of writing, neither OpenAI's nor Anthropic's crawler page states how its crawlers resolve an Allow that overlaps a Disallow. OpenAI's page has no robots.txt examples, and Anthropic's examples use Disallow (plus a non-standard Crawl-delay line) with no Allow rule. Anthropic says its crawlers honor industry-standard robots.txt directives. The RFC pattern above is the standard, but confirm the result in your logs (covered below) rather than assuming it.
Block one folder and allow the rest
Closing a section is the simpler case:
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /members/Nothing else matches, so every other path is allowed. If a User-agent: * group already exists, remember this named group replaces it for these two crawlers (next section).
Match the path your crawler requests. Disallow: /members/ blocks /members/dashboard but not /members or /members-area/join. Dropping the trailing slash, as in Disallow: /members, blocks all three and also /membership, because rules match as prefixes. Pick the one you mean.
Repeat the rules inside each named group
This is where partial-access files usually go wrong. A crawler follows the group that names it and ignores User-agent: *. RFC 9309 says a crawler falls back to the * group only if no group matches its product token, and Google's specification states that "User agent specific groups and global groups (*) are not combined." Here is a file that looks safe:
User-agent: *
Disallow: /members/
Disallow: /admin/
User-agent: GPTBot
Allow: /blog/GPTBot reads only its own group. /blog/ matches Allow, and everything else, including /members/dashboard, matches nothing and is allowed. The Allow line alone restricts nothing. The fix is the Disallow: / line from the first pattern, or repeating the closed folders:
User-agent: *
Disallow: /members/
Disallow: /admin/
User-agent: GPTBot
User-agent: ClaudeBot
Allow: /blog/
Allow: /docs/
Disallow: /The reverse also matters: crawlers you do not name keep following *. In this file, OAI-SearchBot and Claude-SearchBot read the * group, so they can reach everything except /members/ and /admin/. If you want search crawlers limited to the same two folders, add them to the stacked group. OpenAI documents that a site opted out of OAI-SearchBot is not shown in ChatGPT search answers (navigational links can still appear), and it does not describe how a partial block is treated, so decide that trade-off on purpose. Why a named group overrides * explains the selection rules in full.
Trailing slashes, wildcards and $ end anchors
Path rules are literal prefixes. RFC 9309 says matching must start with the first octet of the path, and Google adds that path values are case-sensitive.
| Rule | Matches | Does not match |
|---|---|---|
/blog/ | /blog/, /blog/post-1 | /blog, /blog-news/, /Blog/post-1 |
/blog | /blog, /blog/post-1, /blog-news/ | /Blog |
/$ | / only | /index.html, /blog/ |
/blog$ | /blog only | /blog/, /blog/post-1 |
Google's specification gives the same behavior for its /fish/ example: it matches inside the folder but not /fish itself. If your section index is served at /blog without a slash, add Allow: /blog$ next to Allow: /blog/.
The * wildcard matches zero or more characters and $ ends the pattern, both in RFC 9309. They let you carve out file types:
User-agent: GPTBot
Allow: /docs/
Disallow: /docs/*.pdf$
Disallow: //docs/guide.pdf is blocked, since /docs/*.pdf$ is longer than /docs/. But Google's specification notes that /*.php$ does not match /filename.php?parameters, because the query string is part of what is compared. The same logic means /docs/guide.pdf?download=1 is not caught, and /docs/guide.PDF is not either. If those variants exist on your site, a rule without $, such as Disallow: /docs/*.pdf, is broader. Check that it does not catch URLs you want to keep.
Worked tests for a sample URL list
Here is one complete file. The first group is for the two training crawlers, the * group is for everything else, including the search crawlers that fall back to it:
User-agent: *
Disallow: /members/
Disallow: /admin/
User-agent: GPTBot
User-agent: ClaudeBot
Allow: /$
Allow: /blog/
Allow: /docs/
Disallow: /docs/internal/
Disallow: /docs/*.pdf$
Disallow: /These are the results from applying the longest-match rule to each URL, with the winning rule shown:
| URL path | GPTBot or ClaudeBot | OAI-SearchBot (falls back to *) |
|---|---|---|
/ | Allowed (/$) | Allowed |
/blog/ | Allowed (/blog/) | Allowed |
/blog | Blocked (/) | Allowed |
/blog/post-1?utm_source=x | Allowed (/blog/) | Allowed |
/Blog/post-1 | Blocked (/) | Allowed |
/docs/setup | Allowed (/docs/) | Allowed |
/docs/internal/keys | Blocked (/docs/internal/) | Allowed |
/docs/guide.pdf | Blocked (/docs/*.pdf$) | Allowed |
/docs/guide.pdf?download=1 | Allowed (/docs/) | Allowed |
/pricing | Blocked (/) | Allowed |
/members/dashboard | Blocked (/) | Blocked (/members/) |
/admin/login | Blocked (/) | Blocked (/admin/) |
Read the table for surprises. /blog without the slash is blocked for the training group. The PDF with a query string gets through. And the search crawler column shows how much a * group leaves open. Each of those is a one-line edit once you see it.
To repeat this for your own file, list 10 to 15 real URLs: your homepage, one URL for each section, a URL with a query string, a file download, and the folder you most want closed. Run each against each crawler token. The robots.txt testing guide covers parsers you can use.
Check the live file and see who still reaches blocked paths
First fetch what your server returns, not the file in your repository. A CMS, plugin or CDN may rewrite it:
curl -s https://www.example.com/robots.txtRules apply only to the host, protocol and port that serve the file, and Anthropic says each subdomain needs its own robots.txt. Check docs. and blog. hosts separately.
Then look in your access logs for requests to closed paths from the crawlers you restricted. Match on the token, not the full user-agent string:
grep -E "GPTBot|ClaudeBot" access.log | grep " /members/" | awk '{print $1, $7, $9}' | tailThe columns are IP, path and status in the common log format. A hit after the changes have had time to apply means one of the following:
- The crawler fetched the path before it saw the new rules. OpenAI says its search systems can take about 24 hours to adjust after a robots.txt update, and RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable.
- The request came from a user-triggered agent. OpenAI says robots.txt rules may not apply to ChatGPT-User, since a person asked for the page.
- The user agent is fake. Check the IP against the operator's published ranges, as shown in AI crawlers in server logs.
- Your rule does not match the URL as requested. Compare the path in the log with the rule, character by character, including case and query strings.
For content that must stay private, do not rely on any of this. RFC 9309 says the protocol is not a substitute for real content security measures. Put the content behind a login or an access rule at your server or CDN.
Frequently asked questions
How do I allow GPTBot only in /blog/?
Put Allow: /blog/ and Disallow: / in a User-agent: GPTBot group. The longer Allow rule wins for URLs under /blog/, and the one-character Disallow applies to everything else. Check /blog without a trailing slash and the homepage, because both are blocked by that file.
Do I need to repeat my User-agent: * rules in the GPTBot group?
Yes, for any rule you still want GPTBot to follow. A crawler that finds a group naming it ignores the * group, so Disallow: /members/ under * does nothing for GPTBot. The Disallow: / pattern covers this when you are limiting a crawler to specific folders.
Does the order of Allow and Disallow lines matter?
Not in a parser that follows RFC 9309, which picks the most specific match, meaning the one with the most octets, and defines precedence by specificity rather than position. A parser that does not implement the RFC may behave differently, which is a reason to test with one that does.
Will robots.txt keep my private pages out of AI tools?
Not reliably. RFC 9309 says its rules are not a form of access authorization, and OpenAI says robots.txt rules may not apply to ChatGPT-User. Protect private pages with authentication.
Can I allow one folder for search crawlers and a different one for training crawlers?
Yes, with separate groups. Put the training tokens in one group and the search tokens in another, and write the rules for each. Keep the shared rules, such as /members/, in both groups.
Check what your site serves
After editing, run the free check on a page from an open folder and one from a closed folder. It reports the robots.txt rules for each named AI crawler, the robots meta tag and headers, and whether the main text is in the raw HTML, with no account. For a whole site, an audit runs the same checks across up to 500 pages. If the problem is a rule you wrote by accident, seven robots.txt mistakes that block AI crawlers lists the other ones worth looking for.
Filed under Access and robots.txt. Topics: robots-txt, ai-crawlers, gptbot, claudebot