__usage_ai-training in robots.txt: the shared AI training opt-out
The exact __usage_ai-training robots.txt lines, how the draft's matching rules work, who honours the token today, and why GPTBot and ClaudeBot groups still matter.
By Crawlito · 8 min read
In short
- __usage_ai-training is a robots.txt policy token from an individual IETF draft published on 30 October 2026; it states a preference about AI training and is not a crawling directive.
- Under the draft, a User-agent: * group never matches the token, so a training preference only exists if you write a group that names __usage_ai-training.
- Anthropic began recognising the signal on 30 October 2026, Microsoft says it will honour it from 1 February 2027, and Google aims to support it in 2027.
- Google plans to treat a training opt-out through the token as a Google-Extended disallow when Google-Extended is not named, and Anthropic follows a ClaudeBot rule over the general signal.
- OpenAI's crawler documentation does not mention the token at the time of writing, so keep named groups such as GPTBot, ClaudeBot, Google-Extended and CCBot.
On this page
To state a site-wide AI training opt-out with the shared token, add a group that names it: User-Agent: __usage_ai-training followed by Disallow: /. That line expresses a preference against using your content to train generative AI models; it does not stop any crawler from fetching your pages, and it is defined in an individual IETF draft, not a finished standard. At the time of writing Anthropic reads it, Microsoft and Google have announced dates in 2027, and OpenAI has not documented it, so keep your named training groups as well.
This post covers the token itself: the lines, the matching rules in the draft, and where each operator stands. For the full allow-search, block-training file, see how to allow AI search but block AI training. Bing's separate page-level grounding and training values are covered in Bing's AI training and grounding preferences. Sources below were read on 15 November 2026.
What the __usage_ai-training token is
The token comes from draft-kelley-aipref-training-preference-signal, titled "Machine-Readable Training Preference Signal". The facts that matter for planning:
- It is an individual Internet-Draft. Revision 00 is dated 30 October 2026 and expires on 3 May 2027. The datatracker shows no IETF stream, and the draft states that it has no formal standing. Discussion takes place on the AI Preferences (AIPREF) working group's mailing list.
- It defines one policy token.
__usage_ai-trainingfits the product token syntax of RFC 9309, which allows only letters, underscores and hyphens. The double underscore prefix makes an accidental match with a real crawler name unlikely. - It uses the AIPREF meaning of training. AI training is "using an asset to modify the learned parameters of a generative AI model". The preference that applies is the one in the robots.txt in effect when the content was retrieved.
- It adds no new syntax. There are no new fields. The token goes in an ordinary
User-Agentline and uses ordinaryAllowandDisallowrules, with RFC 9309's path matching and longest-match precedence unchanged.
That last point is what separates it from other usage signals. A Content-Signal line, described in Content Signals in robots.txt, is a record RFC 9309 does not define, so a parser that does not know it has to skip it. This token is a normal group, so every RFC 9309 parser can read it, and a crawler that does not know the token simply finds a group that does not name it.
The exact robots.txt lines
A site-wide opt-out from AI training:
User-Agent: __usage_ai-training
Disallow: /An opt-out for one section only. Paths outside it carry no stated preference from this group:
User-Agent: __usage_ai-training
Disallow: /articles/An opt-out with an exception, which is the example the draft itself uses. Training is disallowed everywhere except the press kit, where the Allow line states a preference in favour of training:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Microsoft's announcement adds that allowing training is the default when no preference is published, so you need no line at all to permit it. An explicit Allow: / is optional.
How the matching rules work
The draft takes RFC 9309 and changes how a reader interprets one group. The table summarises the rules in sections 4.1 to 4.5:
| Rule | What the draft says | What it means for your file |
|---|---|---|
| Name matching | Match the token exactly, apart from case | __USAGE_AI-TRAINING works; _usage_ai-training or usage-ai-training does not |
| Wildcard | A User-Agent: * group must not be read as matching the token | Your * group never expresses a training preference |
| Disallow | A disallowed path means a preference against AI training | Disallow: / is the site-wide opt-out |
| Allow | An allowed path means a preference in favour | Use it for exceptions such as a press kit |
| No group | No preference is expressed | A missing group is not an opt-out |
| Crawling | The token is not a directive for crawling | Crawlers still follow the group for their own product token |
| Precedence | Specific expressions should take precedence over general ones; an operator may document that its own token is more specific | A named group such as ClaudeBot can override the shared token for that operator |
Two consequences catch people out.
The token does not block crawling. The draft says consumers keep interpreting the rules for their own product token exactly as before. A consumer may choose not to fetch a path it cannot use for training, but the draft does not require it. Microsoft says the same from the other side: its training and grounding settings govern use of content, not crawl access, and an Allow setting does not override a robots.txt rule that blocks Bingbot from crawling. If you want a crawler to stay away, you still need a group for that crawler.
Do not put the token in the same group as a crawler name or *. The draft allows policy tokens to share a group with product tokens through consecutive User-Agent lines, and RFC 9309 applies that group's rules to every name in it. This file blocks every crawler without its own group, not just training:
# Mistake: the Disallow applies to * as well
User-Agent: *
User-Agent: __usage_ai-training
Disallow: /Keep the token in a group of its own unless you mean every name in the group to get the same rules. How robots.txt groups work explains group selection in more detail.
Who honours the token at the time of writing
| Operator | Status | Documented interaction with named tokens |
|---|---|---|
| Anthropic | Recognises the signal since 30 October 2026, rolling out across its services over time | A rule that names ClaudeBot is followed even if the general setting differs |
| Microsoft Bing | Will honour the preference from 1 February 2027, as it crawls | A page-level training preference overrides robots.txt |
| Aims to support it in 2027; its crawler documentation says early 2027 | Plans to treat a training opt-out through the token as a Google-Extended disallow when Google-Extended is not specified | |
| OpenAI | Not mentioned in its crawler documentation | GPTBot remains the documented training control |
The sources are Anthropic's robots.txt controls article, Microsoft's Bing announcement, Google's Search Central post and common crawlers page, and OpenAI's bots page. We found no statement from other crawler operators, such as Common Crawl or Apple, at the time of writing.
A few details from those pages are worth knowing:
- Anthropic cites the draft by name and says the way it reads and applies these settings may change while the standards evolve. Its article gives no example lines for the token.
- Microsoft says publishing a preference does not take effect immediately, because Bing has to crawl the page to see it. Choices about training do not affect eligibility in Bing search. Its robots.txt examples use the token directly rather than a Bingbot group.
- Google frames this as a shared control it will support alongside other AI providers, and says company-specific controls such as Google-Extended keep working. Its announcement post does not spell out the token string, but its crawler documentation shows
user-agent: __usage_ai-training. One side effect to weigh: Google documents Google-Extended as covering both Gemini training and grounding in Gemini Apps and Vertex AI, and its post says sites that specifically allow Google-Extended are eligible to appear in Gemini Apps responses through grounding. Google does not spell out how the token opt-out affects grounding, so if you want Gemini grounding while refusing training, decide that in an explicit Google-Extended group. The Google-Extended mechanics are in Google-Extended explained.
Do you still need GPTBot, ClaudeBot and Google-Extended groups?
Yes. The shared token adds one line that several operators may read; it does not replace the tokens they already document. Three reasons:
- Coverage. OpenAI has not documented the token, and Google and Microsoft support starts in 2027. Until an operator says it reads the token, its own token is the only signal you know it acts on.
- Precedence. Anthropic follows a named ClaudeBot rule over the general signal, and Google's plan applies the token only when Google-Extended is not named, so an explicit Google-Extended group decides for Google. A named group that allows everything can therefore cancel your shared opt-out for that operator.
- Crawling. If you want a training crawler not to fetch pages at all, only a group naming that crawler says so.
A file that uses both, with named groups agreeing with the shared token:
# Shared training preference (draft)
User-Agent: __usage_ai-training
Disallow: /
# Named training tokens
User-Agent: GPTBot
User-Agent: ClaudeBot
User-Agent: Google-Extended
User-Agent: CCBot
Disallow: /
# Everyone else, including search crawlers without their own group
User-Agent: *
Disallow: /admin/The second point is where conflicts creep in. If you allow ClaudeBot to crawl a section because you want it to, Anthropic treats that named rule as more specific than the shared token. Decide what you want for each operator, and make the named group say it.
How to check your file
General robots.txt testers are built around crawler matching, and RFC 9309 tells a crawler with no named group to fall back to the * group. A tester that applies that fallback to the token gives an answer the draft says is wrong, so check the file by reading it:
- Fetch the live file the way a crawler does:
curl -s https://example.com/robots.txt | grep -n -i -A3 "__usage_ai-training"- Confirm the token is spelled with two leading underscores and a hyphen before
training. - Confirm the group contains no other
User-Agentlines unless you meant them to share its rules. - Confirm each named group (GPTBot, ClaudeBot, Google-Extended, CCBot) agrees with the shared preference, or deliberately differs.
- Confirm no rule you added blocks the search crawlers you want, such as OAI-SearchBot, Claude-SearchBot or Bingbot.
If a plugin, CMS or CDN writes your robots.txt, check the served file rather than the settings screen, since the two can differ.
Frequently asked questions
Does __usage_ai-training block AI crawlers?
No. The draft says the token is not a directive for crawling, so crawlers keep following the group that names them. To stop a crawler fetching pages, write a group for that crawler.
Does a User-agent: * Disallow cover the token?
No. The draft says a * group must not be interpreted as matching the token, so it expresses no training preference. You need a group that names __usage_ai-training.
Will the token affect my visibility in Google or Bing search?
Microsoft says training choices do not affect eligibility for Bing search results. Google plans to treat the opt-out like a Google-Extended disallow, and Google says Google-Extended does not affect inclusion in Google Search or act as a ranking signal there.
Is __usage_ai-training an official standard?
No. At the time of writing it is an individual Internet-Draft with no formal IETF standing, and its details may change before any standard is published.
Does OpenAI honour the token?
OpenAI's crawler documentation does not mention it at the time of writing. Disallowing GPTBot is OpenAI's documented way to keep content out of training.
Check what your robots.txt tells each crawler
The shared token is one more group to get right, next to the named ones. Run the free check on an important page to see whether the main AI crawlers can fetch it under your current rules, and to confirm a training opt-out has not blocked the search crawlers you want to keep.
Filed under Access and robots.txt. Topics: robots-txt, ai-training, google-extended, bingbot, ai-crawlers