Skip to content

EU AI Act code of practice: what it says about robots.txt

What the EU AI Act and its GPAI code of practice say about robots.txt: who commits to follow RFC 9309, what is voluntary, and what a site owner can publish.

By Crawlito · 8 min read

In short

  • Article 53(1)(c) of the EU AI Act requires providers of general-purpose AI models to have a copyright policy that identifies and complies with rights reservations made under Article 4(3) of Directive 2019/790.
  • In Measure 1.3 of the code's copyright chapter, signatories commit to crawlers that read and follow robots.txt as specified in RFC 9309, plus other appropriate machine-readable protocols.
  • The code is voluntary: the Commission's page lists signatories such as OpenAI, Anthropic, Google, Microsoft, Amazon and Mistral AI, and xAI signed only the safety and security chapter.
  • Measure 1.3 is framed around crawling to collect data for text and data mining and model training; the code sets no separate robots.txt rule for search indexing or user-triggered fetches.
  • Signatory search engines are only encouraged, not required, to keep a training opt-out from hurting the indexing of your pages.
On this page

Yes, robots.txt is named in the EU rules, but not as a general law that every crawler must obey. The AI Act requires providers of general-purpose AI models to respect text and data mining opt-outs, and the voluntary General-Purpose AI Code of Practice turns that into a concrete promise: signatories commit to use crawlers that read and follow robots.txt as defined in RFC 9309 when they collect training data. For a site owner, that makes a named Disallow for a training crawler the one opt-out format the code spells out.

This post reads the primary texts in order: the copyright directive, the AI Act and the code. It ends with what you can publish. It is not legal advice, and whether a particular reservation is effective is a question for courts and counsel.

Three documents sit on top of each other.

TextWhat it saysWho it binds
Directive (EU) 2019/790, Article 4Text and data mining of lawfully accessible content is allowed unless the rightsholder has expressly reserved it "in an appropriate manner, such as machine-readable means" for content published onlineMember States, through national law
Regulation (EU) 2024/1689 (AI Act), Article 53(1)(c)Providers of general-purpose AI models must have a policy to comply with EU copyright law and to identify and comply with Article 4(3) reservations, "including through state-of-the-art technologies"Providers placing such models on the EU market
GPAI Code of Practice, copyright chapter, Measure 1.3Signatories commit to crawlers that follow robots.txt (RFC 9309) and other appropriate machine-readable protocolsOnly the providers that signed it

The directive creates the opt-out but does not say which format counts. The AI Act makes respecting it part of a model provider's obligations. The code is where robots.txt and RFC 9309 appear by name.

Two details from the AI Act matter for sites outside Europe. Recital 106 says any provider placing a general-purpose AI model on the EU market should comply with the copyright policy obligation regardless of the jurisdiction in which the copyright-relevant acts underpinning the training take place, so where the crawling happens is not meant to matter. And according to the Commission's guidelines page for GPAI providers, the obligations applied from 2 August 2025, the Commission's enforcement powers apply from 2 August 2026, and models placed on the market before 2 August 2025 have until 2 August 2027 to comply.

What Measure 1.3 of the code commits signatories to

The copyright chapter has five measures. Measure 1.3, "Identify and comply with rights reservations when crawling the World Wide Web", is the one about robots.txt. When signatories crawl, or have others crawl for them, to collect data for text and data mining and model training, they commit:

  1. To follow robots.txt. They will use crawlers that read and follow instructions written under the Robot Exclusion Protocol as specified in RFC 9309, and any later version the IETF shows to be technically feasible for AI providers and content providers.
  2. To honour other machine-readable protocols. They will identify and comply with other appropriate protocols for expressing an Article 4(3) reservation, "for example through asset-based or location-based metadata", if those protocols are adopted by standards bodies or are state of the art, widely adopted by rightsholders and agreed through an EU-facilitated process.
  3. To publish their crawler details. They will make public information about the crawlers they use, their robots.txt features and other opt-out measures, with a way for rightsholders to be notified automatically of changes, such as a web feed.

The same measure says that the commitment does not limit your right to reserve your content "in any appropriate manner", and that it does not change how copyright law applies to content a signatory obtains from third-party crawls.

Measure 1.2, just before it, adds two crawling commitments: not to circumvent effective technological measures, naming paywalls and subscription models, and to exclude sites that EU or EEA courts or authorities have recognised as persistently infringing copyright on a commercial scale. Measure 1.5 commits signatories to a point of contact and an electronic complaints mechanism for affected rightsholders.

What the code does not say

Most summaries stop at "AI companies must respect robots.txt". The text is narrower than that.

  • It is framed around training and text and data mining crawls. Measure 1.3 applies to crawling "for the purpose of text and data mining" and "the training of their general-purpose AI models". It sets no separate rule for crawling for search indexing or for a fetch made because a user asked an assistant to read a page; the only mention of search is the soft indexing clause below. Operators' own rules for those still apply, and some user-triggered fetchers are documented as not following robots.txt, as covered in do AI crawlers respect robots.txt.
  • It does not name any user agent. The code does not list GPTBot, ClaudeBot or any other token. Your file has to name the crawlers that operators document.
  • It does not create a separate AI token. RFC 9309 has no field for purpose of use. The code points to "any subsequent version" of the protocol. The IETF AI Preferences working group is chartered to define a vocabulary and ways to attach it, including through robots.txt, and an individual Internet-Draft defines the __usage_ai-training token described in the __usage_ai-training guide. Google's crawler documentation says it plans to support that token in early 2027. At the time of writing, none of it is part of RFC 9309.
  • It does not settle what counts legally. The chapter's own recitals say that following the code "does not constitute compliance with Union law on copyright", and that interpreting that law is for national courts and the Court of Justice of the EU.
  • It protects search only softly. Signatories that also run a search engine are "encouraged", not required, to make sure that honouring a training reservation does not directly hurt how the reserved content is indexed in their search engine.

Who signed, and why it is voluntary

The code was published on 10 July 2025, and the Commission describes it as a voluntary tool prepared by independent experts. Article 53(4) of the AI Act lets providers rely on a code of practice to demonstrate compliance until a harmonised standard is published. Providers that do not sign must show other adequate means of compliance.

At the time of writing, the Commission's signatory list includes OpenAI, Anthropic, Google, Microsoft, Amazon, IBM, Mistral AI, Cohere and Aleph Alpha, among others, and notes that xAI signed only the safety and security chapter. Meta does not appear on the list. The list may change, so check the Commission's code page before relying on it.

SituationWhat the code means for you
Training crawler run by a signatoryCommitted to follow your robots.txt for training and TDM crawls
Crawler run by a non-signatory model providerStill bound by Article 53(1)(c) if it places a GPAI model on the EU market, but not by the code's robots.txt wording
Third-party dataset, such as a public web crawlThe code says copyright law still applies to such data; the dataset's own crawler decides what it collects
Search or user-triggered fetcherOutside Measure 1.3; the operator's own documentation applies

What a site owner can publish

The practical answer is to state your reservation in the formats the texts point to, without cutting off the crawlers that bring you search visibility.

1. Name the training crawlers in robots.txt

This is the format the code names. Use the tokens operators document for training, and leave search crawlers allowed:

# Reserve text and data mining for model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Search and other crawlers keep access
User-agent: *
Allow: /

Google-Extended and Applebot-Extended are opt-out tokens rather than separate crawlers, so they do not change how Googlebot or Applebot index you; see Google-Extended explained. For the full split between training and search crawlers, follow allow AI search, block AI training.

2. Add a reservation that travels with the content

robots.txt is location-based. The code also mentions asset-based metadata, which stays attached when a file is copied elsewhere. TDMRep was written for Article 4 and can be sent as a response header:

http
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
tdm-reservation: 1
tdm-policy: https://www.example.com/legal/tdm-policy.json

Whether TDMRep meets the code's test of a widely adopted, agreed protocol is not settled in the code, which names no protocol other than robots.txt.

3. Say it in words, and offer terms if you license

Keep a plain statement in your terms of use that you reserve text and data mining under Article 4(3), and give a contact for licensing. If you want to publish machine-readable licence terms, RSL is one option. None of these replaces a named robots.txt group.

How to check that your reservation is visible

A reservation only works if the crawler can read it. Check the file the way a crawler sees it:

bash
# Expect HTTP 200 and text/plain; a 4xx means crawlers may treat everything as allowed
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://www.example.com/robots.txt

# Confirm each training token has its own group
curl -s https://www.example.com/robots.txt | grep -iE -A1 '^user-agent: *(gptbot|claudebot|google-extended|applebot-extended|ccbot)'

# Confirm a reservation header on a content page, if you use TDMRep
curl -sI https://www.example.com/articles/sample | grep -i '^tdm-'

Then confirm you have not caught search crawlers by mistake: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot should still be allowed if you want those search services to be able to fetch your pages. Keep a dated copy of each robots.txt version: the complaints mechanism in Measure 1.5 asks for "sufficiently precise and adequately substantiated" complaints, and a record of what your file said on a given date is the kind of evidence that supports one.

Frequently asked questions

Does the EU AI Act force AI crawlers to obey robots.txt?

Not in those words. The AI Act requires general-purpose AI model providers to respect Article 4(3) reservations; the voluntary code is where signatories commit to follow robots.txt under RFC 9309 for training and text and data mining crawls.

Does the code cover ChatGPT search or Perplexity answers?

Not directly. Measure 1.3 is framed around crawling to collect data for text and data mining and model training, and Perplexity is not on the signatory list at the time of writing. Search crawlers and user-triggered fetchers follow the rules in their operators' own documentation.

Will blocking training crawlers hurt my visibility in Google?

Google's crawler documentation says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. The code only encourages signatories with search engines to avoid indexing harm from a training reservation, so keep search crawlers explicitly allowed.

Is a robots.txt opt-out legally enough under Article 4(3)?

The code calls robots.txt one machine-readable format signatories will follow, but it also says that following the code is not the same as complying with copyright law. Courts decide what counts as an appropriate reservation, so ask counsel if your licensing depends on it.

Does the code apply to crawlers outside the EU?

Recital 106 of the AI Act says providers placing models on the EU market should meet the copyright policy obligation regardless of the jurisdiction in which the copyright-relevant acts underpinning training take place. The code itself binds only the providers that signed it.

Check what your robots.txt tells AI crawlers

Before you rely on a reservation, make sure it says what you mean. The free check shows which AI crawlers your robots.txt allows or blocks on a page, with no account needed, so you can confirm training crawlers are reserved and search crawlers are not.

Filed under Strategy and audits. Topics: robots-txt, ai-training, ai-crawlers, strategy