Skip to content

TDMRep: how the TDM Reservation Protocol works and who reads it

TDMRep lets you reserve text and data mining rights with tdmrep.json, an HTTP header or a meta tag. Here is how to set it up and what is known about who honors it.

By Crawlito · 6 min read

In short

  • TDMRep is a W3C Community Group final report from May 2024, not a W3C standard, that expresses a text and data mining reservation with two properties: tdm-reservation and tdm-policy.
  • You can publish it as /.well-known/tdmrep.json, as tdm-reservation and tdm-policy HTTP response headers, or as HTML meta tags; the specification says HTTP headers are currently the preferred technique.
  • The specification describes itself as a technical answer to Article 4 of EU Directive 2019/790, where a rights reservation must be expressed by machine-readable means for online content.
  • TDMRep does not block crawling; at the time of writing, OpenAI, Anthropic and Perplexity do not mention it in their crawler documentation, so pair it with robots.txt rules for the crawlers you want to stop.
On this page

The TDM Reservation Protocol (TDMRep) is a small, machine-readable way to say that you reserve your text and data mining rights, and optionally where to find licensing terms. You publish it in one of three places: a /.well-known/tdmrep.json file, two HTTP response headers, or two HTML meta tags. It was written for the EU's copyright rules on text and data mining, and it records a legal position rather than blocking any crawler.

The short practical answer: TDMRep is cheap to add and harmless to search visibility, but no major AI crawler operator documents reading it at the time of writing. If you want a specific crawler to stop fetching your pages, robots.txt and your firewall are still the controls that operators document.

What is TDMRep?

TDMRep is published by the W3C Text and Data Mining Reservation Protocol Community Group. The current version is a Final Community Group Report dated 10 May 2024. The document says plainly that it is not a W3C Standard and is not on the W3C Standards Track. That is normal for community group work, but it means no browser, search engine or crawler is obliged to implement it.

The protocol defines two properties:

PropertyValueMeaning
tdm-reservation1TDM rights are reserved
tdm-reservation0TDM rights are not reserved
tdm-policyA URLWhere a TDM policy describes how to get permission

Any other value for tdm-reservation is a protocol error, and the specification tells agents to treat the property as unset. A policy URL served as text/html is human-readable; served as application/json or application/ld+json, it is machine-readable.

Why it exists: Article 4 of the EU copyright directive

Article 4 of EU Directive 2019/790 (the Digital Single Market or DSM directive) creates an exception that allows text and data mining, including for commercial purposes, unless the rightsholder has expressly reserved that use "in an appropriate manner". For content made publicly available online, the directive gives machine-readable means as the example of an appropriate manner. The TDMRep specification says it was designed as a technical answer to that requirement.

Three limits are worth knowing before you rely on it:

  • It is about the EU exception. The specification notes that outside the EU, the Article 4 exception does not apply, so the absence of a reservation should not be read as permission there.
  • Research is treated separately. Article 3 of the same directive covers mining by research organisations and cultural heritage institutions, and the specification says those beneficiaries need not follow the contact requirements in a TDM policy.
  • It is a declaration, not legal advice. Whether a given reservation is effective is a legal question. Ask counsel if your licensing depends on it.

There is also a policy backdrop. The European Commission's voluntary General-Purpose AI Code of Practice, published in July 2025, includes a copyright chapter in which signatories commit to respecting robots.txt and other appropriate machine-readable rights reservations. As far as we could find, it does not name TDMRep specifically.

How to publish a TDM reservation

The specification asks you to use one technique per server. Pick the one that fits your stack.

Option 1: the well-known file

Create /.well-known/tdmrep.json at the root of the host. It is an array of rules, each with a location path pattern, a tdm-reservation value and an optional tdm-policy:

json
[
  {
    "location": "/blog/*",
    "tdm-reservation": 1,
    "tdm-policy": "https://www.example.com/legal/tdm-policy.json"
  },
  {
    "location": "/press/*",
    "tdm-reservation": 0
  },
  {
    "location": "/*",
    "tdm-reservation": 1
  }
]

Matching is case-sensitive, patterns support * and $, and the specification says the most specific matching rule applies. Serve the file as JSON with a 200 status. The file is read before mining starts and may be cached, so it suits site-wide rules that rarely change.

Option 2: HTTP response headers

Add two headers to responses for GET and HEAD requests. The specification describes this as currently the preferred technique, because it travels with every resource, including images and PDFs:

http
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
tdm-reservation: 1
tdm-policy: https://www.example.com/legal/tdm-policy.json

In nginx, for example:

nginx
location / {
    add_header tdm-reservation "1" always;
    add_header tdm-policy "https://www.example.com/legal/tdm-policy.json" always;
}

nginx does not inherit add_header into a block that sets its own headers, so check every location that serves content.

Option 3: HTML meta tags

For pages where you control the template but not the server, put the values in the <head>:

html
<meta name="tdm-reservation" content="1">
<meta name="tdm-policy" content="https://www.example.com/legal/tdm-policy.json">

This covers HTML only. Images, PDFs and feeds need headers or the well-known file. EPUB and PDF files can also carry the properties in their own metadata, which the specification describes for publishers.

What happens if you use more than one?

If several techniques are present, the specification sets an order: the well-known file is read first, HTTP headers override it, HTML meta tags override headers, and EPUB or PDF metadata overrides everything before it. An absent property never resets a value set earlier. Conflicting signals are easy to create by accident, which is why the specification recommends picking one.

Who honors TDMRep?

This is where most summaries overstate things. At the time of writing:

  • OpenAI, Anthropic and Perplexity describe their crawlers and controls in their own documentation (OpenAI, Anthropic, Perplexity). The controls they document are robots.txt user-agent rules. None of those pages mentions TDMRep.
  • Google documents Google-Extended as its robots.txt token for Gemini training and grounding. Its crawler documentation does not mention TDMRep.
  • Publishers and some platforms have adopted TDMRep, according to a W3C TPAC presentation from November 2025 by the community group, which does not name AI crawler operators that read it.

So the honest position is that support by AI crawlers is not documented. A crawler that does read TDMRep would still fetch your page first: the reservation tells it what it may do with the content, not whether it may request it. That is the opposite of robots.txt, where a compliant crawler never fetches a disallowed URL.

How TDMRep compares with other AI opt-out signals

SignalWhere it livesStops fetching?Documented by major AI crawler operators
robots.txt user-agent rules/robots.txtYes, for crawlers that complyYes: OpenAI, Anthropic, Google, Perplexity, Apple and others
TDMReptdmrep.json, headers or metaNoNot documented at the time of writing
Content signalsComment-style lines in robots.txtNoSee our content signals post
noai and noimageaiRobots meta tag valuesNoNot in the major operators' docs; see noai meta tags

The practical approach for most sites is layered. Use robots.txt for the crawlers you want to stop, following our robots.txt guide for AI crawlers. Add TDMRep if you want a recorded, machine-readable reservation under EU rules, or if your industry body recommends it. Use firewall rules for anything you must actually enforce.

None of these affects Google Search or Bing indexing. TDMRep has no effect on a search engine that does not read it, and a reservation of mining rights is not a noindex.

How to check your TDMRep setup

Check each technique from outside your network:

bash
# The well-known file: expect 200 and a JSON array
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' \
  https://www.example.com/.well-known/tdmrep.json
curl -s https://www.example.com/.well-known/tdmrep.json | python3 -m json.tool

# The headers: expect tdm-reservation and tdm-policy
curl -sI https://www.example.com/blog/some-post | grep -i '^tdm-'

# The meta tags in the raw HTML
curl -s https://www.example.com/blog/some-post | grep -io '<meta[^>]*tdm-[^>]*>'

Things to look for:

  1. A 404 or HTML error page at the well-known path. Some frameworks route unknown paths to the app and return a 200 HTML page, which is not valid JSON.
  2. Missing headers on some responses. Check a page, an image and a PDF. CDN rules often apply headers to HTML only.
  3. Conflicting values. A tdm-reservation: 0 header and a meta tag set to 1 on the same page means the meta tag wins under the specification's precedence order, which may not be what you intended.
  4. A policy URL that fails. The specification says an unreachable policy is not a protocol error, but it leaves agents unable to learn your terms.

Frequently asked questions

Is TDMRep a W3C standard?

No. It is a Final Community Group Report, published in May 2024. The document itself states that it is not a W3C Standard and is not on the Standards Track.

Does TDMRep block AI crawlers?

No. It declares that text and data mining rights are reserved. It does not stop any client from fetching your pages, and at the time of writing the major AI crawler operators do not document reading it. Use robots.txt or firewall rules to stop crawling.

Will adding TDMRep hurt my Google rankings?

There is no documented effect. Google's crawler documentation does not mention TDMRep, and Google Search indexing is controlled by robots.txt and robots meta rules such as noindex.

Should I use the well-known file or HTTP headers?

The specification calls HTTP headers the currently preferred technique and asks you to use one technique per server. The well-known file is easier when you cannot change headers, and it lets you set rules by path in one place.

Check what your site tells crawlers

TDMRep is one line in a wider policy. The free check shows what your robots.txt and server responses tell AI crawlers today, so you can see whether your opt-outs and your search access say what you intend.

Filed under Access and robots.txt. Topics: ai-training, robots-txt, ai-crawlers, content-signals