Content Signals in robots.txt: search, ai-input and ai-train
A Content-Signal line in robots.txt states how your content may be used. See what search, ai-input and ai-train mean, who reads them, and how to edit or remove them.
By Crawlito · 9 min read
In short
- A Content-Signal line sets search, ai-input and ai-train to yes or no; per Cloudflare's policy text, a signal you leave out means you neither grant nor restrict that use.
- Content signals do not allow or block crawling: Allow and Disallow still control access, and Cloudflare says signals are preferences, not technical countermeasures against scraping.
- At the time of writing, the OpenAI, Anthropic and Google crawler documentation we checked does not mention Content-Signal, and Google's robots.txt documentation lists only four supported fields.
- Cloudflare's managed robots.txt adds search=yes, ai-train=no and an experimental use=reference to the wildcard group, and it does not set ai-input.
- A crawler obeys the robots.txt group that names it, and we found no source saying a signal in the * group reaches a crawler that has its own group, so repeat the line where you need it.
On this page
- What is the Content-Signal line and where does it come from?
- What do search, ai-input and ai-train mean?
- What do the example lines state?
- How do content signals relate to Allow and Disallow?
- Where does Cloudflare add the line, and how do you change it?
- Does any crawler act on content signals?
- How do you check the live file?
- Frequently asked questions
- Check what your robots.txt says
A Content-Signal line in robots.txt states how you want your content used after a crawler has fetched it: for search, for feeding AI answers (ai-input) and for AI training (ai-train). Each is set to yes or no, and a signal you leave out says nothing about that use. The line does not allow or block crawling, and at the time of writing the crawler documentation we checked from OpenAI, Anthropic and Google does not say that any of them act on it. Treat it as a published preference, not a control.
What is the Content-Signal line and where does it come from?
Cloudflare introduced the Content Signals Policy in a September 2025 blog post and released the policy text under a CC0 license, so anyone can copy it. It has two parts. A block of # comments explains how to read the signals and defines each one. Then an optional machine-readable line states your actual choices:
User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /The line sits inside a robots.txt group, here the wildcard group. The values are comma-separated name=yes or name=no pairs. The site contentsignals.org generates the text for you, and it is the project's reference page.
RFC 9309, the robots.txt standard, allows this kind of extension. Section 2.2.4 says crawlers may interpret records that are not part of the protocol, and that parsing them must not interfere with the defined records. So an unfamiliar line should not break the rules around it. It also means no crawler is required to do anything with it.
What do search, ai-input and ai-train mean?
These are the definitions in Cloudflare's policy text, in plain words:
| Signal | What it covers | Notes |
|---|---|---|
search | Building a search index and returning results such as links and short excerpts | The definition says search does not include AI-generated search summaries |
ai-input | Putting content into AI models, for example retrieval-augmented generation, grounding or real-time retrieval for generative AI answers | This is the signal for AI answers, not for indexing |
ai-train | Training or fine-tuning AI models | Closest to what GPTBot and ClaudeBot are documented to do |
The values read like this, per the policy text: yes means a crawler may collect content for that use, no means it may not, and a missing signal means you neither grant nor restrict that use. Cloudflare adds that leaving ai-input out does not mean you have no preference. It means you have not used this part of robots.txt to express one.
The definitions describe uses, not crawlers. None of them names a bot, so which crawler counts as doing which use is a judgment each operator makes, and we found no operator documentation that maps its crawlers to these signals.
What do the example lines state?
| Line | What it states |
|---|---|
Content-Signal: search=yes, ai-train=no | Search use is fine, training is not, and no preference on ai-input. This is the example in Cloudflare's post. |
Content-Signal: search=yes, ai-input=no, ai-train=no | Indexing and links are fine, but do not use the content for AI answers or training. |
Content-Signal: ai-train=no | No training. Nothing stated about search or AI input. |
Content-Signal: search=yes, ai-input=yes, ai-train=yes | All three uses are fine. |
You can scope a signal to one crawler by putting it in that crawler's group. This example is adapted from contentsignals.org's own, which repeats the same line in a group for each of three search crawlers:
User-agent: Googlebot
Content-Signal: ai-train=no, search=yes, ai-input=no
Allow: /
User-agent: Bingbot
Content-Signal: ai-train=no, search=yes, ai-input=no
Allow: /
User-agent: OAI-SearchBot
Content-Signal: ai-train=no, search=yes, ai-input=no
Allow: /Writing that file does not make any of those crawlers behave differently by itself. It records what you want.
How do content signals relate to Allow and Disallow?
They answer different questions. Allow and Disallow decide whether a crawler may fetch a URL. Content signals say what you want done with content that has already been fetched. Cloudflare's post describes robots.txt as covering access and the signals as covering use after access, and its example pairs the signal line with Allow: /.
Three consequences follow:
- A signal never opens a door.
ai-train=yesdoes not invite a crawler you have disallowed. IfGPTBothasDisallow: /, a compliantGPTBotfetches nothing, so there is nothing to train on. - A signal never closes one.
ai-train=nodoes not stop a crawler from downloading your pages. If you need to keep a named training crawler out, add aDisallowfor its token. Allow AI search, block AI training shows the file. - Groups still decide which lines a crawler reads. Under RFC 9309 Section 2.2.1, a crawler uses the group that matches its user agent, and falls back to
*only when none does. Our guide to robots.txt groups covers this. We found no source saying aContent-Signalline in the*group is inherited by a crawler that has its own group. If you want a signal to apply to a named crawler, repeat it in that group, as contentsignals.org's examples do.
Cloudflare's managed file shows why this matters. Its * group holds the signal line, while the named groups for GPTBot, ClaudeBot and others contain only Disallow: /.
Where does Cloudflare add the line, and how do you change it?
There are three situations, according to Cloudflare's managed robots.txt documentation and its launch post.
Managed robots.txt is on. When Bot Preference Sync is enabled and at least one of your Search, Agent or Training policies is set to Block, Block on pages with ads or (for Training) Disallow, Cloudflare puts its managed content in front of your own robots.txt. At the time of writing, the * group in that block looks like this:
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /Below it come Disallow: / groups for named crawlers including GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended. The block does not set ai-input; the launch post says Cloudflare does not know your preference and does not want to guess. The use=reference value comes from content-use, an optional extension Cloudflare says it is testing. If your origin already serves a robots.txt, your rules stay below the managed block. If it does not, Cloudflare serves a file with the managed rules. Our Cloudflare crawler settings guide walks through the dashboard.
A free zone with no robots.txt. Cloudflare says it serves only the commented policy text: no Allow, no Disallow and no actual signals. That is a definition of the signals, not a statement of your preferences. The launch post says free zones can turn it off in Security Settings.
You are not on Cloudflare. Write the line yourself, or copy it from contentsignals.org. Some platforms can also send it as an HTTP header; Webflow's Indexing settings offer a Content-Signal header, and our Webflow post covers it.
How to remove or edit the managed line
Cloudflare's documentation does not describe editing the managed block directly. The documented route is to change the policies: with every policy set to allow, or Bot Preference Sync turned off, nothing is prepended and your file is served unchanged.
That removes the whole block, including the Disallow groups for the training crawlers. If you want those to stay, add your own groups to your origin robots.txt first. If you add your own User-agent: * group with a Content-Signal line beneath Cloudflare's, RFC 9309 says groups that match the same crawler are combined, and we found no documentation of how a parser treats two different signal lines.
Does any crawler act on content signals?
Not that the operators document, at the time of writing:
- OpenAI. Its crawler documentation describes separate robots.txt settings for
OAI-SearchBotandGPTBot, says robots.txt rules may not apply to the user-triggeredChatGPT-User, and does not mention Content-Signal. - Google. Its robots.txt documentation lists four supported fields:
user-agent,allow,disallowandsitemap, and says other fields are not supported. Its AI features page does not mention content signals. Google's John Mueller said in a Reddit reply, reported by Search Engine Roundtable on 6 July 2026, that as far as he knows no crawlers or LLMs use the directive. That is one person's statement, not a documentation page. - Anthropic. Its help article on crawling does not name Content-Signal. It describes a separate machine-readable "no AI training" signal tied to an IETF draft, and says a robots.txt rule naming ClaudeBot takes priority over it.
- Cloudflare. Its own post says signaling only works if others recognize the rules, and that some companies might simply ignore them.
The IETF's AI Preferences working group is drafting a separate mechanism. Its attachment draft, draft-ietf-aipref-attach, defines a robots.txt rule named Content-Usage and was still an Internet-Draft in its August 2026 revision, with an expiry date of 20 February 2027. That is a different line from Content-Signal, so do not assume one stands in for the other.
Cloudflare's policy comments also call restrictions "express reservations of rights" under Article 4 of the EU copyright directive 2019/790. Whether that carries weight for your site is a legal question this post does not answer.
If you want an enforceable outcome, use controls that act on requests: a Disallow for the named training token, and a firewall or bot rule for crawlers that do not follow robots.txt. Do AI crawlers respect robots.txt? covers what is documented.
How do you check the live file?
Fetch what the world sees, not the file in your repository, because Cloudflare adds its block at the edge:
curl -s https://example.com/robots.txt | grep -i -B1 -A3 "content-signal"Then check:
- The line is inside a group. It should sit under a
User-agentline, as in the examples above. - Named groups have what you intended. Look for each crawler you care about and see whether its group contains the signal.
- There is one managed block. On Cloudflare, look for the
BEGIN Cloudflare Managed contentandEND Cloudflare Managed Contentcomments, with your own rules after them. - The status code is 200. A 5xx makes crawlers assume everything is disallowed, per RFC 9309; a 404 means no rules apply.
Capitalization varies: Cloudflare's post writes Content-Signal while its managed file shows Content-signal. We found no documentation saying it matters, so copy the form from the source you follow.
Google Search Console may report "Syntax not understood" for the line. Cloudflare says it has seen no impact on crawling or SEO from that report. That matches Google's documentation that it uses only the fields it supports.
Frequently asked questions
Does a Content-Signal line block AI crawlers?
No. It records a preference about use. Blocking needs a Disallow for the crawler's token, or a firewall rule for crawlers that ignore robots.txt.
Should I add ai-input=no?
Only if you do not want your content used for AI answers. Cloudflare leaves ai-input out of its managed block on purpose. Since no operator we checked documents acting on it, adding it records your preference but changes no documented behavior.
Can I keep or remove the line without affecting search?
Google's documentation lists only four supported robots.txt fields. Cloudflare says Search Console's "Syntax not understood" report has had no observed effect on crawling or SEO. Removing the line does not change your Allow and Disallow rules.
Is Content-Signal the same as the IETF standard?
No. The IETF's work uses a rule called Content-Usage and was still a draft at the time of writing. Content-Signal comes from Cloudflare's policy and is not part of RFC 9309.
How is this different from blocking GPTBot?
Disallow for GPTBot is the control OpenAI documents for keeping content out of training. A signal states the same wish in a vocabulary that no operator we checked documents reading.
Check what your robots.txt says
The free check reads one page the way AI crawlers do, with no account needed, including the robots.txt rules for each named AI crawler. For a whole site, an audit runs the checks across up to 500 pages with evidence and prioritized fixes.
Filed under Access and robots.txt. Topics: robots-txt, cloudflare, ai-crawlers, content-signals