RSL standard: what Really Simple Licensing means for your site
The RSL standard lets you publish machine-readable licensing terms for AI use. See what an RSL file declares, where it goes, and whether any crawler reads it.
By Crawlito · 8 min read
In short
- RSL 1.0 was published on 10 December 2025 as an industry specification with Recommendation status, and it defines an XML format served as application/rsl+xml.
- An RSL file lists permitted and prohibited uses such as ai-train, ai-input, ai-index and search, plus optional payment, reporting and legal terms.
- You point crawlers to the file with a License line in robots.txt, an HTTP Link header, an HTML link element, an RSS item or embedded file metadata.
- The specification says the License directive does not change Allow or Disallow, so RSL declares terms and blocks nothing by itself.
- At the time of writing, the crawler documentation from OpenAI, Anthropic and Google that we checked does not mention RSL or a License directive.
On this page
The RSL standard (Really Simple Licensing) is an XML format for publishing the terms under which machines may use your content: which uses are allowed, which are prohibited, and what, if anything, they cost. You host one license file and point to it from robots.txt, an HTTP header or your HTML. RSL describes terms and blocks nothing on its own, and at the time of writing none of the major AI crawler operators we checked documents reading it.
This post explains what RSL 1.0 contains, where the file goes, who backs it and what it changes today. If you have already read our posts on Content Signals and TDMRep, this one covers what RSL adds on top of them: a full license vocabulary and a path to payment.
What is the RSL standard?
The RSL 1.0 specification was published on 10 December 2025 with the status "Recommendation". Its previous version was 0.9. The document calls itself an industry specification: it is not an IETF RFC or a W3C Recommendation. The RSL Technical Steering Committee announced it, and its press release says the nonprofit RSL Collective co-developed it. The specification's editors come from the RSL Collective, Condé Nast, Ziff Davis, Schema.org, Yahoo, Automattic, O'Reilly Media and Fastly.
The specification has a required core and three optional protocols:
| Part | What it does | Required? |
|---|---|---|
| Core vocabulary and discovery | The XML elements for terms, and the ways to point clients to them | Yes |
| Open License Protocol (OLP) | An OAuth 2.0 extension through which a client obtains a license token from a license server | No |
| Crawler Authorization Protocol (CAP) | An HTTP Authorization: License <token> scheme a server can check before serving content | No |
| Encrypted Media Standard (EMS) | Encrypted content whose key is released to licensed clients | No |
The specification says you may adopt RSL as a licensing and discovery framework without implementing any of the network protocols. For most sites, that is the realistic starting point: one XML file and a pointer to it.
What does an RSL file declare?
The root element is <rsl> with the namespace https://rslstandard.org/rsl. Inside it, each <content> element names a path with its url attribute, using the same path rules and * and $ wildcards as RFC 9309, and holds a <license>. A minimal file that allows search indexing and nothing else looks like this:
<rsl xmlns="https://rslstandard.org/rsl">
<content url="/">
<license>
<permits type="usage">search</permits>
<prohibits type="usage">ai-train</prohibits>
</license>
</content>
</rsl>Note how permits works: once a license lists permitted values for a type, only those values are allowed, so this file also rules out ai-input and every other unlisted use. The prohibits line spells out the training ban, and a prohibition always wins if the two conflict. Values are separated by spaces, so <prohibits type="usage">ai-train ai-input</prohibits> is valid.
The permits and prohibits elements take a type of usage, user or geo. The usage values are the part most site owners care about:
| Usage value | What it covers, per the specification |
|---|---|
all | Any automated processing, including AI training and search |
ai-all | Any AI use: training, inference, indexing, retrieval, generation, grounding and evaluation |
ai-train | Training or fine-tuning models |
ai-input | Input into AI models: retrieval-augmented generation, grounding, or generating AI responses or search summaries |
ai-index | Inclusion in an AI system's index or retrieval database |
search | Search indexing and results, excluding AI-generated summaries |
User values are commercial, non-commercial, education, government and personal. Geo values are ISO 3166-1 alpha-2 country or region codes, such as US, DE or EU. The evaluation rules are conservative: a prohibition wins over a conflicting permission, a more specific scope overrides a broader one, and a client that cannot obtain a valid RSL document for an asset must treat it as unlicensed.
Payment, reporting and legal terms
This is where RSL goes further than Content Signals or TDMRep. A license can carry a <payment> element. Its types are purchase, subscription, training, crawl, use, contribution, attribution and free, and a license with no payment element is treated as free. An <amount> holds a number with an ISO 4217 currency attribute. An illustrative license for paid commercial AI answers:
<rsl xmlns="https://rslstandard.org/rsl">
<content url="/articles/">
<license>
<permits type="usage">ai-input</permits>
<permits type="user">commercial</permits>
<payment type="use">
<amount currency="USD">0.01</amount>
</payment>
</license>
</content>
</rsl>Inside <payment>, a <standard> URL identifies a shared license such as a Creative Commons licence and a <custom> URL points to your own licensing page. A license can also carry <reporting> for telemetry, provenance or audit requirements and <legal> for warranties, disclaimers and a contact, and each <content> element can name the rights holder in <copyright>. The root's max-age attribute says how many days a client may rely on the file, with a default of 30.
Where does the RSL file go?
The specification defines several discovery methods. You only need one, but you can combine them.
A License line in robots.txt
The robots.txt guide uses a License directive with a fully qualified URL, which may be on a different host from the robots.txt file:
License: https://example.com/license.xml
User-agent: *
Allow: /The guide presents the line as global. Section 4.4.2 of the specification adds scoping: a License line outside any group applies to all clients, and one inside a User-agent group applies only to clients that select that group, and those clients must then ignore the global lines. That mirrors how crawlers pick a single group, which our guide to robots.txt groups explains. Multiple License lines are allowed.
Two caveats. RFC 9309 only says crawlers may interpret records outside the protocol, so nothing obliges them to read License. And Google's robots.txt documentation lists four supported fields, user-agent, allow, disallow and sitemap, and License is not among them. RFC 9309 also says parsing other records must not interfere with the standard ones, so a License line does not change how your Allow and Disallow rules are read.
An HTTP Link header
Link: <https://example.com/license.xml>; rel="license"; type="application/rsl+xml"The specification says this applies equally to non-HTML responses such as JSON, images, video and datasets. AWS documents setting exactly this header with a CloudFront response header policy in its WAF guide to license terms, and notes that the 402 challenge served by its Monetize action does not include the header.
HTML, feeds and files
In HTML you can link the file or inline it:
<link rel="license" type="application/rsl+xml" href="https://example.com/license.xml">The specification also allows an inline <script type="application/rsl+xml"> block, an <rsl:content> element inside RSS items, and embedding in file metadata such as EPUB <metadata>, XMP, ID3 and PNG iTXt chunks. When two licenses conflict, the specification says clients must honour the most restrictive combination of rights.
Does RSL block crawlers?
No. Section 4.4.2 of the specification says the License directive does not modify the access permissions expressed by Allow or Disallow. A crawler that ignores RSL fetches your pages exactly as before, and a crawler you have disallowed is still disallowed whatever your license says.
Enforcement in RSL is optional and sits elsewhere. The specification says servers may enforce terms through paywalls, bot management services, or its CAP and OLP protocols. Under CAP, a server answers 401 when the license token is missing, invalid or expired, 402 when a license or payment is still required, and 403 when the license prohibits the requested use, and it recommends pairing CAP with bot management or Web Bot Auth so the server knows who is asking. That needs a license server and clients that request tokens, which is far more than publishing a file.
If you want payment enforced at the edge today, Cloudflare's separate Pay Per Crawl is a documented mechanism, built on HTTP 402 responses and its own request headers rather than RSL license tokens.
Who backs RSL, and does any crawler read it?
The backing is mostly on the publisher and infrastructure side. The RSL homepage lists supporters including Akamai, Cloudflare, Creative Commons, Fastly, Reddit, Vox Media, Yahoo and Ziff Davis, and the press release claims more than 1,500 endorsing organisations. Fastly's September 2025 post describes serving the license file and advertising it with a Link header, and says a tighter integration is planned. AWS documents the header setup above.
On the reading side, the picture is thinner. At the time of writing:
- OpenAI. Its crawler documentation documents robots.txt rules for
OAI-SearchBotandGPTBotand does not mention RSL. - Anthropic. Its crawler help article documents robots.txt rules and an IETF-draft "no AI training" signal, not RSL.
- Google. Its robots.txt parser supports four fields, and
Licenseis not one of them. - RSL's own pages. Neither the homepage nor the press release names an AI company that honours RSL.
So the honest answer is that crawler support is not documented by the operators who matter most for AI search. That may change, and the file costs little to keep in place while you wait.
When is publishing RSL terms worth it?
RSL is worth the effort if you license content, or want to, and you need a place to state terms in detail: different rules for training and AI answers, commercial versus non-commercial users, a price, or a contact for licensing. Of RSL, Content Signals and TDMRep, only RSL defines a protocol for acquiring licenses from a license server.
It does not replace the controls crawlers document. A sensible stack looks like this:
| Goal | Tool | Acts on requests? |
|---|---|---|
| Keep a named crawler out | robots.txt Disallow, then firewall rules | Yes, for compliant crawlers |
| State a simple yes or no on AI use | Content Signals | No |
| Reserve TDM rights under EU law | TDMRep | No |
| Publish full license and payment terms | RSL | No, unless you run CAP or OLP |
| Charge crawlers per fetch | An edge product such as Pay Per Crawl | Yes |
Whatever terms you publish, check that they agree with your robots.txt. A file that says search is permitted while robots.txt disallows OAI-SearchBot sends mixed messages, and only the robots.txt rule is documented to change behaviour. Allow AI search, block AI training shows a file that keeps the access rules clean.
To check what you serve:
curl -s https://example.com/robots.txt | grep -i '^license:'
curl -sI https://example.com/ | grep -i '^link:'
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.com/license.xmlExpect an absolute URL, a 200 status and an XML content type, ideally application/rsl+xml.
Frequently asked questions
Does adding a License line to robots.txt affect Google Search?
Google's robots.txt documentation lists four supported fields, and License is not one of them. Your Allow and Disallow rules work as before.
Is RSL a W3C or IETF standard?
No. RSL 1.0 is an industry specification published by the RSL Technical Steering Committee in December 2025, not an IETF RFC or a W3C Recommendation.
Can RSL stop AI companies training on my content?
Not by itself. The specification says the License directive does not change access rules, and the major operators we checked do not document reading RSL. Use a Disallow for the training crawler's token to stop compliant crawlers fetching your pages.
Do I need a license server to use RSL?
No. The specification lets you adopt RSL as a licensing and discovery framework without running the Open License Protocol or the Crawler Authorization Protocol. A static XML file and one discovery method are enough to publish terms.
Check what your site tells crawlers
Publishing terms only helps if your access rules say what you mean. The free check reads one page the way AI crawlers do, with no account needed, and reports the robots.txt rules that apply to each named AI crawler, so you can confirm your access rules match the terms you publish.
Filed under Strategy and audits. Topics: robots-txt, ai-training, content-signals, ai-crawlers