Skip to content

RSL standard: what Really Simple Licensing means for your site

The RSL standard lets you publish machine-readable licensing terms for AI use. See what an RSL file declares, where it goes, and whether any crawler reads it.

By Crawlito · 8 min read

In short

  • RSL 1.0 was published on 10 December 2025 as an industry specification with Recommendation status, and it defines an XML format served as application/rsl+xml.
  • An RSL file lists permitted and prohibited uses such as ai-train, ai-input, ai-index and search, plus optional payment, reporting and legal terms.
  • You point crawlers to the file with a License line in robots.txt, an HTTP Link header, an HTML link element, an RSS item or embedded file metadata.
  • The specification says the License directive does not change Allow or Disallow, so RSL declares terms and blocks nothing by itself.
  • At the time of writing, the crawler documentation from OpenAI, Anthropic and Google that we checked does not mention RSL or a License directive.
On this page

The RSL standard (Really Simple Licensing) is an XML format for publishing the terms under which machines may use your content: which uses are allowed, which are prohibited, and what, if anything, they cost. You host one license file and point to it from robots.txt, an HTTP header or your HTML. RSL describes terms and blocks nothing on its own, and at the time of writing none of the major AI crawler operators we checked documents reading it.

This post explains what RSL 1.0 contains, where the file goes, who backs it and what it changes today. If you have already read our posts on Content Signals and TDMRep, this one covers what RSL adds on top of them: a full license vocabulary and a path to payment.

What is the RSL standard?

The RSL 1.0 specification was published on 10 December 2025 with the status "Recommendation". Its previous version was 0.9. The document calls itself an industry specification: it is not an IETF RFC or a W3C Recommendation. The RSL Technical Steering Committee announced it, and its press release says the nonprofit RSL Collective co-developed it. The specification's editors come from the RSL Collective, Condé Nast, Ziff Davis, Schema.org, Yahoo, Automattic, O'Reilly Media and Fastly.

The specification has a required core and three optional protocols:

PartWhat it doesRequired?
Core vocabulary and discoveryThe XML elements for terms, and the ways to point clients to themYes
Open License Protocol (OLP)An OAuth 2.0 extension through which a client obtains a license token from a license serverNo
Crawler Authorization Protocol (CAP)An HTTP Authorization: License <token> scheme a server can check before serving contentNo
Encrypted Media Standard (EMS)Encrypted content whose key is released to licensed clientsNo

The specification says you may adopt RSL as a licensing and discovery framework without implementing any of the network protocols. For most sites, that is the realistic starting point: one XML file and a pointer to it.

What does an RSL file declare?

The root element is <rsl> with the namespace https://rslstandard.org/rsl. Inside it, each <content> element names a path with its url attribute, using the same path rules and * and $ wildcards as RFC 9309, and holds a <license>. A minimal file that allows search indexing and nothing else looks like this:

xml
<rsl xmlns="https://rslstandard.org/rsl">
  <content url="/">
    <license>
      <permits type="usage">search</permits>
      <prohibits type="usage">ai-train</prohibits>
    </license>
  </content>
</rsl>

Note how permits works: once a license lists permitted values for a type, only those values are allowed, so this file also rules out ai-input and every other unlisted use. The prohibits line spells out the training ban, and a prohibition always wins if the two conflict. Values are separated by spaces, so <prohibits type="usage">ai-train ai-input</prohibits> is valid.

The permits and prohibits elements take a type of usage, user or geo. The usage values are the part most site owners care about:

Usage valueWhat it covers, per the specification
allAny automated processing, including AI training and search
ai-allAny AI use: training, inference, indexing, retrieval, generation, grounding and evaluation
ai-trainTraining or fine-tuning models
ai-inputInput into AI models: retrieval-augmented generation, grounding, or generating AI responses or search summaries
ai-indexInclusion in an AI system's index or retrieval database
searchSearch indexing and results, excluding AI-generated summaries

User values are commercial, non-commercial, education, government and personal. Geo values are ISO 3166-1 alpha-2 country or region codes, such as US, DE or EU. The evaluation rules are conservative: a prohibition wins over a conflicting permission, a more specific scope overrides a broader one, and a client that cannot obtain a valid RSL document for an asset must treat it as unlicensed.

This is where RSL goes further than Content Signals or TDMRep. A license can carry a <payment> element. Its types are purchase, subscription, training, crawl, use, contribution, attribution and free, and a license with no payment element is treated as free. An <amount> holds a number with an ISO 4217 currency attribute. An illustrative license for paid commercial AI answers:

xml
<rsl xmlns="https://rslstandard.org/rsl">
  <content url="/articles/">
    <license>
      <permits type="usage">ai-input</permits>
      <permits type="user">commercial</permits>
      <payment type="use">
        <amount currency="USD">0.01</amount>
      </payment>
    </license>
  </content>
</rsl>

Inside <payment>, a <standard> URL identifies a shared license such as a Creative Commons licence and a <custom> URL points to your own licensing page. A license can also carry <reporting> for telemetry, provenance or audit requirements and <legal> for warranties, disclaimers and a contact, and each <content> element can name the rights holder in <copyright>. The root's max-age attribute says how many days a client may rely on the file, with a default of 30.

Where does the RSL file go?

The specification defines several discovery methods. You only need one, but you can combine them.

A License line in robots.txt

The robots.txt guide uses a License directive with a fully qualified URL, which may be on a different host from the robots.txt file:

License: https://example.com/license.xml

User-agent: *
Allow: /

The guide presents the line as global. Section 4.4.2 of the specification adds scoping: a License line outside any group applies to all clients, and one inside a User-agent group applies only to clients that select that group, and those clients must then ignore the global lines. That mirrors how crawlers pick a single group, which our guide to robots.txt groups explains. Multiple License lines are allowed.

Two caveats. RFC 9309 only says crawlers may interpret records outside the protocol, so nothing obliges them to read License. And Google's robots.txt documentation lists four supported fields, user-agent, allow, disallow and sitemap, and License is not among them. RFC 9309 also says parsing other records must not interfere with the standard ones, so a License line does not change how your Allow and Disallow rules are read.

http
Link: <https://example.com/license.xml>; rel="license"; type="application/rsl+xml"

The specification says this applies equally to non-HTML responses such as JSON, images, video and datasets. AWS documents setting exactly this header with a CloudFront response header policy in its WAF guide to license terms, and notes that the 402 challenge served by its Monetize action does not include the header.

HTML, feeds and files

In HTML you can link the file or inline it:

html
<link rel="license" type="application/rsl+xml" href="https://example.com/license.xml">

The specification also allows an inline <script type="application/rsl+xml"> block, an <rsl:content> element inside RSS items, and embedding in file metadata such as EPUB <metadata>, XMP, ID3 and PNG iTXt chunks. When two licenses conflict, the specification says clients must honour the most restrictive combination of rights.

Does RSL block crawlers?

No. Section 4.4.2 of the specification says the License directive does not modify the access permissions expressed by Allow or Disallow. A crawler that ignores RSL fetches your pages exactly as before, and a crawler you have disallowed is still disallowed whatever your license says.

Enforcement in RSL is optional and sits elsewhere. The specification says servers may enforce terms through paywalls, bot management services, or its CAP and OLP protocols. Under CAP, a server answers 401 when the license token is missing, invalid or expired, 402 when a license or payment is still required, and 403 when the license prohibits the requested use, and it recommends pairing CAP with bot management or Web Bot Auth so the server knows who is asking. That needs a license server and clients that request tokens, which is far more than publishing a file.

If you want payment enforced at the edge today, Cloudflare's separate Pay Per Crawl is a documented mechanism, built on HTTP 402 responses and its own request headers rather than RSL license tokens.

Who backs RSL, and does any crawler read it?

The backing is mostly on the publisher and infrastructure side. The RSL homepage lists supporters including Akamai, Cloudflare, Creative Commons, Fastly, Reddit, Vox Media, Yahoo and Ziff Davis, and the press release claims more than 1,500 endorsing organisations. Fastly's September 2025 post describes serving the license file and advertising it with a Link header, and says a tighter integration is planned. AWS documents the header setup above.

On the reading side, the picture is thinner. At the time of writing:

  • OpenAI. Its crawler documentation documents robots.txt rules for OAI-SearchBot and GPTBot and does not mention RSL.
  • Anthropic. Its crawler help article documents robots.txt rules and an IETF-draft "no AI training" signal, not RSL.
  • Google. Its robots.txt parser supports four fields, and License is not one of them.
  • RSL's own pages. Neither the homepage nor the press release names an AI company that honours RSL.

So the honest answer is that crawler support is not documented by the operators who matter most for AI search. That may change, and the file costs little to keep in place while you wait.

When is publishing RSL terms worth it?

RSL is worth the effort if you license content, or want to, and you need a place to state terms in detail: different rules for training and AI answers, commercial versus non-commercial users, a price, or a contact for licensing. Of RSL, Content Signals and TDMRep, only RSL defines a protocol for acquiring licenses from a license server.

It does not replace the controls crawlers document. A sensible stack looks like this:

GoalToolActs on requests?
Keep a named crawler outrobots.txt Disallow, then firewall rulesYes, for compliant crawlers
State a simple yes or no on AI useContent SignalsNo
Reserve TDM rights under EU lawTDMRepNo
Publish full license and payment termsRSLNo, unless you run CAP or OLP
Charge crawlers per fetchAn edge product such as Pay Per CrawlYes

Whatever terms you publish, check that they agree with your robots.txt. A file that says search is permitted while robots.txt disallows OAI-SearchBot sends mixed messages, and only the robots.txt rule is documented to change behaviour. Allow AI search, block AI training shows a file that keeps the access rules clean.

To check what you serve:

bash
curl -s https://example.com/robots.txt | grep -i '^license:'
curl -sI https://example.com/ | grep -i '^link:'
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.com/license.xml

Expect an absolute URL, a 200 status and an XML content type, ideally application/rsl+xml.

Frequently asked questions

Google's robots.txt documentation lists four supported fields, and License is not one of them. Your Allow and Disallow rules work as before.

Is RSL a W3C or IETF standard?

No. RSL 1.0 is an industry specification published by the RSL Technical Steering Committee in December 2025, not an IETF RFC or a W3C Recommendation.

Can RSL stop AI companies training on my content?

Not by itself. The specification says the License directive does not change access rules, and the major operators we checked do not document reading RSL. Use a Disallow for the training crawler's token to stop compliant crawlers fetching your pages.

Do I need a license server to use RSL?

No. The specification lets you adopt RSL as a licensing and discovery framework without running the Open License Protocol or the Crawler Authorization Protocol. A static XML file and one discovery method are enough to publish terms.

Check what your site tells crawlers

Publishing terms only helps if your access rules say what you mean. The free check reads one page the way AI crawlers do, with no account needed, and reports the robots.txt rules that apply to each named AI crawler, so you can confirm your access rules match the terms you publish.

Filed under Strategy and audits. Topics: robots-txt, ai-training, content-signals, ai-crawlers