Serving Markdown to AI agents: Accept text/markdown explained
How to serve Markdown to AI agents with Accept: text/markdown, Cloudflare Markdown for Agents or .md URLs, and how to avoid canonical, caching and cloaking issues.
By Crawlito · 6 min read
In short
- Content negotiation lets one URL return HTML to browsers and Markdown to clients that send Accept: text/markdown, as long as the response includes Vary: Accept.
- Cloudflare's Markdown for Agents converts HTML to Markdown at the edge for requests that ask for text/markdown, on Pro, Business and Enterprise plans and SSL for SaaS.
- The llms.txt proposal takes a different route: a Markdown copy of each page at the same URL with .md appended.
- A Markdown version should carry the same content as the HTML page; serving crawlers different substance is what Google's cloaking policy targets.
- Markdown served from a separate .md URL should point to the HTML page as canonical, for example with an HTTP Link header.
On this page
You can serve Markdown to AI agents in two ways. With content negotiation, the same URL returns HTML to browsers and Markdown to any client whose Accept header asks for text/markdown. With separate Markdown URLs, each page has a twin such as /pricing.md that agents can fetch directly. Cloudflare now offers the first approach as a setting, Markdown for Agents, and the llms.txt proposal describes the second.
Either way, Markdown is a convenience, not a replacement for good HTML. Most crawlers still request HTML, and the Markdown should say the same thing as the page. This guide covers how each approach works, how to set them up, and the canonical, caching and cloaking details that are easy to get wrong.
Why serve Markdown at all?
An HTML page carries navigation, footers, cookie banners, scripts and styling around its actual content. An agent that wants the content has to strip all of that and guess which parts matter. Markdown gives it the headings, paragraphs, lists, tables and links with almost nothing else, which is smaller to download and simpler to process.
That benefit is real but limited. No major AI search operator documents that it prefers Markdown, ranks it differently or requests it by default. Treat Markdown as an extra format for agents that ask for it, and keep your HTML complete and server-rendered for everyone else. If your HTML is an empty JavaScript shell, fix that first; our guide to raw vs rendered HTML explains why.
How Accept: text/markdown content negotiation works
HTTP content negotiation is the mechanism browsers already use for languages and image formats. The client says what it can handle in the Accept header, and the server picks a representation:
GET /docs/quick-start HTTP/1.1
Host: www.example.com
Accept: text/markdown, text/html;q=0.9HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: AcceptThree details make it work:
- Same URL, same content. The Markdown is a different format of the same page, not a different page.
Vary: Accepton every response. Without it, a CDN or browser cache may store the Markdown and serve it to the next visitor who wanted HTML, or the reverse.- Browsers keep getting HTML. Browsers send
text/htmlfirst in theirAcceptheader, so only clients that explicitly ask for Markdown receive it.
Cloudflare Markdown for Agents
Cloudflare's Markdown for Agents does the conversion at the edge. When a request includes text/markdown in its Accept header, Cloudflare fetches the HTML from your origin, converts it, and returns Markdown. According to Cloudflare's documentation at the time of writing:
| Detail | What Cloudflare documents |
|---|---|
| Plans | Pro, Business and Enterprise, and SSL for SaaS customers, at no extra cost |
| How to enable | Dashboard: Agent Readiness > Markdown for Agents; or a Configuration Rule for specific hosts or paths; or the zone setting content_converter via the API |
| Response type | Content-Type: text/markdown; charset=utf-8 |
| Caching | Adds Accept to Vary, keeping your origin's existing Vary values |
| What is converted | HTML only; up to 6 MiB of decompressed HTML |
| What is stripped | Headers, footers, navigation, scripts and styles; JSON-LD is kept in a trailing code block |
| Metadata | Optional YAML frontmatter built from <meta> tags |
| Removed headers | Including ETag and Last-Modified, so conditional requests cannot be honoured for converted responses |
| Content signals | Adds a content-signal header defaulting to ai-train=yes, search=yes, ai-input=yes unless your origin sets its own |
Two of those deserve attention before you switch it on.
The default content signal. If you have opted out of AI training elsewhere, for example with a robots.txt content signal or by blocking training crawlers, the default ai-train=yes on Markdown responses says the opposite. Cloudflare says your origin's content-signal header is authoritative, so set one at the origin that matches your policy. Our guide to content signals in robots.txt explains the three values.
Lost validators. Because converted responses drop ETag and Last-Modified, agents cannot make cheap conditional requests for the Markdown version. That is a trade-off, not a fault, but it is worth knowing if you rely on conditional requests to keep crawl load down.
To test it after enabling:
curl -sI -H "Accept: text/markdown" https://www.example.com/docs/quick-start | grep -iE "content-type|vary|content-signal"
curl -s -H "Accept: text/markdown" https://www.example.com/docs/quick-start | head -40Separate .md URLs and llms.txt
The llms.txt proposal suggests a different pattern: pages that are useful for language models should offer a clean Markdown version at the same URL with .md appended. Clients can discover it from a link in the HTML head:
<link rel="alternate" type="text/markdown" href="https://www.example.com/docs/quick-start.md">And your llms.txt file can point straight at the Markdown versions. Our llms.txt guide covers that file in detail.
Separate URLs are easier to build on a static site, simpler to cache and simpler to debug, because the URL itself tells you which format you get. The cost is a second URL per page, which brings canonical questions.
Canonical for Markdown URLs
A Markdown file cannot carry a <link rel="canonical"> element, because it is not HTML. Google supports a canonical declared in an HTTP Link header for non-HTML documents, as described in its guide to consolidating duplicate URLs:
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Link: <https://www.example.com/docs/quick-start>; rel="canonical"That tells Google the HTML page is the version to index. Google says it supports this method for web search; other operators do not document how they treat it.
Setting up content negotiation yourself
If you are not on Cloudflare, you can negotiate at your server, provided you already have Markdown files for each page.
Nginx, serving pre-built .md files next to the HTML:
map $http_accept $md_suffix {
default "";
"~*text/markdown" ".md";
}
server {
location /docs/ {
types {
text/markdown md;
text/html html;
}
add_header Vary Accept always;
try_files $uri$md_suffix $uri.html $uri $uri/ =404;
}
}A request for /docs/quick-start with Accept: text/markdown returns /docs/quick-start.md if it exists, and /docs/quick-start.html otherwise. The types block is scoped to this location so it does not replace your server's other MIME types. Note that this map treats any mention of text/markdown as a preference; a stricter setup would compare quality values.
Next.js middleware can do the same by rewriting to a route that returns Markdown when the Accept header contains text/markdown, and adding Vary: Accept to both responses.
Whichever you use, generate the Markdown from the same source as the HTML. Hand-maintained Markdown copies drift.
The cloaking question
Google's spam policies define cloaking as presenting different content to users and search engines with the intent to manipulate rankings or mislead users. Converting the same page into another format is not that. Writing a separate, keyword-stuffed or more flattering Markdown version for machines would be.
A practical test: if a person read the HTML page and the Markdown version side by side, would they find the same claims, prices, facts and caveats? If yes, you are serving a format. If the Markdown says things the page does not, you are serving different content. Our article on serving AI crawlers different content goes through where the line sits.
Two more cautions:
- Do not key it on user agent. Negotiate on the
Acceptheader the client sends, not on whether the user agent looks like a bot. Serving formats by user agent is harder to test and closer to cloaking. - Keep the HTML complete. Googlebot and most AI crawlers request HTML. If the Markdown is good and the HTML is a shell, most crawlers still get the shell.
Frequently asked questions
Do AI crawlers request Markdown?
Which agents send Accept: text/markdown is not well documented, and the major AI search crawlers do not document doing so by default. Check your logs for requests with that header before investing heavily.
Does Cloudflare Markdown for Agents work on the Free plan?
Cloudflare's documentation lists Pro, Business and Enterprise plans and SSL for SaaS customers at the time of writing. Check the current documentation for your plan.
Is serving Markdown to bots cloaking?
Not if the Markdown is the same content in another format and you serve it based on the Accept header. It becomes a problem when the Markdown carries different claims or content from the page people see.
Should .md URLs be in my sitemap?
Generally no. List the canonical HTML pages in your sitemap, link Markdown versions with rel="alternate" or from llms.txt, and point the Markdown back to the HTML with a canonical Link header.
Start with the HTML
Markdown helps agents that ask for it; HTML is what nearly every crawler reads. Run the free check to see what your page's HTML gives a crawler before adding a Markdown layer on top.
Filed under Structure and schema. Topics: llms-txt, cloudflare, html, caching, ai-crawlers