llms.txt: what it is, who reads it, and how to write one

llms.txt is a Markdown file at /llms.txt that gives language models a short, curated summary of a site: its name, a one-paragraph description and lists of links to the pages that matter. It is a proposal from llmstxt.org, not a standard, and at the time of writing no major AI search operator documents reading it on other people's sites. It is cheap to write, Google says it neither helps nor harms Google Search visibility, and it is most likely to be useful on documentation sites, where developers point agents and coding tools at it directly.

This guide covers what the proposal says, who reads the file today, how to write one, and how to check it. Facts come from the proposal and the operators' own documentation, read on 8 November 2026.

What is llms.txt?

The proposal was written by Jeremy Howard and is dated 3 September 2024; the page shows a modified date of 10 August 2026. Its reasoning is simple. Context windows are too small to hold most websites, and turning HTML wrapped in navigation, ads and scripts back into clean text is hard to do well. A small Markdown file that says "here is what this site is, and here are the pages worth reading" is easier for a model to use.

Three things it is not:

• Not access control. It does not allow or block anything. Crawler access is still governed by robots.txt, server responses and any WAF or CDN rules. The proposal itself says llms.txt can complement robots.txt "by providing context for allowed content".
• Not a sitemap. A sitemap lists every URL for search engines to crawl. llms.txt is a hand-picked overview, closer to a table of contents.
• Not a standard. It has no IETF or W3C status, and no operator is obliged to read it.

What goes in an llms.txt file?

The format is ordinary Markdown with a fixed order of sections. Only the first heading is required.

One H2 has a special meaning by convention: a section titled Optional holds secondary links that an agent can skip when it needs a shorter context.

Here is a complete example for a fictional analytics product:

# Acme Analytics

> Acme Analytics is a privacy-first web analytics tool for small sites. It runs without cookies and stores data in the EU.

- Pricing is per site, not per page view.
- The tracking script is under 2 KB and loads asynchronously.

## Docs

- [Quick start](https://example.com/docs/quick-start.md): Add the script and see your first visit.
- [Tracking API](https://example.com/docs/api.md): Custom events and goals.
- [Data retention](https://example.com/docs/retention.md): What is stored, where and for how long.

## Product

- [Pricing](https://example.com/pricing): Plans and limits.
- [Security](https://example.com/security): Hosting, encryption and compliance.

## Optional

- [Changelog](https://example.com/changelog): Release notes since 2023.

Where the file lives

Put it at the root of the host, so https://example.com/llms.txt. The proposal also allows a file at a subpath, such as /docs/llms.txt, that covers the URLs under that path. When more than one file applies, agents should use the most specific one.

llms-full.txt and Markdown versions of pages

Two related ideas often travel with llms.txt, and only one of them is in the proposal.

Markdown versions of pages are in the proposal. It asks sites to offer a clean Markdown copy of useful pages at the same URL with .md appended (page.html.md) or with the extension replaced (page.md), using index.html.md or index.md for URLs without a file name. Clients can discover these through a rel="alternate" link with type="text/markdown", either in the HTML head or in an HTTP Link header:

<link rel="alternate" type="text/markdown" href="/docs/quick-start.md">

llms-full.txt is not in the proposal. It is one Markdown file that holds a site's full content rather than links to it. Mintlify says it developed the format in collaboration with its customer Anthropic. Mintlify also says it was later included in the official proposal, but the llmstxt.org page does not mention it at the time of writing. Some large documentation sites publish one: Anthropic's developer docs have an llms.txt that links to a companion llms-full.txt. Treat it as a useful extra for documentation, not as part of any specification.

Who actually reads llms.txt today?

This is where most articles overstate things. Here is what each operator documents at the time of writing.

Google added its llms.txt note to the mythbusting section of the AI optimization guide on 15 June 2026, according to the Search Central changelog. The Lighthouse audit is about browser agents, not Search ranking, so do not read it as Google endorsing the file for SEO.

You may see GPTBot or other crawlers request /llms.txt in your server logs. That shows the file was fetched, not that it affects ranking or citation. Third-party data suggests fetches are rare anyway: a Vercel guide dated 1 October 2026, citing Ahrefs data across 137,000 sites, reports that 97% of llms.txt files were never fetched. The same guide says agents that did reach a file arrived through links from other pages in 86% of attributed fetches, and it ranks llms.txt below Product structured data, Open Graph tags and a product feed for online stores.

So who does use it? No operator publishes a list. The clearest case is a tool that is told to: a coding assistant, documentation search tool or agent where a developer pastes the URL or configures it as a source. That is why documentation sites, APIs and developer products are the most likely to get value from it. Our AI search myths guide covers other extras that get more attention than the documentation supports.

llms.txt vs robots.txt vs sitemaps

If AI search tools cannot see your site, start with the first column, not the last. Check robots.txt for AI crawlers, WAF challenges and whether your content is in the raw HTML before you think about llms.txt. A sitemap still does more for discovery.

How to write and publish an llms.txt file

1. Write the H1 and summary first. Use your real name and one or two sentences that say what you do, for whom, and anything a model would get wrong without being told. Keep the facts consistent with your homepage.
2. Add a few key facts as a list. Prices, limits, supported platforms, what you do not do. Short, checkable statements work best.
3. Group your best pages under H2 headings. Docs, guides, product, policies. Give each link a one-line note. Ten good links beat two hundred.
4. Put secondary links under ## Optional. Changelogs, archives, legal pages.
5. Use absolute URLs that return 200. Every link should resolve without redirects or login walls. If you publish Markdown versions, link to those.
6. Serve it as plain text from the root. On most hosts, a static file in the web root or public/ folder is enough. Return a 200 status and UTF-8 text.
7. Link to it. A plain link from your footer or docs index costs nothing, and the Vercel guide above found that most agents that reached an llms.txt file got there through a link.
8. Keep it current. A stale file that lists retired pages or old prices is worse than none. Generate it from your content if you can.

Generating it in Next.js

On a Next.js App Router site you can generate the file from your content with a route handler in a folder named llms.txt:

// app/llms.txt/route.ts
import { llmsTxt } from "@/lib/llms";

export function GET() {
  return new Response(llmsTxt(), {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

If the handler reads only local files, Next.js can prerender it at build time; check the build output to confirm the route is static rather than rendered on every request. Crawlito's own crawlito.com/llms.txt is built this way: an H1, a blockquote summary, a short list of facts about how the score works and what the service costs, a Product section, one section of guides per blog category and an Optional section that links to an llms-full.txt with every guide in one file. The Next.js guide covers the rest of a Next.js site's crawler setup.

How to check your llms.txt

Start with the status code and content type:

curl -sI https://example.com/llms.txt

You want a 200 status line (HTTP/2 200 or HTTP/1.1 200) and a text content type such as text/plain or text/markdown. A 404 is fine if you chose not to publish one. A 5xx is the one result Lighthouse flags, and a challenge page from your CDN means agents get HTML instead of Markdown.

Confirm the first line is an H1:

curl -s https://example.com/llms.txt | head -1 | grep -q '^# ' && echo "H1 ok" || echo "missing H1"

If the file starts with a byte order mark, which the proposal allows, this simple check reports a miss even when the H1 is there.

Check every linked URL returns 200:

curl -s https://example.com/llms.txt \
  | grep -oE '\]\(https?://[^)]+\)' | sed 's/^](//; s/)$//' \
  | while read -r url; do
      printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' "$url")" "$url"
    done

To see who requests it, search your access logs. In the common combined log format, the user agent is the sixth field when you split on quotes:

grep -E '"(GET|HEAD) /llms\.txt ' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

A user agent string alone can be faked, so verify real crawlers against published IP ranges before drawing conclusions.

Frequently asked questions

Do I need llms.txt to appear in ChatGPT or Google AI Overviews?

No. Google says Search ignores llms.txt and its AI features need no special files, and OpenAI says to allow OAI-SearchBot in robots.txt and its published IP ranges. None of them name llms.txt as a requirement.

Can llms.txt hurt my site?

Not in Google Search, which says it ignores the file. The real risks are a stale file with wrong facts, or one that lists pages you meant to keep out of AI tools: it does not block anything, so use robots.txt or noindex for that.

Should llms.txt replace robots.txt?

No. robots.txt tells crawlers which paths they may fetch, and OpenAI, Anthropic, Perplexity and Google document honouring it for their search and training crawlers. llms.txt only describes content, and the proposal says it can complement robots.txt.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a short index of links, defined on llmstxt.org. llms-full.txt is a single file with the full text of a site's content, a convention Mintlify says it developed with Anthropic and that the llmstxt.org page does not define.

Which sites benefit most from llms.txt?

Most likely documentation, API and developer product sites, where people point coding assistants and agents at the docs. For a typical business or e-commerce site it is a low-priority extra.

Check the parts that matter first

llms.txt is worth an hour if you run a docs-heavy site, and harmless if you don't. It will not fix a blocked crawler, a challenge page or an empty client-rendered page. As a next step, run the free page check on one URL: it tests the robots.txt rules for each AI crawler, whether the main text is in the raw HTML, and basic structure, with no account needed. If you later want every page reviewed, a full audit also reports whether llms.txt is present and parses, with a very low weight in the score because the file is optional.