Mintlify robots.txt, llms.txt and noindex: the SEO settings
How Mintlify robots.txt works: the generated file and its Content-Signal line, custom overrides, hidden and noindex pages, llms.txt scope and a check routine.
By Crawlito · 8 min read
In short
- Mintlify generates robots.txt with a Content-Signal line of ai-train=yes, search=yes, ai-input=yes for all user agents, and a custom robots.txt at the project root replaces it and is served as-is, without that line.
- Pages with hidden: true get a noindex meta tag and are left out of the sitemap and llms.txt, while a page simply missing from docs.json navigation gets no noindex tag and can still be indexed if a crawler finds the URL.
- llms.txt and llms-full.txt list pages from the default language and version and exclude hidden and noindex pages unless seo.indexing is set to all.
- The Don't index project setting serves a site-wide Disallow robots.txt and a 404 for llms.txt, but llms-full.txt stays available and a custom robots.txt is served unchanged.
- searchable: false keeps a page in the sitemap, search engine indexing and llms.txt, and removes it only from in-product search and the AI assistant.
On this page
- What Mintlify puts in robots.txt by default
- How to replace it with a custom robots.txt
- Hidden, noindex, searchable: which control does what
- What appears in Mintlify's llms.txt
- What Don't index project does, and what it leaves open
- How to check your Mintlify settings
- Frequently asked questions
- Check what crawlers see on your docs
Mintlify generates robots.txt, sitemap.xml and llms.txt for you. The default robots.txt lets every crawler in and adds a Content-Signal line that says yes to search, AI answers and AI training. To change what crawlers are told, you add your own robots.txt at the project root, which Mintlify then serves exactly as written. To keep individual pages out of search engines and out of llms.txt, you use noindex: true or hidden: true in the page frontmatter, and the two behave differently in ways that are easy to get wrong.
This post maps each documented Mintlify control to what it changes, based on Mintlify's SEO, hidden pages and llms.txt pages, read on 15 November 2026. For what llms.txt is in general, see the llms.txt guide; for what the signals mean, see Content Signals in robots.txt.
What Mintlify puts in robots.txt by default
Mintlify's SEO page says the generated robots.txt includes Content-Signal directives that "apply to all user agents":
User-agent: *
Content-Signal: ai-train=yes, search=yes, ai-input=yesSo out of the box a Mintlify site allows crawling and states a preference for all three uses, training included. That differs from Cloudflare's managed robots.txt, which at the time of writing sets search=yes, ai-train=no and states nothing for ai-input. Neither choice is wrong, but you should know which one you are publishing. Remember that a Content-Signal line is a stated preference, not access control: Allow and Disallow still decide who may fetch what.
At the time of writing, Mintlify's own documentation at www.mintlify.com/docs/robots.txt also carries Disallow lines for /cdn-cgi/ and /_next/, an Allow for /_next/image and a Sitemap line. The SEO page does not document those extra lines as part of the default file, so check your own deployed file rather than assuming it matches.
On fully authenticated sites, requests for robots.txt and the sitemap redirect to login, and the sitemap lists no pages. On partially authenticated sites both are public and the sitemap lists only public pages.
How to replace it with a custom robots.txt
Create robots.txt at the root of your Mintlify project. Mintlify's documentation says the custom file overrides the generated one, is served "as-is, without the default Content-Signal directives", and that deleting it brings the default back. The same rule applies to a custom sitemap.xml.
Because nothing is merged, a custom file has to carry everything you want crawlers to see. A file that keeps AI search and user-triggered fetchers allowed, while stating a no to training, could look like this:
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
Sitemap: https://docs.example.com/sitemap.xmlTwo details matter here. First, under RFC 9309 a crawler that matches a named group, such as GPTBot, follows that group and falls back to the * group only when no group names it, so the Content-Signal line above does not reach it. That is fine in this example because it is blocked anyway; repeat the signal in any named group you allow. Second, replace the example host with your real docs domain, including any base path such as /docs. For the reasoning behind the split between training and search crawlers, see allow AI search, block AI training.
Mintlify's GEO guide asks you to make sure a custom file does not block AI agents, points out that User-agent: * with Disallow: / blocks them along with every other crawler, and says the mint score CLI command has a robotsTxtAllowsAI check that passes when no AI user agents are blocked.
Hidden, noindex, searchable: which control does what
This is where the controls are easiest to mix up. Mintlify documents six ways to change a page's visibility, set in page frontmatter or in docs.json, and they do not all affect the same outputs. Mintlify's hidden pages documentation gives this matrix:
| Property | Sidebar | Site search | Sitemap | Search engine indexing | AI assistant | llms.txt |
|---|---|---|---|---|---|---|
hidden: true | Hidden | Excluded | Excluded | Excluded | Excluded | Excluded |
Not listed in docs.json | Hidden | Excluded | Excluded | Excluded from sitemap only | Excluded | Excluded |
noindex: true | Visible | Excluded | Excluded | Excluded | Excluded | Excluded |
searchable: false | Visible | Excluded | Included | Included | Excluded | Included |
searchable: true on hidden tab or group | Hidden | Included | Included | Included | Included | Included |
seo.indexing: "all" | Hidden | Included | Included | Included | Included | Included |
The llms.txt column also applies to llms-full.txt.
Three consequences follow:
- Leaving a page out of navigation is not noindex. Mintlify says
hidden: trueadds anoindexmeta tag, while omitting a page fromdocs.jsononly keeps it out ofsitemap.xml. A crawler that finds the URL through a link elsewhere can still index it. If the page must stay out of search, sethidden: trueornoindex: truein its frontmatter. hiddenimpliesnoindex, but not the reverse.noindex: truekeeps the page in the sidebar while removing it from search engines, site search, the AI assistant andllms.txt.searchable: falseis not a search engine control. It removes a page only from in-product search and the AI assistant. The page stays in the sitemap, stays indexable and stays inllms.txt.
Hidden pages are also not private. Mintlify's warning is explicit: anyone with the URL can open them. Content that must be restricted needs Mintlify's authentication, not hidden.
Noindex for a folder or the whole site
For a section, such as an old API version, seo.paths in docs.json applies meta tags to every page under a folder:
"seo": {
"paths": [
{
"path": "guides/v1",
"metatags": { "robots": "noindex" }
}
]
}Page frontmatter overrides seo.paths, which overrides seo.metatags. Paths match page URLs, so a translated folder such as es/guides/v1 needs its own entry. Mintlify documents that this adds the robots meta tag; it does not say whether a seo.paths noindex also removes those pages from llms.txt, so check the generated file after deploying. Setting "robots": "noindex" in seo.metatags applies the tag to every page.
What appears in Mintlify's llms.txt
Mintlify hosts llms.txt at the site root and at /.well-known/llms.txt, and llms-full.txt at both equivalent paths. Per its documentation:
- Pages are listed in
docs.jsonnavigation order, with links ending in.mdso tools can fetch each page's Markdown version. Pages outside the navigation thatseo.indexing: "all"brings in come last, in alphabetical order. - Each page's description comes from the first paragraph of its frontmatter
description, cut at 300 characters. Pages without one are listed with no description. - Both files cover the default language and default version only, and exclude hidden and
noindexpages unlessseo.indexingis"all". - A generated
llms.txtindex cannot exceed 100,000 characters. When it would, Mintlify moves groups of page links into split indexes under/_llms/, which a reverse proxy or path allowlist has to forward along withllms.txt. The limit does not apply tollms-full.txt. - A custom
llms.txtorllms-full.txtat the project root replaces the generated file. It must have the site title as an H1 heading.
Mintlify also adds a Link header and an X-Llms-Txt: /llms.txt header to every response, including 404 pages, so tools can find the file without guessing its location. Under a base path, the advertised paths carry that prefix: Mintlify's own docs at /docs answered with x-llms-txt: /docs/llms.txt when we checked. Whether a given AI crawler reads these files is a separate question that the llms.txt guide covers; Mintlify's generating them does not mean any search operator uses them.
If your docs live under a reverse proxy at /docs, read documentation sites and AI search for the wider checks on deep links and versions. For a comparison with another docs platform, see GitBook's llms.txt and Markdown pages.
What Don't index project does, and what it leaves open
For staging or internal review, Mintlify's dashboard has a Don't index project setting on its Site settings page. When it is on, Mintlify documents that the site:
- renders every page with
noindex, nofollow, - returns an empty
sitemap.xml, - returns 404 for
llms.txt, including/.well-known/llms.txt, - serves this
robots.txt:
User-agent: *
Content-Signal: ai-train=no, search=no, ai-input=no
Disallow: /In-product search, the AI assistant and the MCP server keep working; to remove pages from those, Mintlify points to noindex: true or searchable: false on each page.
Two gaps are documented. The setting does not gate llms-full.txt, which stays available at /llms-full.txt with your full content, and each page's Markdown export keeps pointing to /llms.txt even though that file returns 404. And a custom robots.txt at the project root is served unchanged, so if you have one, the site-wide Disallow never appears. Changing the setting redeploys the site, and it takes effect once the redeployment finishes. If a preview copy must stay out of search, also read staging noindex leaks, because a public URL with only meta tags is still fetchable.
How to check your Mintlify settings
Run these against the deployed site, not the local preview. Mintlify notes that local builds add /src/_props to URLs, which is not part of the canonical URL.
SITE=https://docs.example.com
# 1. Is the robots.txt the one you expect?
curl -s "$SITE/robots.txt"
# 2. Is a hidden page carrying noindex?
curl -s "$SITE/guides/hidden-page" | grep -io '<meta[^>]*name="robots"[^>]*>'
# 3. Is the hidden page absent from the sitemap and llms.txt?
curl -s "$SITE/sitemap.xml" | grep -c "guides/hidden-page"
curl -s "$SITE/llms.txt" | grep -c "guides/hidden-page"
# 4. Discovery headers
curl -sI "$SITE/" | grep -i -E '^(link|x-llms-txt):'
# 5. With Don't index project on, llms-full.txt still answers
curl -s -o /dev/null -w "%{http_code}\n" "$SITE/llms-full.txt"What to look for:
- If you have no custom file, the output should include the
Content-Signalline. If you have one, it should match your repository file exactly. - A page you meant to exclude should return a robots meta tag containing
noindex. If it returns nothing, it was probably only left out of navigation. The pattern allows for other attributes beforename, because Mintlify's pages render meta tags as<meta data-mint-head name="...">, which a plain<meta name="robots"search misses. Mintlify's own hidden page example, which is only left out of navigation, carried no robots meta tag when we fetched it, as the documentation predicts. - Both counts should be
0for excluded pages, and above0for pages undersearchable: trueorseo.indexing: "all". - Expect a
Linkheader listingllms.txtand anX-Llms-Txtheader, both with your base path if you have one. - A
200here means your full content is still public, whatever the indexing setting says.
Frequently asked questions
Does Mintlify's default robots.txt allow AI training?
Yes. It blocks no crawler and includes Content-Signal: ai-train=yes, search=yes, ai-input=yes for all user agents, which states a yes to training. To change it, add a custom robots.txt at the project root.
Is a Mintlify page that is not in docs.json navigation noindexed?
No. Mintlify documents that such a page is only left out of the sitemap, so a crawler that finds the URL can still index it. Add hidden: true or noindex: true to its frontmatter to get the noindex meta tag.
Do noindex pages appear in Mintlify's llms.txt?
Not by default. Both llms.txt and llms-full.txt exclude hidden and noindex: true pages unless seo.indexing is set to "all".
Does Don't index project block llms-full.txt?
No. Mintlify says the setting returns 404 for llms.txt but leaves llms-full.txt available, and a custom robots.txt is served unchanged while the setting is on.
Check what crawlers see on your docs
After changing any of these settings, run the free check on a key documentation page to confirm the main AI crawlers can fetch it and that the robots rules and meta tags are the ones you intended. If your docs span versions, languages or a reverse proxy, an audit checks those settings across the whole site with evidence for each finding.
Filed under Platform guides. Topics: mintlify, robots-txt, llms-txt, content-signals, documentation