Skip to content

GitBook llms.txt, .md pages and indexing: what AI tools get

What GitBook llms.txt, llms-full.txt, .md pages and the docs MCP server expose, why hidden pages stay readable, and how indexing settings and custom domains apply.

By Crawlito · 9 min read

In short

  • Every published GitBook site automatically gets a Markdown version of each page at its URL plus .md, an /llms.txt index and an /llms-full.txt file, plus an MCP server at /~gitbook/mcp while its page action is enabled.
  • GitBook's docs say hiding a page only removes it from the published table of contents: hidden pages stay in llms-full.txt and remain readable through the site's MCP server.
  • Turning off search engine indexing, for a whole site or per page, controls search engines and web crawlers; GitBook does not document that it removes a page from llms-full.txt or MCP.
  • All the AI outputs live under the site's root URL, so a site on example.com/docs serves example.com/docs/llms.txt, not example.com/llms.txt.
  • Every output can be checked with curl in a minute: status code, content type, and whether a page you meant to keep private shows up.
On this page

Every docs site published on GitBook automatically gets four outputs for AI tools: a Markdown copy of each page at its URL with .md appended, an /llms.txt index, a single /llms-full.txt file with the full content, and a Model Context Protocol (MCP) server at /~gitbook/mcp. GitBook's AI credits and usage FAQ says every plan includes them. The catch for anyone trying to keep pages out: GitBook says a hidden page is only removed from the table of contents, and stays in llms-full.txt and in the MCP server.

This post lists what each output contains, how GitBook's visibility and indexing settings affect them, where they live on a custom domain, and how to check each one. Facts come from GitBook's documentation, read on 15 November 2026. Where we describe how GitBook's own docs site responds, that is what we observed then, not a documented guarantee. For what the llms.txt format is and who reads it, see the llms.txt guide; for content and versioning advice, see documentation sites and AI search.

What GitBook publishes for AI tools

GitBook's LLM-ready docs page describes these outputs as automatic. You do not switch them on, except for the MCP server, which depends on a page action covered below.

OutputURLWhat it holds
Markdown pagepage URL + .md; site root + /.mdOne page as Markdown, without the surrounding HTML
llms.txtsite root + /llms.txtAn index listing every published page with a Markdown version
llms-full.txtsite root + /llms-full.txtThe full documentation content in one place
MCP serversite root + /~gitbook/mcpRead-only access to published docs for MCP clients

The 2025 changelog dates them: llms.txt support on 28 January 2025, llms-full.txt and .md pages on 24 June 2025, and the MCP server on 16 September 2025. An August 2025 update made the Markdown output include reusable blocks and OpenAPI blocks like other content.

The root landing page is the one irregular URL. For a site at https://docs.example.com, its Markdown is at https://docs.example.com/.md, not /index.md.

What the Markdown looks like in practice

On GitBook's own docs at the time of writing, three details stood out:

  • The HTML page links to its Markdown twin with <link rel="alternate" type="text/markdown"> in the head, the link relation the llmstxt.org proposal recommends for finding Markdown versions.
  • The HTML URL also returned Markdown when the request sent Accept: text/markdown. That is content negotiation, explained in serving Markdown to AI agents. GitBook's LLM-ready page documents only the .md URLs.
  • The .md response carried x-robots-tag: noindex, so search engines should not index the Markdown copy as a duplicate of the HTML page. The HTML page carried index, follow.

GitBook's Markdown keeps its own block syntax, such as {% hint %} and {% stepper %} tags, and GitBook's own .md responses ended with an "Agent Instructions" section describing an ?ask= query parameter. Expect GitBook-flavoured Markdown, not plain CommonMark.

llms-full.txt can be split into parts

GitBook's own llms-full.txt was large enough to be paginated when we fetched it. Its first lines said it was part 1 of 8, covering pages 1 to 100 of 764, and pointed to the next part at /llms-full.txt/1. A tool that reads only the first response gets only part of a large site, so check the header lines before assuming the file is complete.

How hidden pages are treated

This is the setting most likely to surprise you. GitBook's Pages documentation is explicit:

Hidden pages are only hidden from the published table of contents.

The same page goes on to say a hidden page remains available through the site's MCP server and in llms-full.txt. The LLM-ready page says the same, and the MCP page repeats the MCP part. GitBook does not say whether hidden pages are listed in llms.txt, so check that file too. With Git Sync, a hidden page carries hidden: true in its front matter.

So Hide page is a navigation setting, not a privacy setting. It is useful for landing pages, redirect targets or pages you link to from elsewhere. It is the wrong tool for an unreleased feature, an internal runbook or a partner-only price list.

You wantHide page doesWhat GitBook documents instead
Keep a page out of the sidebarYes
Keep a page out of llms-full.txtNoDo not publish it on this site
Keep a page away from MCP clientsNoAuthenticated access, which the MCP server also enforces
Keep a page out of GoogleNoPage-level external search indexing control
Keep unfinished edits privateOnly the primary content branch is published; change requests stay private until merged

GitBook's MCP page also says the server serves only the latest published version: drafts and unpublished changes stay private.

Search engine indexing settings, site and page

GitBook has two levels of indexing control, both described on its public publishing page.

Site level. A public site is indexed by search engines by default. You can publish it without search engine indexing: it stays reachable by anyone with the link but is not indexed by search engines such as Google. GitBook suggests this for beta docs or user testing.

Page level. There are two separate controls, available on Essential and Ultimate site plans at the time of writing:

  • Internal search indexing decides whether a page appears in GitBook's own site search and Ask AI.
  • External search indexing (web crawlers) decides whether search engines and web crawlers can index the page. GitBook names Google and ChatGPT as examples.

In the GitBook API these appear as two page fields, noIndex (search and ask features) and noRobotsIndex (search engine robots). Both inherit downward: disabling indexing on a parent disables it for every sub-page, and a child cannot turn it back on.

Two limits are worth knowing. First, GitBook does not document how external indexing is switched off, whether through a robots meta tag, an X-Robots-Tag header or robots.txt, so check the published page yourself (see below). Second, it does not document whether a page with external indexing disabled is also removed from llms.txt, llms-full.txt or the MCP server. The indexing settings are described in terms of search engines and crawlers, and the AI outputs in terms of published content. Until GitBook says otherwise, assume an unindexed but published page can still be read through the AI outputs, and fetch them to confirm. Our guide to removing a page from AI search covers the other side: what the AI operators themselves honour.

GitBook's SEO page adds that a sitemap is generated at /sitemap-pages.xml for any published site that is not set to unlisted, and the January 2025 changelog says it lists all indexable pages.

Access control: what actually keeps content out

If content must not reach AI tools, keep it off the public site. GitBook's publishing page offers three audiences: public, private with share links, and authenticated access.

For the MCP server, the MCP page is specific:

  • Public sites, share-link sites with all content exposed, and fully authenticated sites use /~gitbook/mcp. Partially authenticated sites, with some public content, use /~gitbook/mcp/auth.
  • On authenticated sites, the MCP server uses the same authentication, and clients sign in through OAuth with Dynamic Client Registration. Backends listed include Auth0, Azure AD, Okta, AWS Cognito, OIDC and a custom backend with a fallback URL.
  • Sites that rely only on static visitor tokens in request headers are not supported for MCP authentication, and neither are share-link-only sites.
  • GitBook describes the server as read-only access to published docs that never exposes account data, analytics or internal GitBook data.

The MCP server is also the one output you can turn off. It only works while Connect with MCP server is in the Enabled actions list under Site customization > Page actions. Move it to Available actions, or turn off page actions, and /~gitbook/mcp returns 404. GitBook does not document an equivalent switch for .md pages, llms.txt or llms-full.txt.

GitBook's docs do not spell out how .md, llms.txt and llms-full.txt behave on authenticated sites, so test them from a signed-out session after you enable access control.

Custom domains and subdirectories

All four outputs are relative to the published site's root URL, so where they live depends on how you publish:

Published atllms.txtMCP server
yourorg.gitbook.io/productyourorg.gitbook.io/product/llms.txtyourorg.gitbook.io/product/~gitbook/mcp
docs.example.com (custom domain)docs.example.com/llms.txtdocs.example.com/~gitbook/mcp
example.com/docs (custom subdirectory)example.com/docs/llms.txtexample.com/docs/~gitbook/mcp

GitBook's custom domain page uses a CNAME record and notes that on Cloudflare the record must be set to "DNS only", not proxied. A subdirectory setup is documented separately.

Two consequences follow. A tool that only looks for example.com/llms.txt will not find a docs index published at /docs; the llmstxt.org proposal allows files at a subpath, but nothing guarantees a client looks there. Link to the file from your main site if you want it found. And robots.txt only counts at the root of a host, as explained in robots.txt for subdomains and ports: on a subdirectory setup, crawlers read example.com/robots.txt, which you control, not a file under /docs.

How to check each output

Run these against your own site, replacing the base URL. Use a signed-out terminal, not a browser where you are logged in to GitBook.

bash
BASE=https://docs.example.com

# llms.txt and llms-full.txt: expect 200 and text/markdown or text/plain
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' "$BASE/llms.txt"
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' "$BASE/llms-full.txt"

# Is llms-full.txt split into parts?
curl -s "$BASE/llms-full.txt" | head -4

# Markdown version of the landing page and of one page
curl -s "$BASE/.md" | head -5
curl -sI "$BASE/getting-started.md" | grep -iE '^HTTP|content-type|x-robots-tag'

To see whether a page you hid or set to noindex still appears in the AI outputs, search for a phrase that only appears on that page:

bash
curl -s "$BASE/llms.txt" | grep -i "internal-runbook"
curl -s "$BASE/llms-full.txt" | grep -i "phrase only on the hidden page"

If llms-full.txt is paginated, repeat the search on each part.

To confirm how external indexing is applied on a page you excluded, look at both the HTML and the headers:

bash
curl -s "$BASE/private-page" | grep -io '<meta name="robots"[^>]*>'
curl -sI "$BASE/private-page" | grep -i x-robots-tag

The MCP server rejects plain GET requests (GitBook's own returned 405), so test it with a JSON-RPC call:

bash
curl -s -X POST "$BASE/~gitbook/mcp" \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

A list of tools means the server is on. On GitBook's own docs at the time of writing, it offered four tools: searchDocumentation, getPage, askQuestion and sendFeedback. GitBook documents a 404 for a site where the page action is disabled, so if you get one and the URL is right, check that setting. For usage, GitBook's Agents & MCP report under Analyze counts reads through Markdown, llms.txt and MCP, and groups clients that do not identify themselves as Unidentified.

For a comparison with another docs platform's defaults, see the Mintlify guide.

Frequently asked questions

Does GitBook create llms.txt automatically?

Yes. GitBook generates /llms.txt and /llms-full.txt for every published site, and its billing FAQ says every plan includes them. There is no documented setting to edit or disable either file.

Are hidden GitBook pages private?

No. GitBook says hiding a page only removes it from the published table of contents, and hidden pages stay in llms-full.txt and in the site's MCP server. Use authenticated access, or do not publish the page, if it must stay private.

Does turning off search engine indexing hide pages from AI tools?

Not that GitBook documents. The setting controls search engines and web crawlers, and GitBook does not say it removes pages from llms.txt, llms-full.txt or MCP, so fetch those files to check.

Can I turn off the GitBook MCP server?

Yes. Move Connect with MCP server from Enabled actions to Available actions under Site customization > Page actions, or turn page actions off, and the endpoint returns 404.

Will GitBook's .md pages create duplicate content in Google?

On GitBook's own docs at the time of writing, .md responses carried x-robots-tag: noindex while the HTML pages were indexable. Check one of your own .md URLs with curl -sI to confirm the same header.

Check what a crawler sees on your docs

GitBook's AI outputs only matter if crawlers can reach the site at all. Run the free check on an important docs URL to see whether the main AI crawlers can fetch it under your robots.txt and whether the text is in the raw HTML. If your docs span a custom domain, a subdirectory and the main site, an audit reviews them together.

Filed under Platform guides. Topics: llms-txt, documentation, ai-crawlers, gitbook