Docusaurus SEO: robots.txt, sitemap plugin and noIndex
Docusaurus SEO settings in one place: robots.txt in static, sitemap plugin options, url, baseUrl and trailingSlash, the noIndex flag and per-page head tags.
By Crawlito · 8 min read
In short
- Docusaurus has no robots.txt setting: you add the file to the static folder, which is copied unchanged to the root of the build output.
- In Docusaurus 3.10, the sitemap plugin ships with preset-classic, runs only in production builds and writes sitemap.xml at the base URL, with changefreq weekly and priority 0.5 by default.
- The site-wide noIndex option adds a noindex, nofollow robots meta tag to every page and stops the sitemap from being generated.
- Canonical, og:url and hreflang links are built from url, baseUrl and the route, so a wrong url value spreads to every page.
- Content inside BrowserOnly or useEffect is missing from the static HTML, while Tabs render every panel at build time unless you set lazy.
On this page
Docusaurus SEO comes down to five site-level controls: a robots.txt file you place in static/, the sitemap plugin, the url, baseUrl and trailingSlash settings that shape every canonical URL, the global noIndex flag, and head metadata per page. Docusaurus builds static HTML for every route, so most of what crawlers need is already in the raw response. The gaps are content you wrap in client-only components, and settings that are wrong for the host you deploy to.
This post covers those site-level settings, as documented by Docusaurus at the time of writing (version 3.10). If your problem is search results landing on an old docs version, see Docusaurus search results lead to an old docs version. For content and versioning questions that apply to any docs platform, see documentation sites and AI search.
Where Docusaurus puts robots.txt
There is no robots.txt option in docusaurus.config.js. The Docusaurus SEO guide tells you to put the file in static/. The static assets page explains that files in static are copied into the root of the build folder with their directory hierarchy preserved, so static/robots.txt becomes build/robots.txt.
A file that allows every crawler and points to the sitemap looks like this:
User-agent: *
Disallow:
Sitemap: https://docs.example.com/sitemap.xmlAdd groups for specific crawlers below it if you want different rules for them, for example to treat search crawlers and training crawlers differently.
Two details catch people out.
robots.txt only works at the host root. RFC 9309 says the rules must be in a file named /robots.txt in the top-level path of the service. The static assets page notes that with baseUrl: '/subpath/', static files are served under /subpath/. Combining the two: if your docs live at https://example.com/docs-site/, your static/robots.txt is published at /docs-site/robots.txt, where crawlers will not look for it. The file that counts is whatever the main site serves at https://example.com/robots.txt, so the rules for the docs belong there.
robots.txt does not stop indexing. The Docusaurus SEO guide warns that robots.txt does not prevent HTML pages from being indexed, and points to noIndex or page-level robots meta tags instead. A disallowed page can still appear in results as a bare URL, because the crawler never reads the page that would tell it not to. robots.txt vs meta robots explains the difference.
How the Docusaurus sitemap plugin works
The sitemap plugin (@docusaurus/plugin-sitemap) is included in preset-classic, so most sites already have it. It works on the build output, which means it does nothing during docusaurus start. You only get a sitemap from docusaurus build.
Its options, with the documented defaults:
| Option | Default | What it does |
|---|---|---|
lastmod | null | 'date' writes YYYY-MM-DD, 'datetime' writes an ISO 8601 datetime, null leaves it out |
changefreq | 'weekly' | Value for <changefreq>; null removes it |
priority | 0.5 | Value for <priority>; null removes it |
ignorePatterns | [] | Glob patterns for routes to leave out; you may need to include the base URL |
filename | 'sitemap.xml' | Output path relative to the build directory |
createSitemapItems | undefined | Function to change or filter the final list of items |
These defaults are for the 3.10 release. At the time of writing, the unreleased canary version of the same page lists lastmod: 'date' as the default and no longer has changefreq or priority options, so check the page for the version you run after an upgrade.
A configuration that adds dates and drops blog tag, docs tag and blog pagination pages:
// docusaurus.config.js
export default {
presets: [
[
'classic',
{
sitemap: {
lastmod: 'date',
changefreq: null,
priority: null,
ignorePatterns: ['/blog/tags/**', '/docs/tags/**'],
createSitemapItems: async (params) => {
const {defaultCreateSitemapItems, ...rest} = params;
const items = await defaultCreateSitemapItems(rest);
return items.filter((item) => !item.url.includes('/page/'));
},
},
},
],
],
};The patterns are matched against route paths, which is why the plugin page warns that you may need to include the base URL. The documentation example uses /tags/**; on a default setup, tag pages live under /blog/tags/ and /docs/tags/, so name the full path.
Things worth knowing about what ends up in the file:
lastmodis off by default. Even when you turn it on, the plugin page says a<lastmod>only appears for routes that carrysourceFilePathorlastUpdatedAtmetadata. The official docs, blog and pages plugins provide it for file-backed routes; routes from third-party plugins may not. Google's view of the other two tags is blunt: its sitemap documentation says it ignores<priority>and<changefreq>, which is why the example sets them tonull.- Noindex pages are left out. The SEO guide says the plugin skips pages with a
noindexrobots directive, so a page you hide with a meta tag will not also be advertised in the sitemap. - The site-wide
noIndexsetting removes the sitemap completely. Nositemap.xmlis generated at all. - Sitemap URLs follow
trailingSlash. Whatever you set there is what the<loc>values look like. - The file sits at the base URL. With a subpath, it is
/docs-site/sitemap.xml, so give that full address in the root robots.txt and in Search Console or Bing Webmaster Tools.
How url, baseUrl and trailingSlash shape canonical URLs
Docusaurus writes a canonical link, an og:url and, on translated sites, hreflang alternates into every page. You can see this in the raw HTML of the Docusaurus site itself:
<link data-rh="true" rel="canonical" href="https://docusaurus.io/docs/seo" />
<meta data-rh="true" property="og:url" content="https://docusaurus.io/docs/seo" />
<link data-rh="true" rel="alternate" href="https://docusaurus.io/fr/docs/seo" hreflang="fr" />These values are assembled from three settings documented in the config reference and the deployment guide:
| Setting | Documented meaning | What goes wrong |
|---|---|---|
url | The top-level hostname, for example https://my-org.com | A staging host or the wrong www variant lands in every canonical tag |
baseUrl | The path after url, with a trailing slash; default '/' | A missing subpath produces canonicals and sitemap URLs that 404 |
trailingSlash | undefined keeps URLs untouched, true adds slashes, false removes them | The host redirects to one form while canonicals and the sitemap use the other |
trailingSlash also changes the files on disk. With undefined or true, docs/myDoc.md becomes /docs/myDoc/index.html; with false, it becomes /docs/myDoc.html. Hosts treat those differently. The deployment guide says GitHub Pages adds a trailing slash to Docusaurus URLs by default and recommends setting trailingSlash to true or false rather than leaving it undefined. For Netlify, it says Netlify adds trailing slashes by default and recommends disabling the Pretty URLs setting, or adjusting trailingSlash to match if you keep it on.
To check, request a page in both forms and compare with its canonical:
curl -sI https://docs.example.com/docs/intro | grep -iE '^(HTTP|location)'
curl -sI https://docs.example.com/docs/intro/ | grep -iE '^(HTTP|location)'
curl -s https://docs.example.com/docs/intro/ | grep -o '<link[^>]*rel="canonical"[^>]*>'The form that returns 200 should be the one in the canonical tag and the sitemap. If one form redirects to the other, the canonical should name the destination. Trailing slash URLs and redirects covers how to pick a form and keep it consistent.
How to keep pages out of the index
The config reference documents noIndex as a boolean, default false, that adds this tag to every page:
<meta name="robots" content="noindex, nofollow">Use it for preview and staging builds, not production. Because it also disables the sitemap, a production site that inherits it by mistake loses both. One way to guard against that is to drive it from an environment variable set only in preview deployments:
// docusaurus.config.js
export default {
url: 'https://docs.example.com',
baseUrl: '/',
trailingSlash: false,
noIndex: process.env.DOCS_PREVIEW === 'true',
};For a single page, the SEO guide shows a robots meta tag in the page's head. In Markdown or MDX you can write a <head> block directly:
<head>
<meta name="robots" content="noindex" />
</head>In a React page under src/pages, use the Head component from @docusaurus/Head instead. The guide also mentions an X-Robots-Tag: noindex response header as an option on hosts that support custom headers, and notes that GitHub Pages does not. A whole docs version can be excluded with the docs plugin's version-level noIndex, which the old docs version post discusses.
Head metadata per page and site-wide
The SEO guide documents four front matter fields supported by the official docs, blog and pages plugins:
---
title: Configure webhooks
description: Create a webhook endpoint, verify signatures and retry failed deliveries.
keywords: [webhooks, signatures, retries]
image: /img/social/webhooks.png
---title can differ from the page heading. description fills both the meta description and og:description. image is the social card thumbnail. If you leave out description, check what the generated tag contains before assuming it is usable.
Site-wide tags go in two places. themeConfig.metadata adds meta tags to every page, and headTags inserts arbitrary tags such as links:
export default {
headTags: [
{tagName: 'link', attributes: {rel: 'preconnect', href: 'https://cdn.example.com'}},
],
themeConfig: {
metadata: [{name: 'twitter:card', content: 'summary_large_image'}],
},
};The SEO guide also says the blog supports rich results out of the box when you supply publish dates, authors and images, and that you can add JSON-LD with a <script type="application/ld+json"> inside a page's head.
What the static build includes and what it leaves out
The static site generation page explains that Docusaurus renders every route to HTML at build time with React DOM Server, where there is no window or document, and then hydrates that HTML in the browser. Anything that only runs in the browser is therefore missing from the file a crawler downloads:
| Pattern | In the static HTML? |
|---|---|
| Markdown and MDX content, headings, tables, code blocks | Yes |
Canonical, description, og:*, hreflang, front matter metadata | Yes |
<Tabs> panels, including hidden ones | Yes, by default |
<Tabs lazy> | Only the default tab |
Children of <BrowserOnly> | No; children are only rendered on the client |
The true branch of useIsBrowser() | No; it returns false during server rendering |
Data fetched in useEffect | No |
The Tabs documentation states that all tabs are rendered eagerly during the build and search engines can index hidden tabs, and that lazy renders only the default tab. If you show code samples per language, leaving lazy off keeps every language in the raw HTML.
To confirm, build and serve locally, then fetch a page from a second terminal without running JavaScript. docusaurus serve uses port 3000 by default; add your baseUrl to the path if it is not /:
npm run build && npm run serve
# in a second terminal
curl -s http://localhost:3000/docs/intro | grep -c 'a phrase from the page'A count of zero for text you can see in the browser means it is client-only. Raw vs rendered HTML explains why that matters for crawlers that do not run scripts.
Frequently asked questions
Does Docusaurus generate a robots.txt file?
No. You create static/robots.txt yourself, and the build copies it to the root of the output. With a subpath baseUrl, it ends up under that subpath, where crawlers do not read it.
Why is there no sitemap.xml when I run docusaurus start?
The sitemap plugin only runs in production because it works on the build output. Run docusaurus build and check build/sitemap.xml, or docusaurus serve and open /sitemap.xml.
Why did my sitemap disappear?
Check whether noIndex is true in the config used for that build. The plugin documentation says no sitemap is generated when it is set.
Should I set trailingSlash?
Set it explicitly to match how your host serves URLs. Docusaurus recommends true or false for GitHub Pages, and canonical tags and sitemap URLs follow whatever you choose.
Check a built page
After a deploy, run the free check on one of your docs pages to see whether AI crawlers can fetch it and what its raw HTML contains. If you run several docs versions or a subpath setup, a full-site audit checks robots.txt, sitemap, canonicals and indexing directives across a sample of templates.
Filed under Platform guides. Topics: docusaurus, documentation, robots-txt, sitemaps, canonical