Discourse robots.txt and crawler settings for AI bots
How Discourse robots.txt, the blocked, allowed and slow-down crawler user agent settings and the crawler view treat GPTBot, ClaudeBot and AI search bots.
By Crawlito · 9 min read
In short
- Discourse generates robots.txt from its site settings, and an admin can replace it entirely at /admin/customize/robots.
- A user agent listed in blocked crawler user agents gets a Disallow: / group in robots.txt and a 403 response from the app; robots.txt and llms.txt stay reachable.
- Setting allowed crawler user agents turns on a strict allowlist: every detected crawler not on it, Googlebot included, is disallowed in robots.txt and answered with 403.
- By default Discourse slows gptbot, claudebot, anthropic-ai and brightbot to one request per 60 seconds per matched name, answering extra requests with 429 and Retry-After.
- Crawlers get a separate server-rendered crawler layout with full topic text; you can see it by adding ?print to a URL or by sending a Discourse-Render: crawler header with curl.
On this page
Discourse controls AI crawlers through three site settings: blocked crawler user agents (block), slow down crawler user agents (rate limit) and allowed crawler user agents (a strict allowlist that blocks everything else). It writes the matching rules into its generated robots.txt, enforces them in the app with 403 and 429 responses, and serves every detected crawler a server-rendered "crawler view" instead of the JavaScript app. Out of the box, GPTBot and ClaudeBot are slowed rather than blocked, and AI search crawlers such as OAI-SearchBot are not touched.
This guide covers what each setting does according to Discourse's documentation and source code, the one setting that can quietly block Google, and how to test each crawler with curl. For the markup Discourse adds to topic pages, see QAPage vs DiscussionForumPosting; for rate limiting in general, see rate limiting AI crawlers.
What the default Discourse robots.txt contains
Discourse builds /robots.txt on request from its settings, in the robots.txt controller. With default settings it has three kinds of groups:
- One
Disallow: /group for each name inblocked crawler user agents. The defaults aremauibot,semrushbot,ahrefsbot,blexbotandseo spider. - A
User-agent: *group that disallows admin, auth, email, session and API key paths, plus/badges,/my,/search,/g, tag list views and RSS feeds. - A
User-agent: Googlebotgroup with only the core paths, so Googlebot can still fetch RSS feeds and the other paths in the second list.
A Sitemap: line pointing at /sitemap.xml is added when the sitemap is enabled and the site does not require login. With default settings the generated file looks like this (two comment lines at the top and the repeated blocked groups are shortened here):
User-agent: mauibot
Disallow: /
User-agent: semrushbot
Disallow: /
# ...one group each for ahrefsbot, blexbot and seo spider
User-agent: *
Disallow: /admin/
Disallow: /auth/
Disallow: /assets/js/browser-update*.js
Disallow: /email/
Disallow: /session
Disallow: /user-api-key
Disallow: /*?api_key*
Disallow: /*?*api_key*
Disallow: /badges
Disallow: /my
Disallow: /search
Disallow: /tag/*/l
Disallow: /g
Disallow: /t/*/*.rss
Disallow: /c/*.rss
User-agent: Googlebot
Disallow: /admin/
Disallow: /auth/
Disallow: /assets/js/browser-update*.js
Disallow: /email/
Disallow: /session
Disallow: /user-api-key
Disallow: /*?api_key*
Disallow: /*?*api_key*
Sitemap: https://forum.example.com/sitemap.xmlNothing in the default file names GPTBot, ClaudeBot, OAI-SearchBot or PerplexityBot, so under RFC 9309 they follow the * group and may crawl topics and categories. Two other cases change the whole file. If allow index in robots txt is turned off, Discourse serves a short file that allows only Googlebot (everything except /uploads/*) and disallows every other crawler, and it adds an X-Robots-Tag noindex header to page responses. Discourse's indexing reference describes this as disallowing all crawlers, but the template in the source keeps the Googlebot group. If Discourse runs in a subfolder, the file at your domain root belongs to your main web server, and Discourse exposes /robots-builder.json with the rules it would have generated.
Overriding robots.txt in the admin panel
Admins can replace the generated file at /admin/customize/robots. A customized file overrides the default entirely: the generated groups, the crawler settings and the sitemap line are no longer written into it. If you edit it, copy the default content first and keep the Sitemap: line yourself. The crawler settings still work at the HTTP level, because the 403 and 429 responses come from middleware that does not read robots.txt; only their robots.txt lines disappear. When an admin views /robots.txt while logged in, Discourse prepends a comment saying the file has been customized; anonymous visitors and crawlers do not see that comment.
Blocked, slowed or allowlisted: what each setting does
All three settings live under Admin > Settings and take a list of user agent fragments. Discourse's guide to controlling web crawlers describes them, and the source shows how they are enforced:
| Setting | Default | robots.txt effect | HTTP effect |
|---|---|---|---|
blocked crawler user agents | mauibot, semrushbot, ahrefsbot, blexbot, seo spider | A Disallow: / group per name | 403 with a one-line refusal |
slow down crawler user agents | gptbot, claudebot, anthropic-ai, brightbot | None | 429 with Retry-After once over the rate |
slow down crawler rate | 60 | None | Seconds between allowed requests per name |
allowed crawler user agents | Empty | A group per allowed name, then User-agent: * with Disallow: / | 403 for every detected crawler not on the list |
Matching is a case-insensitive substring match on the user agent. That has consequences for AI bots, because each operator runs several bots with similar names:
gptbotmatches GPTBot only, notOAI-SearchBotorChatGPT-User, whose full strings OpenAI publishes.claudebotmatches the ClaudeBot token, and the tokensClaude-SearchBotandClaude-Userdo not contain it. Anthropic's crawler article does not publish their full user agent strings, so check your logs that the full strings you see do not containclaudebotelsewhere, for example in a contact address.- A broad fragment such as
claudeoropenaimatches every bot from that operator, including the search and user-triggered ones. OpenAI's bot documentation shows ChatGPT-User's string ending in+https://openai.com/bot, soopenaicatches it too.
The 403 is applied in Discourse's anonymous cache middleware to GET requests from user agents Discourse detects as crawlers. Paths ending in robots.txt and llms.txt are exempt, as are requests carrying an API key, so a blocked crawler can still read your rules. The block does not depend on the crawler obeying robots.txt.
The slow-down rate is per crawler name, not per IP
The slow-down limiter in Discourse's request tracker allows one request per slow down crawler rate seconds for each matched name, keyed on the name rather than the IP address, and only for requests without a login cookie. With the default of 60, all of GPTBot's addresses together get one request per minute, and the rest receive:
HTTP/1.1 429 Too Many Requests
Content-Type: text/plain
Retry-After: 42
Discourse-Rate-Limit-Error-Code: gptbot_crawler_rate_limitThat is a throttle, not a block, and 429 with Retry-After is the standard way to say "come back later". It is also tight: by our arithmetic, at most about 1,440 requests a day per crawler name. The slow-down check runs in the request tracker, ahead of the block, and has no exemption for robots.txt or llms.txt, so a slowed bot's robots.txt fetch uses up its one request and can itself get a 429, which some crawlers treat as an error on robots.txt (see rate limiting AI crawlers). A large forum will take a long time to be fully crawled by a slowed bot. Discourse's guide recommends starting with slow-down rather than a full block and checking the effect over time. The same logic is behind Crawl-delay, except that Discourse enforces it at the server instead of asking politely.
Why an allowlist can block Googlebot
allowed crawler user agents is the setting most likely to cause damage. Discourse's SEO overview warns that setting it blocks every crawler not explicitly listed, including Googlebot, and that it takes precedence over the blocked list. In code, once the allowlist has any value the blocked list is ignored, robots.txt gets a final User-agent: * group with Disallow: /, and every detected crawler whose user agent does not match a listed fragment gets a 403.
If you set it to Googlebot|bingbot, for example, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, DuckDuckBot and user-triggered fetchers such as ChatGPT-User and Perplexity-User are refused. Only use an allowlist if you have written down every search and AI crawler you want, and remember that only the exact value Googlebot gets Googlebot's more permissive path list in the generated file.
For most forums the safer pattern is to leave the allowlist empty and name only what you want to stop:
| Goal | Setting | Example value |
|---|---|---|
| Keep AI search, slow AI training | slow down crawler user agents | defaults plus ccbot |
| Keep AI search, block AI training | blocked crawler user agents | defaults plus gptbot|claudebot|ccbot |
| Block one heavy scraper | blocked crawler user agents | defaults plus its exact token |
Decide per crawler using the difference between training, search and user crawlers. Blocking a training crawler does not remove you from AI search; blocking a search or user-triggered crawler can.
How Discourse decides who gets the crawler view
Discourse is a JavaScript application, and a JavaScript shell with no content is the classic reason crawlers see empty HTML. Discourse avoids it by serving detected crawlers a separate server-rendered layout with the full topic text, topic lists, pagination links and schema.org markup, and by adding X-Discourse-Crawler-View: true to the response. Its SEO overview says the content of the two views is identical. Ordinary browsers with JavaScript disabled get a <noscript> version inside the normal layout.
The detection logic in crawler_detection.rb works in two steps, using hidden site settings:
- If the user agent contains a browser token (
trident,webkit,gecko,chrome,safari,msie,opera,goannaordiscourse), it is a crawler only if it also contains a known bot fragment:bot,spider,crawler,rss,facebook,archive,lighthouse, and named agents includinggptbot,claudebot,perplexity-userandclaude-user. - If it contains no browser token at all, Discourse treats it as a crawler. A plain
curlrequest therefore gets the crawler view.
The published user agents of GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Perplexity-User all pass step 1, because they contain bot (ChatGPT-User through its openai.com/bot link) or are named in the list. Requests with ?print in the URL, or with a Discourse-Render: crawler request header, also get the crawler layout, according to the SEO overview, while JSON and RSS requests never do. Because the crawler view is chosen by user agent, a crawler that presents a normal browser string without any bot token gets the JavaScript app and its <noscript> fallback. Serving bots a different layout with the same content is not the problem described in cloaking; serving them different content would be.
How to test each AI crawler with curl
Test from outside your network, against a real topic URL. Start with robots.txt, then check the status and view each crawler receives:
# 1. What robots.txt says
curl -s https://forum.example.com/robots.txt
# 2. Status and crawler-view header for each bot
for ua in "GPTBot/1.4" "OAI-SearchBot/1.4" "ChatGPT-User/1.0" \
"ClaudeBot/1.0" "Claude-SearchBot/1.0" "PerplexityBot/1.0" "Googlebot/2.1"; do
printf '%-22s ' "$ua"
curl -s -o /dev/null -D - \
-A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; $ua" \
https://forum.example.com/t/some-topic/123 \
| grep -iE '^HTTP/|x-discourse-crawler-view|retry-after' | tr '\n' ' '
echo
done
# 3. The crawler view itself, forced with the documented header
curl -s -H "Discourse-Render: crawler" https://forum.example.com/t/some-topic/123 | head -40Read the results like this:
| Result | Meaning | Fix |
|---|---|---|
200 with x-discourse-crawler-view: true | Allowed, gets the server-rendered view | None |
403 | Matched the blocked list, or missing from the allowlist | Remove the fragment, or add the bot to the allowlist |
429 with Retry-After | Matched the slow-down list and over the rate | Expected for slowed bots; remove it if this bot matters for search |
200 without the header | Not detected as a crawler | Check the user agent string you sent |
Three cautions. A slowed bot can return 200 on the first request and 429 on the next within the window, so run the loop twice. Your test requests share the slowed bot's budget, because the limit is keyed on the name, so testing as GPTBot can briefly cost the real GPTBot its request. And a CDN or firewall in front of Discourse can block a bot before Discourse sees it; if the response does not look like Discourse's, check that layer. To see the crawler view in a browser, add ?print to any topic URL, and compare its text with the normal page.
You can also watch real traffic: the Web Crawler User Agent report at /admin/reports/web_crawlers lists crawler names sorted by pageviews, and the Site Traffic report at /admin/reports/site_traffic breaks pageviews down into logged-in users, anonymous users, crawlers and other sources.
Frequently asked questions
Does Discourse block GPTBot and ClaudeBot by default?
No. At the time of writing, both are in the default slow down crawler user agents list, so they are limited to one request per 60 seconds per name and get 429 beyond that. They are not in robots.txt and are not blocked.
Will blocking ClaudeBot on Discourse also block Claude-SearchBot?
Not by token: Discourse matches substrings, and claudebot is not part of the Claude-SearchBot or Claude-User tokens. Anthropic does not publish their full user agent strings, so confirm in your logs; a broader fragment such as claude would block all three.
Where do I edit robots.txt on Discourse?
Go to /admin/customize/robots. A custom file replaces the generated one completely, including the groups created by the crawler settings and the sitemap line, although the settings' 403 and 429 responses keep working.
Can I add an llms.txt file to Discourse?
Yes. Discourse's SEO overview describes an llms_txt upload setting under Admin > Settings > Security that accepts a .txt or .md file up to 512 KB and serves it at /llms.txt. Blocked crawlers can still fetch it, though slowed crawlers are still rate limited on it.
Check what each crawler gets from your forum
The settings above interact with each other and with any CDN in front of Discourse, so test the result rather than trusting the admin panel. Run the free check on a busy topic page to see whether the main AI crawlers can fetch it and what HTML they receive.
Filed under Platform guides. Topics: robots-txt, ai-crawlers, rate-limiting, rendering, discourse