Skip to content

Ghost robots.txt: block AI training crawlers, allow AI search

Ghost documents no robots.txt editor in Admin. Add a robots.txt file to your theme to block AI training crawlers and keep AI search open, then verify the live file.

By Crawlito · 8 min read

In short

  • Ghost's help article describes one way to change robots.txt: save a robots.txt file in the root of your theme, install that theme version and activate it, after which it replaces Ghost's default file.
  • On Ghost(Pro), installing a custom theme requires a Creator plan or higher, so lower plans have no documented way to change the file.
  • A custom robots.txt replaces Ghost's default entirely, so copy your live file first and keep its Disallow lines and its Sitemap line, written with your full site URL.
  • At the time of writing, Ghost's demo site serves a default file that names no AI crawler, so GPTBot, ClaudeBot and the AI search crawlers all follow the User-agent: * group.
  • Blocking GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot opts out of training without touching OAI-SearchBot, Googlebot or other search crawlers, but verify the file after you activate the theme.
On this page

To block AI training crawlers on a Ghost site, add a file named robots.txt to the root of your theme, install that theme version and activate it. Once the theme is active, its robots.txt replaces Ghost's default file. Ghost's help article on modifying robots.txt describes only this method, and we found no robots.txt editor documented in Admin. The file you write can disallow the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) and leave search crawlers alone.

The mechanics are short. The risks are in what the replacement file leaves out, in the plan you are on, and in a few problems Ghost users have reported. Facts below come from Ghost's and the crawler operators' documentation, plus Ghost's own demo site and its forum, all read on 11 November 2026. Forum posts are user reports, and the text says so where it matters.

How does Ghost serve robots.txt by default?

Ghost includes a default robots.txt. Ghost's theme structure docs list robots.txt as an optional theme file and say a theme's copy overrides the default. If your theme has none, the default is served.

Ghost does not document the default file's contents in either place. Here is what Ghost's own demo site, demo.ghost.io, served when we fetched it on 11 November 2026:

User-agent: *
Sitemap: https://demo.ghost.io/sitemap.xml
Disallow: /ghost/
Disallow: /email/
Disallow: /members/api/comments/counts/
Disallow: /r/
Disallow: /webmentions/receive/
Disallow: /.ghost/analytics/api/

Three things follow from it. It names no AI crawler, so GPTBot, OAI-SearchBot, ClaudeBot and every other AI crawler fall under User-agent: *, which only blocks the listed paths. It points to the sitemap. And it blocks Ghost's admin and internal endpoints, which you want to keep blocked. An older forum thread mentions a /p/ disallow, which a Ghost staff reply says keeps draft URLs out of search engines. It is absent from the demo file, so the default can differ between sites and versions. Treat the demo file as an example and copy the file your own site serves.

What is the official method to change it?

Ghost's help article describes this sequence:

  1. Write your robots.txt and save it to the root of your theme, next to index.hbs and package.json, not in a subfolder.
  2. Install the new version of the theme.
  3. Activate it. The theme's robots.txt is then used in place of Ghost's default.

The article assumes you are comfortable writing a robots.txt and installing a theme version. It gives no syntax, and it does not say whether Handlebars or other templating works inside the file. Write plain text and put your full site URL in the Sitemap: line instead of relying on a variable.

Keep the file name lowercase and the location at the top level of the host. RFC 9309 says robots.txt is served from /robots.txt in the top-level path, and its rules apply to the scheme and host that serve it. Our guide to robots.txt for AI crawlers covers how crawlers choose a group.

Ghost(Pro) plan limits and self-hosted differences

On Ghost(Pro), the help article says installing a custom theme requires a Creator plan or higher. It documents no other way to change robots.txt, so on a lower plan we found no documented route short of upgrading.

If you self-host, the help article does not say the theme method is limited to Ghost(Pro), and the theme docs list robots.txt as an optional theme file without any hosting condition, so the same method should apply. Because you control the server, you could instead serve /robots.txt from your reverse proxy so Ghost never handles it. One forum user suggested this in a thread about private mode. That is a user suggestion, not Ghost's guidance, and the trade-off is that Ghost's own file is bypassed, so you own the contents and every update.

A starter robots.txt that blocks training and keeps AI search open

This file disallows the five training tokens and keeps Ghost's default path rules in the * group. Search crawlers are not named, so they follow *. Replace the sitemap URL, and replace the * lines with whatever your own live file contains.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /ghost/
Disallow: /email/
Disallow: /members/api/comments/counts/
Disallow: /r/
Disallow: /webmentions/receive/
Disallow: /.ghost/analytics/api/

Sitemap: https://www.example.com/sitemap.xml

What each training line does, according to the operators:

  • GPTBot. OpenAI says disallowing GPTBot indicates your content should not be used to train its generative AI foundation models. It also says each of its settings is independent, so OAI-SearchBot can still surface your pages in ChatGPT search.
  • ClaudeBot. Anthropic's crawler article says restricting it signals that your future material should be excluded from training datasets.
  • Google-Extended. Google says it has no separate user agent and works only as a robots.txt token. It controls whether content Google crawls is used for Gemini training and grounding, and Google says it does not affect inclusion in Google Search. See Google's common crawlers page.
  • Applebot-Extended. Apple says it does not crawl and that pages which disallow it can still appear in Apple search.
  • CCBot. Common Crawl's CCBot page gives User-agent: CCBot with Disallow: / as the rule to block it.

Do not add search crawlers to the blocked group. If you give OAI-SearchBot or Claude-SearchBot a group of its own, it ignores the * group, so you would have to repeat the Disallow lines inside it. Leaving them unnamed avoids that. Do not add ChatGPT-User as a way to opt out of training either: OpenAI says robots.txt rules may not apply to it because users trigger its visits. For the reasoning behind the split, see how to allow AI search but block AI training.

robots.txt is a request that well-behaved crawlers choose to honor, not access control. Anthropic says its bots honor it and OpenAI says robots.txt applies to GPTBot and OAI-SearchBot, but operators add new tokens over time, so review the file against your server logs now and then.

Common problems: theme upload, private mode and forum-reported breakage

The replacement drops Ghost's default lines. The custom file replaces the default instead of extending it. If you write only the AI groups, the admin and internal paths are no longer disallowed and the sitemap line is gone. Start from your live file.

Editing the theme in Admin reset the theme settings. An August 2026 forum thread describes a Ghost(Pro) user who added a robots.txt to the stock Source theme and found posts missing from the home page. A reply in the thread explains that Source and Casper cannot be edited directly, so the editor creates a renamed copy that starts with default settings, and says a fix that copies current settings was due in a later release. The same reply says manual uploads are unchanged. The thread does not establish that the robots.txt file itself caused the problem. Before you upload, write down your Design settings so you can re-enter them, and check the home page right after you activate.

Changes do not show immediately. In a forum thread started in 2020, one user reports an override taking roughly 30 to 60 minutes to appear, and a Ghost staff reply in 2022 says you may need to wait for the cache to clear because robots.txt is rarely updated but highly requested. The demo site's robots.txt response carried cache-control: public, max-age=3600 when we fetched it, which is one hour. RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable, and OpenAI says its systems can take about 24 hours to adjust after a change.

Private mode. In a forum thread about headless sites, users report that turning on private mode replaces the theme's custom robots.txt with a rule that blocks everything. Ghost's help article does not mention this. If your site is in private mode, check the file it serves before you debug anything else.

Sitemap and canonical checks on a Ghost site

Ghost's demo site serves /sitemap.xml as an index that points to separate sitemaps for pages, posts, authors and tags. The default robots.txt references it, and your custom file needs to as well. The sitemaps protocol says the Sitemap: line takes a full URL and is independent of any user-agent group, so its position in the file does not matter.

After you activate the theme, check three things:

  1. /sitemap.xml returns 200 on the host you want crawled, and the child sitemaps list the posts you expect. Our guide to XML sitemaps for AI discovery covers what belongs in one.
  2. Your home page and a post page each have a <link rel="canonical"> pointing at the public URL you want indexed. The demo site's home page head includes one. Canonical tags and duplicate URLs explains what to look for.
  3. Your RSS link still resolves. The demo page's head also advertises /rss/, and RSS feeds for content discovery describes how to keep a feed accurate.

How to verify the live file after activating the theme

Fetch the file from the public site, not from your theme folder. Check the status, content type and caching headers first.

bash
curl -sD - -o /dev/null https://www.example.com/robots.txt

You want 200 and content-type: text/plain. Under RFC 9309 a 4xx means crawlers treat the file as unavailable and may fetch anything, while a 5xx means they must assume everything is disallowed. Then read the body.

bash
curl -s https://www.example.com/robots.txt

Confirm the five training groups are present, that the * group does not contain a bare Disallow: /, and that the Sitemap: line has your real URL. Repeat for every hostname that serves pages, such as the apex domain and www, because each host needs its own file. To test how a parser reads each token, follow how to test robots.txt against each AI crawler. If the old file still appears, wait out the cache, confirm the new theme is the active one, and check that robots.txt sits at the theme root.

Frequently asked questions

Can I edit robots.txt from Ghost Admin?

Ghost's help article describes only the theme method, and the theme docs list robots.txt as an optional theme file. We found no Admin editor documented at the time of writing.

No. OpenAI says GPTBot and OAI-SearchBot settings are independent, and that sites which opt out of OAI-SearchBot are the ones not shown in ChatGPT search answers.

Will blocking Google-Extended affect my Google Search results?

Google says Google-Extended does not impact a site's inclusion in Google Search. It controls whether content is used for Gemini training and grounding.

Do I need a paid Ghost(Pro) plan to change robots.txt?

Ghost says installing a custom theme on Ghost(Pro) requires a Creator plan or higher. The help article documents no other method for lower plans.

Check your own file

The free check reads one page the way AI crawlers do, with no account needed, including the robots.txt rules for each named AI crawler. It runs from Crawlito's servers, so it shows what your published file says, not what a specific operator's network sees. For a whole site, an audit covers up to 500 pages with evidence and prioritized fixes.

Filed under Platform guides. Topics: ghost, robots-txt, ai-crawlers, sitemaps