Skip to content

Pantheon robots.txt: why platform domains block most crawlers

Pantheon robots.txt on platform domains disallows almost every crawler and adds X-Robots-Tag noindex. When each applies, and how to test and go live cleanly.

By Crawlito · 8 min read

In short

  • Pantheon serves its own robots.txt on every pantheonsite.io platform domain and on agency vanity domains, with a User-agent * group that disallows the whole site.
  • Pantheon's edge adds an X-Robots-Tag noindex header to responses served from platform domains, including the Live environment's pantheonsite.io address.
  • A custom domain on Dev or Test removes only the X-Robots-Tag header; your own robots.txt is served only on the Live environment of a paid site with a custom domain.
  • The platform robots.txt allows only a short list of SEO, site search and site auditing tools, so GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot fall into the disallow-all group.
  • To test before launch, Pantheon suggests a custom domain such as seo.example.com on Live or a hosts file entry that points the production domain at Pantheon.
On this page

On Pantheon, a disallow-all robots.txt and an X-Robots-Tag: noindex header on a pantheonsite.io address are platform behaviour, not a bug in your site. Pantheon serves its own robots.txt on platform and vanity domains, and its edge adds the noindex header to platform domain responses. Your Drupal or WordPress robots.txt only takes over on the Live environment of a paid site, requested through a custom domain. Everything else stays blocked for every crawler outside a short list of SEO tools, AI crawlers included.

This post covers exactly what Pantheon serves on each kind of domain, what the platform file means for AI crawlers, how to test crawler access before launch without fighting the platform, and the checks to run on launch day. Facts about Pantheon come from its Bots and Indexing and platform domains documentation, read on 15 November 2026. For the general staging leftovers that hit any host, see how staging settings hide live sites.

What Pantheon serves on each domain

Pantheon issues a platform domain for every environment, in the form dev-site-name.pantheonsite.io, test-site-name.pantheonsite.io, live-site-name.pantheonsite.io and multidev-env-site-name.pantheonsite.io. Agencies can also use vanity domains such as dev-sites.myagency.com. Pantheon says platform domains are intended for development and cannot be used for production. Its platform domains page also lists *.pantheon.io, *.gotpantheon.com and agency domains among the hosts that get the default robots.txt.

Two separate controls are involved, and they do not switch off together:

Domain you requestrobots.txt servedX-Robots-Tag header
Platform domain on Dev, Test or MultidevPantheon's disallow-all filenoindex
Platform domain on Live (live-site-name.pantheonsite.io)Pantheon's disallow-all file, even with a custom domain attachednoindex
Vanity domain (dev-sites.myagency.com)Pantheon's disallow-all fileNot stated in Pantheon's docs
Custom domain on Dev or Test (dev.example.com)Pantheon's disallow-all fileRemoved
Custom domain on Live of a paid site (www.example.com)Your site's own robots.txtNot added (Pantheon documents the header for platform domains only)

The row that catches people is the fourth. Pantheon's docs say adding subdomains such as dev.example.com or test.example.com to Dev or Test "will remove the X-Robots-Tag: noindex header only", while the Pantheon robots.txt is still served. A page there looks indexable in a header check, yet crawlers that respect robots.txt are still told to fetch nothing.

The platform robots.txt, line by line

At the time of writing, Pantheon documents the file it serves on platform and vanity domains as:

# Pantheon's documentation on robots.txt: https://docs.pantheon.io/bots-and-indexing/
User-agent: *
Disallow: /

User-agent: RavenCrawler
User-agent: rogerbot
User-agent: dotbot
User-agent: SemrushBot
User-agent: SiteAuditBot
User-agent: SplitSignalBot
User-agent: PowerMapper
User-agent: Swiftbot
User-agent: lyticsbot
User-agent: Dubbotbot
User-agent: PopeTech-CrawlBot
User-agent: PopeTech-ScanBot
Allow: /

The second group lists the crawlers Pantheon lets in to support pre-launch SEO and site search testing, mostly SEO and site auditing tools. Swiftbot, for example, was added on 1 October 2019 according to a Pantheon release note, to support site search testing before release. The list changes over time, so read the live file rather than relying on this copy.

Under RFC 9309, a crawler looks for a group whose user-agent line matches its product token, case-insensitively, and obeys the * group only when no group names it. No AI crawler appears in the allow group, so all of them fall through to Disallow: /:

CrawlerPurposeOn a Pantheon platform domain
GPTBot, ClaudeBot, CCBotTrainingDisallowed by the * group
OAI-SearchBot, Claude-SearchBot, PerplexityBotAI searchDisallowed by the * group
Googlebot, BingbotSearchDisallowed by the * group
SemrushBot, rogerbot, dotbotSEO toolsAllowed, but pages still carry noindex

That is the intended result for a development address. It also means you cannot use a platform domain to check how AI crawlers will treat your production robots.txt: the only file they will ever see there is Pantheon's. If you want to understand how per-host files work in general, robots.txt across subdomains, protocols and ports explains why each hostname gets its own answer.

Why the noindex header and the disallow work against each other

Google's robots meta tag documentation says robots meta tags and X-Robots-Tag headers "are discovered when a URL is crawled", and that if robots.txt blocks crawling, those indexing rules are not found and are ignored. On a Pantheon platform domain, Googlebot is disallowed, so it never reads the noindex header that Pantheon also sends.

In practice the two layers cover different crawlers. The disallow stops compliant crawlers from fetching pages at all. The header is a backstop for crawlers that do fetch pages: the allowed SEO tools, tools set to ignore robots.txt, and anything else that reaches a page. Google's robots.txt introduction notes that a URL blocked by robots.txt can still appear in results without a description if other pages link to it. A pantheonsite.io URL that has been linked publicly can therefore turn up as a bare listing, and Google cannot see the noindex header or any canonical tag on it. You cannot change the platform file, so keep platform URLs out of public links. If one does appear, Search Console's Removals tool can hide it for about six months, but only for a property you have verified. The X-Robots-Tag noindex guide covers how Search Console reports the header when Google does see it.

How to test crawler access before launch

You have three documented options, and one practical shortcut.

  1. Use a tool on the allow list. Site Auditor by Raven, SEMrush, Moz's rogerbot and dotbot, PowerMapper, Swiftbot, lyticsbot and Dubbotbot can crawl platform domains. Pantheon notes that tools such as Siteimprove and Screaming Frog can be set to ignore robots.txt, and you can ask support to add another tool to the file through a feature request.
  2. Attach a test custom domain to Live. Pantheon suggests connecting a domain such as seo.example.com to the Live environment and testing through it. Because it is a custom domain on Live, it serves your own robots.txt and no platform noindex header, which is the closest match to production. That also makes it a crawlable copy of your production site, so remove it when testing is done.
  3. Point your production domain at Pantheon locally. Pantheon documents a hosts file entry, /etc/hosts on macOS and Linux or C:\Windows\System32\Drivers\etc\hosts on Windows, using the IP addresses Pantheon gives you:
203.0.113.10    example.com
203.0.113.20    www.example.com

For a quick check of one URL, curl's --resolve option does the same thing for a single request without editing system files. Replace the address with the one Pantheon provides for your site:

bash
curl -sI --resolve www.example.com:443:203.0.113.20 https://www.example.com/ \
  | grep -i -E '^(HTTP|x-robots-tag)'
curl -s --resolve www.example.com:443:203.0.113.20 https://www.example.com/robots.txt

Both methods assume the production domain has already been added to the Live environment, because your own robots.txt is served only for custom domains on Live. If HTTPS has not yet been provisioned for that domain, the HTTPS request fails with a certificate error. They show you what a crawler will get after DNS changes, before anyone outside can reach it.

Launch checks for Drupal sites on Pantheon

Drupal's scaffold regenerates web/robots.txt on each composer install, whether you build locally or with Pantheon's Integrated Composer. Pantheon recommends keeping your additions in a separate file, assets/my-robots-additions.txt in its example, and appending it through the file-mapping section of composer.json. The Drupal Composer robots.txt guide walks through the mapping and how to verify the build, so the Pantheon-specific checks are short:

  • .gitignore still excludes /web/robots.txt. Pantheon's Drupal upstream ignores the generated file. If someone committed it, Pantheon says composer runs hit a merge conflict and Integrated Composer builds fail. The fix is git rm --cached web/robots.txt and restoring the ignore line.
  • Your rules appear on the custom domain, not the platform domain. After the Live deploy, fetch https://www.example.com/robots.txt and look for a line you know is in your additions file. Fetching live-site-name.pantheonsite.io/robots.txt will always show Pantheon's file and proves nothing about your build.
  • Canonical and sitemap URLs use the custom domain. If the site was built and reviewed on a platform domain, check that no canonical tag or sitemap entry still uses a pantheonsite.io host, and regenerate the sitemap after launch. Pantheon also notes that the XMLSiteMap option to prefetch URL aliases during generation can exhaust PHP memory and produce a blank page.

Launch checks for WordPress sites on Pantheon

Pantheon warns against ticking Discourage search engines from indexing this site under Settings > Reading on Dev or Test. The platform robots.txt normally overrides what WordPress generates, but Pantheon notes that it does not when the robots.txt URL has a trailing slash. Pantheon's note describes the setting as writing a disallowing robots.txt. According to the WordPress do_robots() reference, core stopped writing Disallow: / for this setting in version 5.3 and uses a robots meta tag instead, which current versions output as noindex, nofollow. The setting is stored in the database, so it reaches Live if that database is ever copied there. Leave it unticked and rely on the platform controls.

If you need custom rules, WordPress builds a virtual robots.txt when no physical file exists, and Pantheon points to the robots_txt filter in a small plugin or your theme's functions.php. A minimal version, which keeps rules inside a group:

php
add_filter( 'robots_txt', function ( $output, $public ) {
    $rules  = "User-agent: *\n";
    $rules .= "Disallow: /wp-admin/\n";
    $rules .= "Allow: /wp-admin/admin-ajax.php\n\n";
    $rules .= "Sitemap: https://www.example.com/wp-sitemap.xml\n";
    return $rules;
}, 10, 2 );

Adjust the sitemap URL to whatever your SEO plugin generates. The WordPress virtual robots.txt guide explains when a physical file, a plugin setting or this filter wins.

On launch day, run the same three requests against the custom domain for either CMS:

bash
curl -sIL https://www.example.com/ | grep -i -E '^(HTTP|x-robots-tag)'
curl -s https://www.example.com/robots.txt | head -20
curl -s https://www.example.com/ | grep -io '<meta[^>]*robots[^>]*>'

You want a 200, no x-robots-tag containing noindex, a robots.txt that is yours rather than Pantheon's comment line and allow list, and no noindex meta tag. Repeat on the bare domain and on a deep page, and remember that crawlers may keep a cached robots.txt for up to 24 hours under RFC 9309.

Frequently asked questions

Why does my Pantheon site show Disallow: / in robots.txt?

You are almost certainly requesting it through a platform domain, a vanity domain, or a custom domain on Dev or Test. Pantheon serves its own disallow-all file on all of those, and your site's robots.txt is used only on the Live environment of a paid site through a custom domain.

Does adding dev.example.com to the Dev environment make it crawlable?

No. Pantheon says a custom domain on Dev or Test removes only the X-Robots-Tag: noindex header, and the platform robots.txt is still served, so compliant crawlers remain disallowed.

Can I let GPTBot or OAI-SearchBot crawl a pantheonsite.io domain?

Not through your own rules, because the platform file is fixed and no AI crawler is in its allow group. To see how AI crawlers will treat your production rules before launch, use a test custom domain on Live or a hosts file entry for your production domain.

Will the Live pantheonsite.io URL stay blocked after launch?

Yes. Pantheon's platform domains documentation says the default robots.txt is still served on Live through the platform domain even when a custom domain is attached, so make sure canonicals and sitemaps point at the custom domain.

Check the live domain after launch

Once DNS points at Pantheon, test the address people and crawlers will actually use, not the pantheonsite.io one. The free check looks at one public page on that domain, with no account needed.

Filed under Platform guides. Topics: robots-txt, noindex, drupal, wordpress, deployment