HubSpot robots.txt: control AI crawlers on CMS Hub
Edit robots.txt in HubSpot, add rules for AI crawlers, and avoid blocking pages you want AI search to read. Covers per-domain files and the hs-sites noindex.
By Crawlito · 7 min read
In short
- HubSpot's robots.txt editor is in Settings > Content > Pages > SEO & Crawlers, and each connected domain can override the default settings, so a rule added to the default may not reach an overridden domain.
- HubSpot documents User-agent and Disallow in its robots.txt editor and does not mention Allow or Sitemap, so check the live file after every save.
- HubSpot says content on system domains containing hs-sites is always set to noindex and you cannot change it, so test the domain you actually publish on.
- HubSpot says not to combine a robots.txt block with a noindex meta tag on the same content, because a crawler that cannot fetch the page cannot read the tag.
- Disallowing a training crawler such as GPTBot in its own group signals a training opt-out without touching OAI-SearchBot, because OpenAI says each crawler setting is independent.
On this page
- Where to edit robots.txt in HubSpot
- What the editor lets you change
- Add AI crawler groups without blocking search
- Per-domain files and the hs-sites noindex
- Why robots.txt plus noindex on the same page backfires
- Page-level controls: noindex in page settings
- How to verify the live file and test each AI user agent
- Frequently asked questions
- Check what your domain serves
You edit robots.txt in HubSpot under Settings > Content > Pages > SEO & Crawlers, in the Robots.txt section, then save. To add rules for AI crawlers, add a User-agent group for each crawler and a Disallow line for the paths you want to keep it out of. HubSpot's own guide does not mention AI crawlers at all, so this post covers the part it leaves out, and says plainly where HubSpot's documentation stops.
One caveat first. These controls remove or add technical barriers. They do not make an AI assistant cite you, and nothing here promises traffic.
Where to edit robots.txt in HubSpot
HubSpot's robots.txt article gives these steps, which are available on all products and plans at the time of writing:
- Click the settings icon in the top navigation bar.
- In the left sidebar, go to Content > Pages.
- Choose Default settings for all domains, or pick one domain from the dropdown and click Override default settings if needed.
- Open the SEO & Crawlers tab.
- Edit the file in the Robots.txt section.
- Click Save.
Menu labels can change, so treat HubSpot's current article as the reference if your screen differs from these steps.
What the editor lets you change
HubSpot documents two directives in the editor:
- User-agent sets which crawler a rule applies to. It defaults to all crawlers (
*), and you can name specific ones. - Disallow blocks a URL slug, written as
Disallow: /url-slug. HubSpot's example turnswww.hubspot.com/welcomeintoDisallow: /welcome.
HubSpot's article does not mention Allow or Sitemap, and it gives no size limit. It does not say whether you can write several user-agent groups. The field accepts the file's contents, so multiple groups are the normal way to write robots.txt, but treat that as something to confirm in the live file instead of an assumption. Under RFC 9309, matching is by path prefix from the first character, so Disallow: /welcome also covers /welcome-guide. HubSpot's article does not describe that, so write each rule with the prefix effect in mind.
HubSpot presents robots.txt as the method for keeping new content from appearing in search results, and the noindex meta tag as the method for content that search engines have already indexed.
One more documented item: if you use HubSpot's site search module, add HubSpotContentSearchBot as a separate user agent so the search feature can still crawl your pages. Do not disallow your whole site for every agent without checking that.
Add AI crawler groups without blocking search
The safest pattern is to name only the crawlers you want to opt out, and leave everything else alone. OpenAI documents its crawlers separately: GPTBot is for content that may be used to train its models, OAI-SearchBot surfaces sites in ChatGPT search, and OpenAI says each setting is independent. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, so blocking it is a visibility decision, not a training one.
A training opt-out that leaves search crawlers alone looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /Those five are the training tokens we list in How to allow AI search but block AI training, which also explains the search tokens (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and why Google-Extended and Applebot-Extended are opt-out tokens rather than crawlers. Check each operator's documentation before you rely on a token, because lists like this change.
Two traps matter on HubSpot specifically:
- Your existing
*group still governs unnamed crawlers. If the default file already hasDisallowlines underUser-agent: *, OAI-SearchBot and every other search crawler follows them. That is correct if those paths are private, and a problem if you copied a blanket rule. See how robots.txt groups work for why a named group ignores*completely. - Allowing a path inside a blocked folder needs
Allow. Since HubSpot does not documentAllow, test it on a spare path before you depend on it.
robots.txt is a voluntary signal. RFC 9309 says its rules are not a form of access authorization, and operators differ on whether their user-triggered fetchers follow it. OpenAI, for example, says robots.txt rules may not apply to ChatGPT-User. Read GPTBot, OAI-SearchBot and ChatGPT-User explained before you decide what to block.
Per-domain files and the hs-sites noindex
HubSpot serves a robots.txt per domain, and the settings page reflects that. The default settings apply to all connected domains. When a domain has an override, HubSpot says the override replaces the default robots.txt settings for that domain. The practical consequence: if you add AI rules to the default and a domain was overridden earlier, that domain does not get them. Open each domain's override and check.
This also matches how robots.txt works in general: Google's robots.txt specification says rules apply only to the host, protocol and port where the file is hosted. Testing robots.txt for each AI crawler covers checking every host. HubSpot's article says nothing about subdomains, so do not assume a rule on www reaches blog or shop.
Then there is the system domain. HubSpot states that content on HubSpot system domains containing hs-sites is always set to noindex in robots.txt. You cannot change that. If a page works on yourportal.hs-sites.com during a preview and you judge crawler access from there, you are testing a domain that HubSpot restricts. Test the domain you connected and publish on.
Only content on a domain connected to HubSpot can be blocked in HubSpot's robots.txt. Files and pages hosted elsewhere need their own robots.txt.
Why robots.txt plus noindex on the same page backfires
HubSpot's guide to preventing content from appearing in search results says the robots.txt method should not be combined with the noindex method. The reason is mechanical: a crawler reads noindex from the page, and a Disallow stops the crawler fetching the page, so the tag is never seen. HubSpot's troubleshooting list repeats it: if a page keeps showing up, check for a Disallow rule that blocks it.
Pick one control per page:
| Goal | Use | Result |
|---|---|---|
| Keep a page out of listings and let crawlers read the tag | noindex, no Disallow | Crawler fetches the page, sees the tag |
| Stop a crawler requesting a path | Disallow | Crawler does not fetch it; Google says a blocked URL can still appear in results without a description |
| Opt out of training | Named Disallow group | Training crawler stays out; search is unaffected |
robots.txt vs meta robots explains the split in more depth.
Page-level controls: noindex in page settings
For a single page, HubSpot's documented route is a meta tag in the Head HTML field:
<meta name="robots" content="noindex">For website pages, landing pages and blog posts: open the content, click Settings at the top right, choose Advanced, and paste the tag under Additional code snippets in Head HTML. Close the dialog, then click Update or Publish. Knowledge base articles use Settings > Advanced Options instead.
HubSpot's article does not mention X-Robots-Tag headers, so a header is not a documented HubSpot route. It also says the page must be public: a private or password-protected page cannot be crawled, so the tag is not read. For what nosnippet and related directives do, see noindex, nosnippet and X-Robots-Tag.
How to verify the live file and test each AI user agent
Do not trust the editor: fetch what the domain serves.
curl -s https://www.example.com/robots.txt
curl -s -o /dev/null -w "%{http_code}\n" https://www.example.com/robots.txtRun both for every connected domain. Under RFC 9309, a 4xx response means a crawler may access any resource, and a 5xx means it must assume everything is disallowed.
Then confirm the page-level tag reaches the live HTML:
curl -s https://www.example.com/your-page | grep -io '<meta[^>]*robots[^>]*>'Finally, trace which group each token follows. Setting a user agent in curl does not do that, because curl does not read robots.txt. Use a parser against the fetched file, once per token, as described in testing robots.txt for each AI crawler. OpenAI says its search systems can take about 24 hours to adjust after a robots.txt change, so wait before judging the result. Your CDN or firewall can also challenge crawlers whatever robots.txt says.
Frequently asked questions
Does HubSpot's robots.txt editor support Allow and Sitemap?
HubSpot's article documents only User-agent and Disallow, and does not mention Allow or Sitemap. Add them only if you can confirm in the live file that they behave as expected.
Will a Disallow remove pages that are already indexed?
Not reliably. HubSpot describes robots.txt as the method for new content and recommends the noindex meta tag for content that is already indexed. Leave that page crawlable so the tag can be read.
Can I change robots.txt on my hs-sites domain?
No. HubSpot says content on system domains containing hs-sites is always set to noindex in robots.txt. Connect your own domain and manage the rules there.
Does blocking GPTBot hide my HubSpot site from ChatGPT search?
No. GPTBot covers training, while OAI-SearchBot covers ChatGPT search, and OpenAI says each setting is independent. OpenAI says an OAI-SearchBot opt-out is what keeps a site out of ChatGPT search answers.
Check what your domain serves
The free check reads one page the way AI crawlers do, with no account: the robots.txt rules for each named AI crawler, the robots meta tag and whether the text is in the raw HTML. For a whole HubSpot site across several domains, an audit covers up to 500 pages with evidence and prioritized fixes.
Filed under Platform guides. Topics: hubspot, robots-txt, ai-crawlers, meta-robots