Skip to content

Beehiiv AI crawlers: how AI Crawl Control works

Where beehiiv's AI Crawl Control lives, which AI crawlers you can block, the plan and custom domain it needs, and how to keep AI search while limiting training.

By Crawlito · 8 min read

In short

  • Beehiiv's AI Crawl Control is under Website, Builder, Settings, Site settings, and gives each cataloged crawler its own block toggle, where on means blocked.
  • Beehiiv says the blocks are enforced at the network level through Cloudflare, so a blocked crawler is stopped whether or not it follows robots.txt.
  • At the time of writing the feature needs a Pro or Enterprise plan, the Website Builder and a custom web domain; sites on a beehiiv subdomain are not eligible.
  • The older Discoverable on the web toggle is all or nothing: turning it off updates robots.txt to ask every search engine and AI crawler not to index the site.
  • Beehiiv says blocking is not retroactive and has no effect on email, so it cannot reach copies already collected or newsletters sitting in inboxes.
On this page

Beehiiv's AI Crawl Control lets you block individual AI crawlers from the web version of your newsletter, one toggle per crawler. Beehiiv says it enforces those blocks at the network level through Cloudflare, so a blocked crawler is stopped before it reaches your site, whether or not it reads robots.txt. To allow AI search but limit AI training, block the training crawlers such as GPTBot and ClaudeBot and leave the AI search, AI assistant and search engine crawlers allowed. The feature needs a Pro or Enterprise plan and a custom web domain at the time of writing.

Facts about beehiiv below come from its help article on controlling which AI crawlers access your content, last updated 1 October 2026, and its SEO settings article, both read on 15 November 2026. Beehiiv changes its interface, so treat labels and menu paths as "at the time of writing". If you are comparing platforms, the Substack setting is a robots.txt signal and Ghost needs a theme file; beehiiv works differently from both.

Where is AI Crawl Control in beehiiv?

Beehiiv's article gives this path:

  1. Open Website, then Builder.
  2. Click the Settings icon and open Site settings.
  3. Choose AI Crawl Control.

The same Site settings panel has an SEO & sharing section, which holds the older Discoverable on the web toggle. The two controls do different things, covered below.

Who can use it

Beehiiv lists three conditions:

RequirementWhat beehiiv says
PlanPro or Enterprise
SiteThe Website Builder
DomainA custom web domain; sites on the default beehiiv subdomain are not eligible

The setting applies only to that custom web domain. Beehiiv says it does not cover your email sending domain, branded link domains or other domain types. If your posts are still on a beehiiv subdomain, the only crawler control beehiiv's articles describe for you is Discoverable on the web.

How the per-crawler toggles work

Every cataloged crawler has its own toggle. Beehiiv's wording is easy to misread: a toggle that is on means the crawler is blocked. Access is allowed by default.

When you switch a toggle on, a confirmation window opens and you click Block crawler to proceed. For crawlers that carry a search visibility warning, that window also reminds you that blocking will reduce your visibility in search results; the article does not list which crawlers carry it. Settings save automatically and, according to beehiiv, take effect as Cloudflare processes the update, typically within a short period after saving. Beehiiv gives no exact time.

The crawler catalog

Beehiiv tracks 22 crawlers in four categories, and says it maintains the catalog and adds crawlers as they appear:

Beehiiv categoryCrawlers
AI model training (10)GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot, Diffbot, Cohere AI, AI2Bot, PanguBot
AI search (3)OAI-SearchBot, PerplexityBot, Claude-SearchBot
AI assistants (5)ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot, MistralAI-User
Search engines (4)Googlebot, Bingbot, Applebot, DuckDuckBot

Two details are worth knowing. Beehiiv notes that the search engine category includes crawlers with an AI component, giving Google's extended crawler as the example. Its catalog also says blocking Bingbot removes your site from Bing and Copilot, and that Applebot powers Siri, Spotlight suggestions and Apple's AI features. But Google documents Google-Extended as a robots.txt token with no separate HTTP user agent string, and it does not appear as a separate toggle. Applebot-Extended is not listed either. So on beehiiv you cannot opt out of Gemini or Apple Intelligence training separately from Google or Apple search through this panel. Also, a crawler that is not in the catalog does not appear in the settings at all, so you cannot block it here. Beehiiv says platform-level protections for known malicious or abusive bots apply regardless.

How it differs from Discoverable on the web

The two settings work through different mechanisms, and that is the main thing to understand before you change either.

AI Crawl ControlDiscoverable on the web
LocationSite settings, AI Crawl ControlSite settings, SEO & sharing
ScopeOne crawler at a timeEvery search engine and AI crawler
MechanismNetwork-level block through CloudflareUpdates your site's robots.txt
Depends on the bot complyingNo, according to beehiivYes
RequirementsPro or Enterprise, custom web domainBeehiiv's SEO article is marked for all plans

Discoverable on the web is on by default. Beehiiv says turning it off updates robots.txt to ask all search engines and AI crawlers not to index your content. Beehiiv calls this a convention with no technical enforcement, so it works only on crawlers that honor robots.txt. It is also all or nothing: it asks Google and Bing to stop indexing your site as well as AI tools. Beehiiv's SEO article adds that turning it off does not password-protect the site, so anyone with a direct URL can still read it.

Discoverability also stacks. Each page has a Search engine indexing setting in Page settings, and beehiiv says a page is excluded if either the site-level or the page-level toggle is off. A page cannot override a site-wide off.

Beehiiv's articles do not say that AI Crawl Control writes anything to robots.txt, and we found no documented way to edit robots.txt yourself on beehiiv. A crawler blocked by AI Crawl Control therefore may still see a robots.txt that allows it, and simply be refused at the edge. For how Cloudflare's own version of this feature works when you run the zone yourself, see the Cloudflare AI crawler settings guide.

Allow AI search, block AI training: a beehiiv setup

The operators document training, search and user-triggered fetching as separate jobs. OpenAI's crawler page says GPTBot settings are independent of OAI-SearchBot, and that sites which opt out of OAI-SearchBot will not appear in ChatGPT search answers, though they can still show as navigational links. Anthropic's crawler article says blocking ClaudeBot signals that future material should be excluded from training, while blocking Claude-SearchBot or Claude-User may reduce your visibility in Claude's search results. The training, search and user crawlers guide explains the split in full.

Translated into beehiiv's toggles:

CategoryToggle positionWhy
AI model trainingOn (blocked) for the crawlers you object toBeehiiv says blocking training crawlers has no SEO impact
AI searchOff (allowed)OpenAI and Anthropic say blocking their search crawlers keeps you out of, or less visible in, their search results
AI assistantsOff (allowed)Beehiiv says blocking them prevents assistants from surfacing your content in responses
Search enginesOff (allowed)Beehiiv warns that blocking them reduces visibility immediately and removes content over time

Some training crawlers belong to companies that also run other products, such as Amazonbot and Meta-ExternalAgent. Check each operator's documentation before you block one, because beehiiv's category label is its own summary. The same policy written as robots.txt, for a site where you control the file, is in allow AI search, block AI training.

Allowing a crawler removes a technical barrier. It does not mean an assistant will quote your posts, which depends on many things outside your site.

What AI Crawl Control cannot do

Beehiiv's own article sets out most of these limits:

  • It is not retroactive. Blocking does not remove data already collected. Beehiiv says removal requires contacting the AI company.
  • It does not touch email. Beehiiv says it has no effect on deliverability, and it applies only to the web domain. Anything a subscriber forwards, pastes into an assistant or publishes elsewhere is outside its reach.
  • It covers only cataloged crawlers. A new or unnamed crawler is not in the panel.
  • It does not change what allowed bots see. Beehiiv points to structured data (JSON-LD) in AEO settings for that, and says it generates an LLMs.txt file automatically; neither controls access.
  • It does not cover your beehiiv subdomain, only the connected custom web domain.

How to check your settings from outside

Start in the dashboard. The AI Crawl Control page shows crawler requests, blocked requests, AI crawler share and unique crawlers, with a crawler activity chart of served and blocked requests, a list of top crawled pages and a breakdown of response status codes. You can filter by crawler and time range; the default view is all crawlers over the last 4 weeks. After you block a crawler, you would expect its later requests to show as blocked. Beehiiv says the dashboard updates in nearly real time once crawler traffic arrives on the custom domain, and that you may see fewer than 22 unique crawlers if not all of them have visited.

Then read what your domain publishes:

bash
curl -s https://www.example.com/robots.txt

If Discoverable on the web is on, the file should not disallow everything for User-agent: *. If it does, look at that toggle first: beehiiv's articles tie robots.txt changes to Discoverable on the web, not to AI Crawl Control.

You can also send a request with a crawler's user agent string. This is the example GPTBot string from OpenAI's documentation, which notes the version number may change:

bash
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://www.example.com/p/your-post

Read the result with care. Beehiiv does not document how Cloudflare identifies each crawler or what status a blocked request receives. A request from your laptop that only borrows a user agent is not the real crawler, so a 200 does not prove GPTBot gets through and a 403 does not prove the block is what stopped it. Compare it with the same request using a normal browser user agent, and treat the dashboard as the record of what real crawlers received. Our guide to seeing your page as an AI crawler with curl covers the method and its limits.

Frequently asked questions

No. OpenAI says GPTBot and OAI-SearchBot settings are independent, and OAI-SearchBot has its own toggle in beehiiv's AI search category. Leave that one off if you want ChatGPT search to be able to read your posts.

Can I use AI Crawl Control on a beehiiv subdomain?

Not at the time of writing. Beehiiv says it needs a Pro or Enterprise plan, the Website Builder and a custom web domain, and that sites on the default beehiiv subdomain are not eligible.

Does turning off Discoverable on the web block AI crawlers?

It asks them to stay away by updating robots.txt, which only works for crawlers that comply. It also asks every search engine to stop indexing your site, so it is not a way to block only AI training.

Will blocking a crawler delete what it already collected?

No. Beehiiv says blocking is not retroactive and that removal of existing data requires contacting the AI company directly.

Check what crawlers can reach

After you change your toggles, run the free check on one post URL to see what your domain publishes to an outside request; it is a single-page check with no account needed. Like the curl test, it cannot show what Cloudflare decided for each real crawler on beehiiv's network, so keep the dashboard as your record of that.

Filed under Platform guides. Topics: beehiiv, ai-crawlers, bot-protection, robots-txt, ai-training