Webflow robots.txt and AI crawlers: the settings that matter

The Webflow settings that decide what AI crawlers can reach are the robots.txt field and the Traffic control toggles in Site settings > SEO > Indexing, the Staging indexing switch for your webflow.io address, each page's Sitemap indexing toggle, and any JSON-LD you add as custom code. Most of them are changed once and then forgotten, and a single toggle can ask a whole group of crawlers to stay away. This guide covers each setting, how to check it from a terminal, and how to fix it.

The Webflow settings described here follow the Webflow Help Center at the time of writing (November 2026). Menu names change, so match on the setting name if your dashboard looks different.

What AI crawlers see on a published Webflow site

Many AI crawlers read only the HTML your server sends and do not run JavaScript. Vercel and MERJ's December 2024 analysis found that none of the major AI crawlers it measured, including OpenAI's crawlers, ClaudeBot and PerplexityBot, rendered JavaScript at the time. Our guide to whether AI crawlers run JavaScript covers the details.

Content you build in the Designer, including the items a Collection list shows, is normally part of the HTML Webflow publishes rather than assembled in the browser. Webflow does not publish a rendering specification, though, so treat this as something to confirm on your own site, not a guarantee.

Content is most likely to go missing when it comes from somewhere other than the Designer:

• Third-party scripts that load CMS items, such as "load more", filtering or infinite-scroll libraries that fetch extra items after the page loads. Usually only the first batch is in the HTML.
• Custom code embeds that build text with JavaScript, such as review widgets, pricing calculators or tables pulled from another service.
• Iframes and video: text inside them is not part of your page.

Check it: fetch a page the way a crawler would and search for a sentence you can see in the browser.

curl -sL -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://www.example.com/blog/some-post | grep -c "a sentence from the page"

The user-agent string is OpenAI's documented GPTBot example; the version number changes over time. A count of 0 means that sentence is not in the raw HTML. Seeing your page as an AI crawler with curl shows more checks along these lines.

Where Webflow robots.txt rules live

Webflow's robots.txt is edited in Site settings > SEO > Indexing, according to Webflow's robots.txt guide. Changes take effect only after you save and publish. Webflow adds a Sitemap: line pointing at your sitemap by default, and a Remove sitemap.xml from robots.txt toggle takes it out.

A file that leaves AI search crawlers alone but opts out of three common training tokens looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /search

The first three groups opt out of training; every other crawler, including OAI-SearchBot, Claude-SearchBot and PerplexityBot, follows the * group.

Google-Extended is a robots.txt token, not a separate crawler: it controls whether Google may use your content for Gemini training and grounding, and Google says it does not affect Google Search. AI Overviews and AI Mode follow your Googlebot rules instead, per Google's AI features guide.

Two rules catch people out. First, under RFC 9309 a crawler that finds a group naming it follows that group and ignores the * group, so a User-agent: GPTBot group replaces your * rules for GPTBot rather than adding to them. Second, robots.txt applies only to the exact host it is served from, so www.example.com and example.webflow.io each have their own file. The robots.txt guide for AI crawlers explains groups and precedence, and allowing AI search while blocking training lists which tokens control which use.

After publishing, allow for caching. OpenAI says its search systems can take about 24 hours to adjust after a robots.txt change, per its crawler documentation, and Perplexity says changes may take up to 24 hours.

What the Traffic control toggles actually block

Webflow also has Traffic control toggles in Site settings > SEO > Indexing. Each one generates robots.txt rules for a group of known crawlers, as Webflow's indexing guide describes: one group of search engine crawlers and one of AI bots. Both groups are allowed by default. The labels below describe the groups; the exact wording in your dashboard may differ.

OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, although they can still appear as navigational links.

The AI bots group mixes three kinds of crawler. Some collect training data (GPTBot, ClaudeBot, CCBot). PerplexityBot is the crawler Perplexity uses to surface and link sites in its search results, per Perplexity's crawler documentation. Claude-User fetches pages when a Claude user asks a question, and Anthropic says disabling it may reduce your visibility in user-directed web search, per its crawler help article.

If your goal is "no training, yes AI search", leave both groups allowed and write named groups for the training tokens yourself, as in the example above. Keep in mind that a Disallow is a request, not a lock: OpenAI says robots.txt rules "may not apply" to ChatGPT-User because a user started the fetch, and Perplexity says Perplexity-User generally ignores robots.txt.

Check it: read the published file and look for groups you did not write.

curl -s https://www.example.com/robots.txt

Content-Signal: an optional usage header

The same Indexing settings let you send a Content-Signal HTTP header that states how your content may be used:

Content-Signal: ai-train=no, search=yes, ai-input=no

Webflow notes the header is based on a proposed extension to RFC 9309 and is not yet an accepted standard. Like robots.txt, it is a request that crawlers may or may not honour, so treat it as an extra signal rather than a replacement. Before setting ai-input=no, decide whether you want your pages used for real-time AI answers: in Cloudflare's Content Signals definitions, ai-input covers feeding content into AI models for answers such as retrieval and grounding.

Keep the webflow.io staging subdomain out of search

Every Webflow site has a staging address on webflow.io. To stop search engines indexing it, set Staging indexing to off in Site settings > SEO > Indexing and publish. Webflow then serves a separate robots.txt on the webflow.io subdomain only, leaving your custom domain untouched. The setting needs a Site plan or a paid Workspace, and older guides call it "Disable Webflow subdomain indexing".

Check it: compare the two files.

curl -s https://example.webflow.io/robots.txt
curl -s https://www.example.com/robots.txt

The staging file should block crawlers; the live file should not. If the live file contains a blanket Disallow: /, staging rules have reached production. The guide to staging settings that hide live sites covers that problem and the other launch-day leftovers.

Noindex, canonicals and redirects

Noindex. Webflow says not to use robots.txt to remove pages that are already indexed. Use the page-level Sitemap indexing toggle instead: turning it off adds a robots noindex meta tag and removes the page from the auto-generated sitemap. The toggle depends on the auto-generated sitemap being turned on. A crawler has to fetch the page to see that tag, which a robots.txt Disallow would prevent.

Canonicals. Set Global canonical tag URL in Site settings > SEO to your live base URL, such as https://www.example.com, with the protocol, matching www and no trailing slash, per Webflow's canonical tags guide. Webflow combines it with each page's slug, and a trailing slash would produce double slashes in every subpage canonical. Avoid adding canonicals a second way, for example through Google Tag Manager, because search engines may ignore duplicate canonical tags. Publish after changing it.

Redirects. Add 301 redirects in Site settings > Publishing > 301 redirects, per Webflow's redirects guide. They apply to the published site, so publish after adding them. Redirect deleted or renamed pages, so crawlers that saved the old URL reach the new one.

curl -sL https://www.example.com/pricing | grep -ioE '<meta[^>]*robots[^>]*>|<link[^>]*canonical[^>]*>'

Add JSON-LD with custom code

Webflow has no structured data panel, so JSON-LD goes in custom code, per Webflow's custom code guide:

• Site-wide: Site settings > Custom code > Head code, for markup such as Organization that belongs on every page.
• One page: the head code field in Page settings > Custom code. Webflow says head and body code fields hold up to 50,000 characters each.
• CMS items: open the Collection template page's settings and add the code to its custom code field. It applies to every item page, and the + Add Field button inserts that item's values.

A BlogPosting block on a blog Collection template might look like this, where the bracketed names stand for fields inserted with Add Field:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "[Name]",
  "description": "[Summary]",
  "image": "[Main image]",
  "datePublished": "[Published date]",
  "author": { "@type": "Person", "name": "[Author name]" },
  "mainEntityOfPage": "https://www.example.com/blog/[Slug]"
}
</script>

Three cautions. Custom code appears only on the published site, so validate the live URL, not the Designer preview. A CMS value that contains a double quote or a line break can break the JSON, so test items with long or punctuated titles in the Schema Markup Validator. And schema.org expects dates in ISO 8601 form, such as 2026-11-08, so check what the date field actually outputs on the live page. Our guide to structured data for AI search covers which types are worth adding; Google, for example, says no special schema.org markup is needed to appear in AI Overviews or AI Mode.

Let Webflow generate the sitemap

Turn on Auto-generate sitemap in Site settings > SEO, per Webflow's sitemap guide. Webflow then generates sitemap.xml on your default domain and leaves out pages and CMS items whose Sitemap indexing toggle is off. You can turn auto-generation off and paste a custom sitemap instead, but you then have to keep it current yourself.

curl -s https://www.example.com/sitemap.xml | grep -o '<loc>[^<]*' | head -5

Every URL should use your live domain, not webflow.io. If it does not, check which domain is set as the default.

A quick Webflow check before you publish

1. robots.txt on the live domain has no blanket Disallow: / and no groups you did not mean to add.
2. Traffic control: both groups allowed, unless you have chosen to block one, knowing which crawlers it covers.
3. Staging indexing is off, and example.webflow.io/robots.txt blocks crawlers.
4. Key pages return 200, with no unexpected noindex and one canonical on the live domain.
5. Main text, including CMS content, is in the raw HTML from curl.
6. JSON-LD on the live page parses, with no broken quotes from CMS fields.
7. The sitemap lists live URLs only, and the site has been published after every settings change.

Frequently asked questions

Does Webflow block AI crawlers by default?

No. Webflow's Traffic control toggles for search engine crawlers and AI bots are both allowed by default, so AI crawlers are blocked only if you turn a group off or write rules yourself. Check the published /robots.txt to be sure.

Can I block AI training on Webflow but stay in AI search?

Yes. Leave both Traffic control groups allowed and add named groups such as User-agent: GPTBot, ClaudeBot and Google-Extended with Disallow: / to your robots.txt. Training and search use different tokens, so blocking GPTBot does not block OAI-SearchBot.

Why is my webflow.io site showing up in Google?

Staging indexing is probably on, or the site has not been republished since you turned it off. Set Staging indexing to off in Site settings > SEO > Indexing and publish; it requires a Site plan or a paid Workspace. Pages Google indexed earlier can take a while to drop out.

Do I need llms.txt on a Webflow site?

It is optional. llms.txt is a proposal, and no major AI search operator documents that its crawlers read it on third-party sites, while Google says Search ignores it. Clear HTML, correct robots rules and a sitemap matter more.

Check your Webflow site

Webflow settings are easy to change in the dashboard and easy to forget after a publish, so check what crawlers actually receive. As a next step, run the free Crawlito check on one public page; it needs no account. For a whole site, including every CMS template, a full audit runs deterministic checks across up to 500 pages and returns a PDF with evidence and prioritized fixes.