Squarespace AI crawlers: the settings to check

Squarespace gives you one switch for AI crawlers: a checkbox called Block known artificial intelligence crawlers under Settings > Crawlers. It is off by default, and turning it on adds rules to your robots.txt that ask 26 named bots, mostly training crawlers, not to crawl your site. It does not name the crawlers behind ChatGPT search, Perplexity, Claude's search or Google Search, so the settings that can hide a Squarespace site from AI search are elsewhere: the search engine checkbox, site availability, page passwords and page-level noindex.

This guide covers each setting, what it changes, and how to check the result from the outside. Squarespace and crawler facts come from their own documentation, read on 8 November 2026.

What the "Block known artificial intelligence crawlers" setting does

Squarespace's help article Request that AI models exclude your site describes the setting. Open the Settings panel, click Crawlers, and check Block known artificial intelligence crawlers. Squarespace then adds requests to your robots.txt asking these user agents not to crawl the site:

AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended,
Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler,
DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image,
GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent,
MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot

The same article lists the limits, and they are worth knowing before you tick the box:

• The box is unchecked by default, so nothing is excluded unless you turn it on.
• robots.txt is a request. Crawlers are not guaranteed to comply.
• It does not remove content that was already collected for training.
• It applies to the whole site. You cannot limit it to specific pages.
• It covers known bots only. Squarespace may add more over time.
• Squarespace warns that excluding AI crawlers might reduce traffic to your site.

Squarespace's article was last updated on 8 January 2026 at the time of writing, so check it again before you rely on any single name.

Which AI crawlers the checkbox blocks, and which it leaves alone

Most of the list is made up of training crawlers and opt-out tokens. The crawlers that fetch pages for AI search answers are mostly absent. The table compares the checkbox with the purposes each operator documents. For background on the three purposes, see training, search and user-triggered crawlers.

Two practical readings follow from the table:

1. Turning the box on is mostly a training opt-out. Your pages stay open to OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. Google says AI Overviews and AI Mode are controlled by Googlebot rules and snippet controls, not by Google-Extended, as its AI features page explains.
2. It is not purely a training opt-out. DuckAssistBot fetches pages for DuckDuckGo's AI-assisted answers that cite sources, so the checkbox also asks it not to fetch your pages for those answers, although DuckDuckGo says opting out does not affect organic rankings. Amazonbot and Meta-ExternalAgent also cover product uses, not only training. If you want training blocked and every search use open, you need finer control than one checkbox; see how to allow AI search but block AI training.

The setting that blocks AI search: "Block Search engine crawlers"

The same Crawlers panel has a second checkbox, Block Search engine crawlers. Squarespace's article Hiding your site from search results says checking it adds Disallow: / to your robots.txt. A whole-site disallow under User-agent: * applies to every crawler that respects robots.txt and has no group of its own, which includes OAI-SearchBot, Claude-SearchBot and PerplexityBot as well as Googlebot. Squarespace does not show the exact output, so after ticking the box, read the file and see which group the line sits in.

Check this box first on any site that was hidden from search while it was being built: a setting chosen before launch is easy to leave on.

How site availability and page passwords affect crawlers

Squarespace's article Controlling who can access your site's pages lists the site-wide availability options and what search engines can do with each:

A crawler that reaches a Password or Private site gets a lock screen, not your content. The same applies to individual pages:

Squarespace also notes that changes do not show in Google until it recrawls the site, which usually takes some time, so a page that was already indexed does not disappear at once.

How page-level noindex works on Squarespace

To keep one page public but out of search results, Squarespace's hiding article gives two methods:

1. Page settings. In the Pages panel, hover over the page, click the gear icon, open the SEO tab and switch on Hide page from search results. This works on all plans and removes the page from the sitemap.
2. Code injection. Add the tag below to the page's header code injection. This is available on Core, Plus, Advanced, Business, Commerce Basic and Commerce Advanced plans, not on Basic or Personal, and the page stays in the sitemap.

<meta name="robots" content="noindex">

The limits matter for AI search planning:

• The page settings toggle is not available for the homepage or for individual collection items. Code injection works on the homepage.
• Hiding a collection page, such as a blog or shop, also hides its items. You cannot exclude a single collection item without excluding the whole collection page.
• On index pages, hiding the index does not hide its sub-pages. Each one needs the tag.
• Neither method hides pages from your own site search. Both affect external search engines only.

A noindex tag only works if the crawler can fetch the page and read it, so do not also block the same page in robots.txt. For how Google, OpenAI and others treat noindex and nosnippet, see noindex, nosnippet and X-Robots-Tag for AI search.

What structured data Squarespace adds by default

Squarespace's SEO overview, How does Squarespace optimize my site for search results?, says it automatically generates structured data for six types: blog post, event, local business, organization, product and website. You cannot edit or remove it. The same page says Squarespace generates meta tags, uses clean static URLs, redirects connected domains to your primary domain, and keeps a sitemap updated automatically.

Not every type appears on every page. On one Squarespace homepage we fetched at the time of writing, the only JSON-LD block was a WebSite object:

<script type="application/ld+json">{"url":"https://www.example.com","name":"example","description":"","@context":"http://schema.org","@type":"WebSite"}</script>

In that example the description field was empty. Check your own output, and if fields are blank, review the site and page SEO descriptions you have set in Squarespace. If you need a type Squarespace does not list, such as Service or Person, you can add JSON-LD through code injection on the plans that support it. Keep it consistent with the built-in markup rather than declaring a second, conflicting organization. Google says no special structured data is needed to appear in AI Overviews or AI Mode, so treat schema as a way to state facts clearly, not as an AI ranking switch. Structured data for AI search separates what helps from what is hype.

How to check what crawlers actually get from your Squarespace site

Settings panels tell you what you asked for. These checks show what a crawler receives. Replace www.example.com with your primary domain.

1. Read your robots.txt

curl -s https://www.example.com/robots.txt

On the live Squarespace sites we read at the time of writing, with the AI checkbox off, the file starts with # Squarespace Robots Txt and lists the 26 AI user agents (plus Google's AdsBot agents) directly above User-agent: *, sharing one rule set that disallows system paths such as /config, /search, /api/ and /static/. That shared group does not block those bots from your pages, even though their names appear in the file. Under RFC 9309, consecutive User-agent lines share the rules that follow them. How robots.txt groups work explains the grouping.

After you change either checkbox, fetch the file again and compare. Look for a Disallow: / line and check which user agents it applies to.

2. Test each crawler against a real URL

For a quick check, Python's standard library can parse the file:

python3 - <<'EOF'
from urllib.robotparser import RobotFileParser
site = "https://www.example.com"
rp = RobotFileParser(site + "/robots.txt"); rp.read()
for ua in ["GPTBot", "ClaudeBot", "OAI-SearchBot", "Claude-SearchBot",
           "PerplexityBot", "Googlebot", "Bingbot"]:
    print(f"{ua:18} {rp.can_fetch(ua, site + '/about')}")
EOF

With both checkboxes off, every line should print True. With only the AI checkbox on, GPTBot and ClaudeBot should print False if the new rules disallow the whole site, while the search crawlers stay True. Python's parser is simpler than RFC 9309: it applies the first matching rule rather than the longest and does not understand * wildcards in paths, so treat it as a sanity check. Testing robots.txt against each AI crawler covers more exact tools.

3. Check a page for noindex and a lock screen

curl -sL -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  -o page.html -w "%{http_code} %{url_effective}\n" https://www.example.com/about
grep -io '<meta name="robots"[^>]*>' page.html
grep -c "a sentence copied from your page" page.html

The user agent is OpenAI's documented OAI-SearchBot string; the version number may change. The first command prints the status and final URL. The second shows any robots meta tag. The third should return at least 1. If it returns 0, the crawler got something other than your content, such as a password page.

4. Check the sitemap and the structured data

curl -s https://www.example.com/sitemap.xml | grep -o "<loc>" | wc -l
grep -o '"@type":"[A-Za-z]*"' page.html | sort | uniq -c

Squarespace's site map article says the sitemap only loads when the site is Public and leaves out disabled, hidden and password-protected pages. A page you expected to be in AI search that is missing from the sitemap is a sign one of those settings is on. Paste the page URL into Google's Rich Results Test or the Schema Markup Validator to confirm the markup parses.

Frequently asked questions

Does Squarespace block AI crawlers by default?

No. Squarespace says the "Block known artificial intelligence crawlers" box is unchecked by default. On the Squarespace sites we checked, robots.txt names the AI user agents even with the box off, but they share the default rules and can crawl your pages.

Will the Squarespace AI checkbox remove my site from ChatGPT search?

Not according to its published list. The checkbox includes GPTBot, which OpenAI uses for training, but not OAI-SearchBot, which OpenAI uses for ChatGPT search. The "Block Search engine crawlers" checkbox, by contrast, adds Disallow: / and would block OAI-SearchBot.

Can I block AI crawlers on only some Squarespace pages?

Not with the AI checkbox, which Squarespace says applies to the whole site. For individual pages you can use a page password, a Member Sites page or the "Hide page from search results" toggle, though these keep pages out of search generally, not only out of AI tools.

Can I edit robots.txt directly on Squarespace?

Squarespace's help articles describe the two Crawlers checkboxes as the way to change robots.txt and do not describe an editor for the file. If you need per-bot rules beyond those checkboxes, page-level meta tags through code injection are the remaining built-in option, and they control indexing rather than crawling.

Can I add my own schema to a Squarespace site?

Yes, on plans with code injection (Core, Plus, Advanced, Business, Commerce Basic and Commerce Advanced, not Basic or Personal). The built-in structured data stays in place and cannot be removed, so add only what it does not already cover.

Check your Squarespace site

To check one page without creating an account, run the free check. For a whole site, the audit covers up to 500 pages with deterministic checks and human review, and delivers a PDF with evidence and prioritized fixes.