Shopify and AI crawlers: robots.txt, schema and theme checks

On a standard Shopify theme, AI crawlers can reach your products by default. Shopify's generated robots.txt does not name any AI crawler, so GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot all fall under the User-agent: * group, which blocks admin, cart, checkout, order and account paths, some internal endpoints, and a set of sort, filter and tag-combination URLs. What usually goes wrong on Shopify is not the default file: it is a hand-edited robots.txt.liquid, an app that adds reviews or schema with JavaScript, a pile of duplicate collection URLs, or a theme and an app that output two conflicting JSON-LD blocks.

This guide goes through each of those, with a way to check it and a way to fix it.

What Shopify's default robots.txt allows

Every Shopify store serves a generated robots.txt at /robots.txt. Shopify updates the defaults over time, so your store may show a slightly different version. Here is a shortened excerpt from a current default file, fetched at the time of writing:

User-agent: *
Allow: /
Disallow: /admin
Disallow: /cart/
Disallow: /checkout
Disallow: /checkouts/
Disallow: /orders
Disallow: /account
Disallow: /recommendations/products
Disallow: /collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*filter*&*filter*
Disallow: /blogs/*+*
Disallow: /*?*preview_theme_id=*

User-agent: adsbot-google
Allow: /products/
Allow: /collections/
Disallow: /checkout
Disallow: /checkouts/
Disallow: /orders

Sitemap: https://example-store.com/sitemap.xml

The full file is longer. It adds Allow lines for paths such as /account/login and /products/checkout, a /*/ copy of most rules for localized paths, and a few more internal endpoints in both groups.

Four things follow from this:

• No AI crawler is named. Each one uses the * group, so product, collection, page and blog URLs are open to all of them.
• Sort, filter and tag-combination URLs are blocked for every crawler. That is deliberate: it keeps crawlers off near-duplicate listing pages.
• The product recommendations endpoint is blocked, which matters later when you look at what loads with JavaScript.
• The adsbot-google group exists because Google says AdsBot ignores the global * group and only obeys rules that name it.

The current default file also starts with comment lines aimed at AI agents, pointing to an agents.md file and Shopify's agent endpoints. Lines starting with # are comments, so they change nothing about which URLs crawlers may fetch.

If you have never touched robots.txt on Shopify, the access part is probably fine. Check it anyway, because themes, agencies and apps sometimes add a template you don't know about.

How to edit robots.txt with robots.txt.liquid

Shopify's robots.txt customization guide says to create a robots.txt.liquid template in your theme's Templates folder. Adding that file replaces Shopify's generated output. The template supports only the robots, group, rule, user_agent, sitemap and request Liquid objects.

Shopify recommends looping over robots.default_groups rather than pasting a static copy of the file. The loop keeps the rules Shopify maintains; a pasted copy freezes them on the day you saved it. A template that keeps the defaults and opts out of OpenAI and Anthropic training while staying eligible for ChatGPT search looks like this:

{% for group in robots.default_groups %}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

The two groups at the bottom are plain text outside the Liquid loop, which is where Shopify's guide puts rules for crawlers that are not in the default set. OpenAI documents that GPTBot and OAI-SearchBot are controlled independently, so this keeps the store eligible for ChatGPT search while signalling that content should not be used for training. Anthropic's crawler article gives the same User-agent: ClaudeBot and Disallow: / pattern for a training opt-out. Whether you opt out of training is a business choice; the guide to allowing AI search but blocking training covers the other tokens.

The mistake to avoid: a named group for a search crawler

It is tempting to add a group that "welcomes" an AI search crawler:

User-agent: OAI-SearchBot
Allow: /

That group is not needed, since the * group already allows the crawler. It also has a side effect: under RFC 9309, a crawler that finds a group naming it ignores the * group entirely. OAI-SearchBot would then no longer be bound by the cart, checkout, sort and filter rules in the default group. If you do add a named group, repeat the rules you still want inside it. The post on how robots.txt groups work explains the matching.

How to check your robots.txt

Before publishing a template, Shopify advises saving the current /robots.txt output and comparing it with what the template renders, because the two don't always match exactly. From a terminal:

curl -s https://your-store.com/robots.txt > robots-before.txt

Publish the theme change, then fetch the file again and compare:

curl -s https://your-store.com/robots.txt > robots-after.txt
diff robots-before.txt robots-after.txt

Look for a Disallow: / under User-agent: *, a missing Sitemap: line, and two rules run together on one line. A hand-edited template can produce that last one when Liquid whitespace control ({%- and -%}) strips a line break, and a crawler cannot read a line like Disallow: /a*Disallow: /b as two rules. Allow time after a change: OpenAI says its search systems can take about 24 hours to adjust, and Perplexity says changes may take up to 24 hours.

Theme JSON-LD for products

Most Shopify themes print product data straight into the HTML from Liquid, which is good for crawlers: the title, description, price and availability are in the server response, not loaded later. Structured data is where themes differ.

Shopify's own Dawn theme uses the structured_data filter in its product section:

<script type="application/ld+json">
  {{ product | structured_data }}
</script>

The filter accepts product and article objects. For a product it returns a schema.org Product when the product has no variants and a ProductGroup when it has variants; for a blog post it returns an Article. Older or heavily customised themes often write the JSON-LD by hand instead, and SEO apps frequently add their own block on top.

Things to check on a product page:

1. How many blocks describe the product. Two separate blocks with different prices or availability give machines two answers for one page.
2. Whether values match the page. A hand-written block that reads a compare-at price, or a currency that differs from the one shown, is a defect.
3. Whether ratings are real and visible. Review markup should come from genuine reviews shown on the page.

Count the JSON-LD blocks, then list the types inside them:

curl -s https://your-store.com/products/your-product \
  | grep -o 'application/ld+json' | wc -l

curl -s https://your-store.com/products/your-product \
  | grep -o '"@type": *"[A-Za-z]*"' | sort | uniq -c

Read the second result with care. A ProductGroup lists each variant as a Product inside it, so several Product entries next to one ProductGroup are expected, and types such as Offer or BreadcrumbList are normal. The problem is two blocks that each describe the same product, for example the theme's ProductGroup plus an app's separate Product. Keep one source: either the theme's filter output or the app's, not both. The Product schema guide lists which properties matter. Keep expectations honest: Google says no special schema is needed to appear in AI Overviews or AI Mode, so structured data is about accuracy, not a shortcut into AI answers.

Apps that inject content with JavaScript

Shopify apps add things to a theme in two ways, according to the theme app extension docs. App blocks place inline content inside sections, such as product reviews or star ratings. App embed blocks are for apps with no visible component or with floating elements such as chat bubbles; Shopify injects them before the closing </head> or </body> tag, and apps that add SEO meta tags, analytics or tracking pixels use them.

Both are Liquid, so an app can render its text on the server. Many don't: they output an empty container and fill it with JavaScript after the page loads. That matters because most AI crawlers read only the HTML the server sends. Vercel and MERJ's December 2024 analysis found that none of the major AI crawlers it measured, including OpenAI's crawlers, ClaudeBot and PerplexityBot, rendered JavaScript at the time.

Common Shopify content that may only exist after JavaScript runs:

To check, compare a phrase from the visible widget with the raw HTML:

curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://your-store.com/products/your-product | grep -c "a phrase from one review"

A count of 0 means that text is not in the server response. The user agent only labels the request: it does not come from OpenAI's IP ranges, so bot protection may treat it differently from the real crawler. The fix is app-specific: look in the app's settings and documentation for a server-rendered option or a Liquid snippet that prints a review summary into the page. If there isn't one, put the essential facts (materials, sizing, care, shipping) in the product description itself, which the theme renders on the server. For the general technique, see how to see your page as an AI crawler with curl.

Duplicate collection and product URLs

Shopify lets one product be reached at several URLs:

• /products/linen-shirt
• /collections/summer/products/linen-shirt
• /collections/sale/products/linen-shirt
• /products/linen-shirt?variant=12345

The collection-scoped form comes from the within filter, which builds a product URL in the context of a collection. Shopify's documentation notes that the standard and collection-scoped pages have the same content on separate URLs and asks you to consider the SEO implications before using it.

Two checks:

1. The canonical tag. Dawn prints <link rel="canonical" href="{{ canonical_url }}"> in its layout. On every one of the URLs above, the canonical should point to the one URL you want indexed, normally /products/linen-shirt. Shopify's canonical_url reference does not spell out what it returns on collection-scoped or ?variant= URLs, so check the output rather than assume it.
2. Which URL your theme links to. If collection grids link with product.url | within: collection, crawlers find a separate URL for every collection a product sits in. Dawn's product cards link to card_product.url without within. If your theme uses within, removing it from the product card snippet stops the duplicates at the source.

curl -s https://your-store.com/collections/summer/products/linen-shirt \
  | grep -o '<link rel="canonical"[^>]*>'

Canonicals are a hint, not a command, and crawlers that don't run JavaScript only see the one in the raw HTML. The post on canonical tags and duplicate URLs covers how conflicting signals get resolved.

Bot protection on the storefront

Shopify's help article on bots says Shopify uses Cloudflare to protect stores from bots, that Cloudflare is always active and needs no action from the merchant, and that visitors whose behaviour seems automated may be asked to verify their connection. hCaptcha is on by default for contact and comment forms and customer account pages.

What that means in practice:

• There is no edge setting to manage. The article describes no setting for allowing or blocking individual crawlers, unlike a site behind your own Cloudflare account. Your levers are robots.txt and the theme.
• Your own audit tools may need a signature. Shopify offers signed crawler access in the Crawler access section of Online Store > Preferences. It creates Web Bot Auth signatures, sent as three headers (Signature-Input, Signature and Signature-Agent), that tools such as Screaming Frog, Sitebulb, Jet Octopus and OnCrawl can add to their requests. A signature lasts at most three months, cannot be renewed, covers one domain and does not grant access to checkout. Shopify says search engines and LLMs can index the store without signatures, and that rate-limiting errors on a signed crawler mean the signature isn't valid.
• Private mode blocks everyone. Turning on Private mode in the Store access section of Online Store > Preferences shows visitors a password landing page, and Shopify says it keeps search engines out too. A store left in Private mode after launch is closed to every crawler.

Frequently asked questions

Does Shopify block AI crawlers by default?

Not in robots.txt. At the time of writing, Shopify's generated robots.txt names no AI crawler, so they follow the User-agent: * group, which allows product, collection, page and blog URLs.

Can I edit robots.txt on Shopify?

Yes. Create a robots.txt.liquid template in your theme's Templates folder. It replaces the generated file, so keep the loop over robots.default_groups and add your own groups as plain text after it.

Does Shopify add product schema automatically?

Not by itself: the theme does. Dawn and other themes that use {{ product | structured_data }} output Product or ProductGroup JSON-LD. Check older themes and SEO apps for a second, conflicting block.

Are /collections/.../products/... URLs a problem for AI crawlers?

They duplicate /products/<handle>. They do little harm when the canonical tag on each points to the product URL, but linking to them from every collection multiplies the URLs crawlers have to fetch.

Will blocking GPTBot on Shopify remove my store from ChatGPT search?

No. OpenAI says GPTBot covers training use and OAI-SearchBot covers ChatGPT search, and the two robots.txt settings are independent.

Check your own store

Pick your best-selling product page and run the curl checks above: robots.txt, JSON-LD blocks and raw HTML text. The free single-page check looks at one URL without an account, and a full audit covers up to 500 pages with evidence and prioritized fixes.