BigCommerce robots.txt: the AI crawler rules you can't edit
BigCommerce adds a locked robots.txt rule set for known AI and high-volume crawlers. See where you edit the file, what you can add and how to check it.
By Crawlito · 9 min read
In short
- At the time of writing, BigCommerce's help article says a separate rule set for known high-volume and AI crawlers is applied automatically and cannot be viewed or edited in your store's website settings; you can see it by opening yourstoredomain/robots.txt.
- The article says you do not need to add any robots.txt entries to allow AI bot crawling, and the rules it lists for those crawlers block cart, checkout, account, admin and filter URLs, not product or category pages.
- You edit the file under Settings > Website > Search Engine Robots, which needs the Manage Settings permission, plus Edit Channels for a separate file per storefront.
- The article lists Crawl-delay: 10 in the platform's AI crawler rules, but Crawl-delay is not part of RFC 9309, so whether a given AI crawler honors it is up to that operator.
- A CDN or WAF in front of the store can still challenge AI crawlers even when robots.txt allows them, and robots.txt cannot fix that.
On this page
- Where do you edit robots.txt in BigCommerce?
- What are the AI crawler rules BigCommerce adds for you?
- What can you add, and what can't you remove?
- Can agencies manage robots.txt through the settings API?
- How do you keep product, category and policy pages open to AI search?
- Can a CDN or WAF still block AI bots?
- How do you read the live file and test each AI user agent?
- Frequently asked questions
- Check your own store
BigCommerce lets you edit robots.txt under Settings > Website, but not all of the file. The platform adds a separate set of rules for known high-volume and AI crawlers that you cannot view or change in settings. Those rules, as BigCommerce documents them, add a crawl delay and block cart, checkout, account, admin and filter URLs. They do not block product or category pages.
This guide covers where to edit the file, what the locked rules say, what you can safely add, how agencies can read and write it through the API, and how to check that AI crawlers can still reach the pages that matter. It does not promise that any AI assistant will cite your products. It helps you remove technical barriers and know which ones are not yours to remove.
Where do you edit robots.txt in BigCommerce?
According to BigCommerce's article Understanding the robots.txt file (it shows a last review date of 4/2/2026; check the live page for changes), the setting lives in the control panel:
- Go to Settings > Website.
- Scroll to the Search Engine Robots section.
- Add your rules and save.
Two permissions matter. Updating the file requires Manage Settings. Updating it for one storefront in a multi-storefront setup also requires Edit Channels. If a colleague can see the section but cannot change it, check those permissions before assuming the file is locked.
BigCommerce stores start with a default file. The article shows this as the text you get when you revert:
User-agent: *
Disallow: /account.php
Disallow: /cart.php
Disallow: /checkout.php
Disallow: /checkout
Disallow: /finishorder.php
Disallow: /login.php
Disallow: /orderstatus.php
Disallow: /postreview.php
Disallow: /productimage.php
Disallow: /productupdates.php
Disallow: /remote.php
Disallow: /search.php
Disallow: /viewfile.php
Disallow: /wishlist.php
Disallow: /admin/None of those lines hides product pages, category pages or blog posts. The article also warns that editing without understanding robots.txt can hurt your SEO, and that changes can take days or weeks to show up in crawling and indexing.
What are the AI crawler rules BigCommerce adds for you?
The same article says "a separate set of rules applies specifically to known high-volume and AI crawlers." It lists this rule set:
Crawl-delay: 10
Disallow: /account.php
Disallow: /cart.php
Disallow: /checkout.php
Disallow: /checkout
Disallow: /admin/
Disallow: /*?_bc_fsnf=1*
Disallow: /*&_bc_fsnf=1*The article also says BigCommerce adds the two _bc_fsnf filter rules to every store's robots.txt, and it explains why you see them in the live file but not in your store settings. In plain terms, the rules for AI and high-volume crawlers:
- ask them to wait 10 seconds between requests,
- keep them out of cart, checkout, account and admin URLs,
- keep them out of URLs carrying the
_bc_fsnf=1filter parameter.
What the article does not give you is the list of crawlers the rules apply to. It names no AI user agents, and it does not say whether Googlebot or Bingbot are on the list. The platform keeps that list, and per the article you cannot view or edit it from your store's settings.
You can still see the result. The article says these entries are visible at yourstoredomain/robots.txt, and that is where to read them.
What does Crawl-delay: 10 mean for a big catalog?
The article describes it as a hidden crawl delay that advises bots to limit crawling to 10-second intervals. If one crawler made one request at a time and honored it, that would cap it at 8,640 requests a day (86,400 seconds divided by 10). That is a ceiling for a well-behaved bot, not a measurement of anything.
Crawl-delay is not defined in RFC 9309, the robots.txt standard, so each operator decides whether to follow it. BigCommerce's own article adds that robots.txt directives are advisory and some crawlers may ignore them.
What can you add, and what can't you remove?
The article's description of editing is short. You can add rules such as Disallow: /content/. When you save, BigCommerce "automatically combines your custom rules with the system defaults," and if the same crawler appears in both, the rules are merged into a single entry with duplicates removed.
So the practical split is:
| You can | You cannot (in settings) |
|---|---|
Add Disallow rules for paths on your store | See or remove the platform's AI and high-volume crawler list |
| Add rules for a named crawler | See the automatic _bc_fsnf filter rules |
| Manage a separate file per storefront with Edit Channels | Remove the Crawl-delay: 10 in the platform rule set |
Do you need to add anything to allow AI crawlers?
No. The article says that by default you do not need to include any robots.txt entries to allow AI bot crawling. Pasting in a copied AI allow list is therefore not a required step on BigCommerce, and it can cause side effects.
Here is why, under RFC 9309. The User-agent: * group applies only to crawlers that no other group matches. If you add a short group for a crawler such as OAI-SearchBot, that crawler follows the group naming it and may stop following the cart and checkout rules in the * group unless your group repeats them. The BigCommerce article does not say whether it copies the * rules into groups you add, so assume it does not.
How do you opt out of training but stay in search?
If you want to opt out of model training, a group for the training crawler is enough:
User-agent: GPTBot
Disallow: /OpenAI's crawler documentation says GPTBot and OAI-SearchBot settings are independent, and that sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links. So disallowing GPTBot does not remove you from ChatGPT search. Do not add a group for OAI-SearchBot unless you mean to change its behavior. The wider logic is in robots.txt for AI crawlers.
What about per-storefront files?
With multiple storefronts, each can have its own file, as long as you hold Edit Channels. Check each storefront's domain separately. A rule added in one storefront's own settings does not appear on the others, so a missing rule on storefront B is not a bug in storefront A.
Can agencies manage robots.txt through the settings API?
BigCommerce documents a REST endpoint for the storefront robots.txt settings in its settings reference. The reference lists a GET and a PUT on the same path:
GET https://api.bigcommerce.com/stores/{store_hash}/v3/settings/storefront/robotstxt
PUT https://api.bigcommerce.com/stores/{store_hash}/v3/settings/storefront/robotstxtWhat the reference states:
- An optional
channel_idquery parameter selects channel-specific settings. If you leave it out, you work with global settings only. - The scopes are Information & Settings:
store_v2_information(modify) andstore_v2_information_read_only(read). - The body and the response carry a string field named
robots_txt_ssl. - A
nullvalue means a channel does not override the global value and inherits it. To remove a channel's overrides, the reference says you send every setting asnull; partial updates are not supported.
A read for one channel looks like this, with your own hash, channel and token:
curl -s "https://api.bigcommerce.com/stores/STORE_HASH/v3/settings/storefront/robotstxt?channel_id=CHANNEL_ID" \
-H "X-Auth-Token: YOUR_TOKEN" \
-H "Accept: application/json"The reference gives no description for robots_txt_ssl, and it says nothing about the AI crawler rules. So the API does not tell you whether the platform's rules are part of that string. Read the API response and the live /robots.txt side by side before you rely on either, and treat the live file as the evidence of what crawlers see.
How do you keep product, category and policy pages open to AI search?
Start from the fact that the documented defaults leave them open. Your risk is mostly what gets added later: a Disallow for a content folder, a rule copied from another platform, or a group that changes which rules a crawler follows.
- Read the live file and search it for any
Disallowthat matches product, category, brand, blog or policy paths (returns, shipping, privacy). - Keep filters and internal search out, not products. The
_bc_fsnfandsearch.phprules already do part of this. If you add more, test them against real product URLs. Faceted navigation and crawl traps explains how to separate useful landing pages from filter noise. - Check the raw HTML of a product page, not the rendered view, for the product name, price and description. Apps and headless builds that inject this with JavaScript can leave the first response thin. See category pages for AI search and product schema for what to expose.
curl -s https://www.example-store.com/product-name/ | grep -c "Your product name"Swap in the product name or a sentence from the description as it appears on the page. A count of 0 means that text is not in the HTML the server sends, so a crawler that does not run scripts will not see it.
Being readable does not decide whether an assistant chooses to mention a product. It only removes a reason it could not.
Can a CDN or WAF still block AI bots?
Yes, and robots.txt cannot help. robots.txt is a request to well-behaved crawlers. A firewall rule or bot challenge acts on the request itself, before the crawler ever reads your rules. BigCommerce's article points to a service such as Cloudflare for more robust bot controls, so some stores add one. If you do, its defaults are your next place to look.
OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges, which it lists in files such as openai.com/searchbot.json. If a bot rule challenges those addresses, OAI-SearchBot may not be able to read your pages even though robots.txt allows it. For Cloudflare specifically, see Cloudflare and AI crawlers.
How do you read the live file and test each AI user agent?
Fetch the file the way a crawler does, from your public domain:
curl -s https://www.example-store.com/robots.txt
curl -s -o /dev/null -w "%{http_code}\n" https://www.example-store.com/robots.txtThen work through this checklist:
- Confirm the status is
200. Under RFC 9309, a5xxmakes crawlers assume everything is disallowed, while a4xxmeans no restrictions apply. - Find the groups that name crawlers. Note any group for OAI-SearchBot, GPTBot, ChatGPT-User or others, and check that each still contains the cart and checkout rules you want.
- Compare responses by user agent. The article does not say whether the file varies by user agent, so fetch it once with a browser user agent and once with a crawler token, and diff the two.
- Request a product URL with each token and look at the status code. A
403or429often points to a firewall or rate limit rather than robots.txt.
for ua in "OAI-SearchBot" "GPTBot" "ChatGPT-User"; do
printf "%s " "$ua"
curl -s -o /dev/null -w "%{http_code}\n" -A "$ua" https://www.example-store.com/product-name/
doneThis sends the token from your own IP address, not OpenAI's, so rules based on IP or reputation can treat the real crawler differently. Your CDN's security log is better evidence. Our guide to verifying real AI crawlers covers checking a visit against published IP ranges. Remember that OpenAI says search-related robots.txt changes can take about 24 hours to take effect.
Frequently asked questions
Can I remove the AI crawler rules BigCommerce adds to robots.txt?
Not from your website settings. BigCommerce's article says you cannot view or edit that rule set or its list of crawlers there, and that you can see the entries by opening the live /robots.txt. The article names no other way to change them, so ask BigCommerce support if you need that.
Do the locked rules block AI crawlers from my products?
Not as the article documents them. The listed rules add a crawl delay and disallow cart, checkout, account, admin and filter URLs. They do not list product or category paths. Check your live file, because rules you or an app added are separate.
Do I need to add GPTBot or OAI-SearchBot to robots.txt on BigCommerce?
No. The article says you do not need robots.txt entries to allow AI bot crawling by default. Add a group only if you want a specific behavior, such as disallowing GPTBot to opt out of training.
Why does my robots.txt show rules I never added?
Two sources: the store's default file, which you can see and edit in settings, and the platform rules that you can see at /robots.txt but not in settings. The article says BigCommerce combines your custom rules with its defaults when you save.
Will fixing robots.txt get my products cited in AI answers?
No one can promise that. A readable, reachable page can be used; whether an assistant uses it depends on factors the operators do not publish.
Check your own store
The free check reads one page the way AI crawlers do, with no account: the robots.txt rules for each named AI crawler, the robots meta tag and headers, and whether the main text is in the raw HTML. Run it on a product page and a category page. For a whole catalog, an audit covers up to 500 pages with evidence and prioritized fixes.
Filed under Platform guides. Topics: bigcommerce, robots-txt, ai-crawlers, ecommerce