Adobe Commerce robots.txt: edit it and set AI crawler rules
Where the Magento robots.txt is edited in Adobe Commerce, how cloud and Fastly change it, and how to add AI crawler groups without blocking products.
By Crawlito · 7 min read
In short
- Adobe's documentation puts the Magento robots.txt editor in the Admin under Content > Design > Configuration, in the Search Engine Robots section, so you change it there instead of uploading a file.
- On Adobe Commerce cloud the project root is not writable, and Adobe's playbook serves robots.txt from pub/media through a custom Fastly VCL snippet.
- Multi-site cloud projects need one file per site, named with a site prefix, plus a VCL rule that picks the file by host.
- A robots.txt group that names an AI crawler replaces the User-agent: * rules for that crawler, so copy over the default Disallow lines you still want.
- Adobe says not to list your Admin path in robots.txt, and RFC 9309 says the file's rules are not a form of access authorization.
On this page
- Where robots.txt lives in Adobe Commerce and how to edit it
- Cloud projects: read-only file system and the Fastly rewrite
- Multi-store setups: one robots.txt per site
- Adding named AI crawler groups safely
- Do not expose the Admin path in robots.txt
- When changes do not show: caches and the knowledge-base fix
- Keep product and category pages open to AI search bots
- Frequently asked questions
- Check your store's rules
To change the Magento robots.txt on Adobe Commerce, open the Admin, go to Content > Design > Configuration, edit the Global row and use the Search Engine Robots section. You do not upload a file. To control AI crawlers, add a named group for each one, copy over the default Disallow lines you still want, and leave your Admin path out of the file entirely.
Adobe's robots.txt best practices page says it applies to the native Adobe Commerce storefront, on cloud infrastructure and on-premises. It does not cover Adobe Experience Manager or headless storefronts. This post follows the same scope.
Where robots.txt lives in Adobe Commerce and how to edit it
Adobe's SEO overview gives these steps:
- In the Admin sidebar, go to Content > Design > Configuration.
- In the first row, click Edit for the Global configuration.
- Expand Search Engine Robots.
- Enter your rules in the custom instruction box for robots.txt. Reset to Default reverts them.
- Click Save Configuration.
The same section has a Default Robots setting with four values: INDEX, FOLLOW, NOINDEX, FOLLOW, INDEX, NOFOLLOW and NOINDEX, NOFOLLOW. Adobe's page describes them in terms of whether crawlers index the site and follow links, and it does not say how they are applied. On a live store, check that it is not set to a NOINDEX value left over from development.
Adobe says the robots.txt that Commerce generates by default tells web crawlers to avoid indexing certain parts of the site. At the time of writing, the list on that page includes /checkout/, /customer/, /catalog/, /app/, /lib/, /var/, /review/ and /sendfriend/, all under User-agent: *. The list is longer than that, so copy from the box in your own Admin, not from the documentation.
To check what the public sees, fetch the file:
curl -s https://www.example.com/robots.txt
curl -s -o /dev/null -w "%{http_code}\n" https://www.example.com/robots.txtA 200 with your text is the goal. Under RFC 9309, a 5xx makes crawlers assume everything is disallowed, while after a 4xx they may access any resource on the server.
Cloud projects: read-only file system and the Fastly rewrite
Adobe's cloud guide says cloud infrastructure only allows writes to directories such as var, pub/media, pub/static and app/etc, not the root. It also says the robots.txt content is generated on demand and stored in the database, so you do not create a physical file.
The playbook describes the root-level URL like this for a single site:
- Use
ece-toolsversion 2002.0.12 or later. - Edit robots.txt in the Admin, and generate
sitemap.xmlwith thepub/mediapath set, because the file system is read-only. - Add a custom Fastly VCL snippet named
sitemaprobots_rewrite, typerecv, priority90, that maps root/robots.txtand/sitemap.xmltopub/media/. - Open
/robots.txtand/sitemap.xmlon the root path to test.
Adobe's pages describe the mechanism in slightly different terms, one as a generated file served from pub/media and one as content generated from the database. Do not rely on either description. Fetch the public URL and see what comes back.
If /robots.txt returns a 404, the cloud guide says to submit an Adobe Commerce Support ticket to remove the redirect from /robots.txt to /media/robots.txt. It also says that if you use PWA Studio and cannot reach robots.txt, you should add robots.txt to the Front Name Allowlist under Stores > Configuration > General > Web > UPWARD PWA Configuration.
Multi-store setups: one robots.txt per site
RFC 9309 puts the file at the top level of the service, at scheme:[//authority]/robots.txt, so each domain needs its own file. Adobe's playbook handles this on cloud by prefixing file names with the site name:
domainone_robots.txt
domaintwo_robots.txt
domainone_sitemap.xml
domaintwo_sitemap.xmlIt then modifies the sitemaprobots_rewrite snippet so it matches the exact /robots.txt and /sitemap.xml requests, checks the host header against a pattern and rewrites to pub/media/ plus the site name. The sample hardcodes two example domains, so you must adapt the pattern to your own. The cloud guide describes a similar per-domain setup with pub/media/domain_robots.txt and the same recv type and priority 90. Copy the snippet and naming from the page that matches your project instead of improvising.
For on-premises installs, Adobe says to use /path/to/commerce/pub/media/ or /path/to/commerce/media, whichever applies. The pages I read do not spell out a multi-site on-premises recipe, so treat that as undocumented there and test each domain's /robots.txt yourself.
For the host-by-host rules in general, see robots.txt for AI crawlers.
Adding named AI crawler groups safely
Adobe's default block sits under User-agent: *. Under RFC 9309, a crawler follows the group that names it and falls back to * only when no group does. So if you add User-agent: OAI-SearchBot with Allow: /, that crawler stops following your default Disallow: /checkout/ and Disallow: /customer/ lines. Our guide to user-agent groups shows the mechanics.
Pick the crawlers by purpose. OpenAI's crawler documentation says OAI-SearchBot surfaces sites in ChatGPT search answers, GPTBot may be used for training content, and the two settings are independent. Anthropic's crawler help article lists ClaudeBot for training, Claude-SearchBot for search and Claude-User for user requests. Here is a pattern that keeps the search crawlers on the same rules as everyone else and opts out of training:
User-agent: *
Disallow: /checkout/
Disallow: /customer/
# ...the rest of your default Disallow lines
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
Disallow: /checkout/
Disallow: /customer/
# ...repeat the same default Disallow lines here
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /Stacked User-agent lines share the rules beneath them. Keep every line you want for the search crawlers inside their own group. This is a policy choice, not a requirement. If you would rather not opt out of training, leave the last group out.
OpenAI says ChatGPT-User acts on a user's request and that robots.txt rules may not apply to it, so do not count on a Disallow for it. After you save, OpenAI says search-related changes can take about 24 hours to take effect.
Do not expose the Admin path in robots.txt
Adobe's playbook says not to expose your Admin path in robots.txt and calls it a vulnerability for site hacking and potential data loss. It says to remove all entries for the Admin path, and points to the Marketing User Guide under SEO and Search > Search Engine Robots.
The reason is simple. robots.txt is public, and RFC 9309 says its rules are not a form of access authorization and that listing a path can expose it. A Disallow line tells every visitor where the door is. Open your custom instruction box, search for the path your Admin uses and delete any line that names it. Protecting the Admin is a job for authentication and network controls, not robots.txt.
When changes do not show: caches and the knowledge-base fix
Adobe's knowledge-base article on robots.txt not updating covers cloud 2.4.x. The symptom: you save custom text in Content > Design > Configuration and the public URL still shows the default. The cause: search engine indexing is disabled, which stops the custom content from being applied. The fix is to enable indexing in the Cloud Console, or run the command the article gives (check it against your magento-cloud CLI help first):
magento-cloud environment:info -p <project_id> -e production restrict_robots falseAdobe says this does not apply to Staging, where indexing is disabled by default and cannot be enabled. The article does not mention flushing a cache. If indexing is enabled and the public file is still old, a cache in front of the site is the next suspect, and CDN caching and stale content explains how to compare layers. Adobe's pages I read do not document how Fastly caches robots.txt, so I will not guess.
Keep product and category pages open to AI search bots
The most common failure is a blocked store, not a stale file. Go through this list:
- Fetch your product, category and CMS URLs and compare each path with every
Disallowline in the group that applies to a search crawler. - Check that no leftover
Disallow: /from development is in the box, since Adobe gives that as an example for sites in development. - Check that the Default Robots setting is not a NOINDEX value in production.
- If you want to limit filter URLs, test rules against valid category pages first. Faceted navigation covers how to avoid hiding the catalog.
Being allowed in robots.txt only makes pages reachable. Whether any assistant cites a product page depends on much more. Clear markup helps readers and parsers alike; see Product schema. For another platform's equivalent, Shopify and AI crawlers shows how its generated file differs.
Frequently asked questions
Where is robots.txt in Magento?
In the Admin under Content > Design > Configuration, on the Global row's Search Engine Robots section. On cloud, Adobe says the content is generated and stored in the database, so there is no physical root file to edit.
Why does my custom robots.txt not appear on Adobe Commerce cloud?
Adobe's knowledge base gives the cause as search engine indexing being disabled, which stops the custom content from being applied. Enable it in the Cloud Console or set restrict_robots to false with the CLI. It does not apply to Staging.
Do I need a separate robots.txt for each store?
Each domain serves its own file. On cloud, Adobe's playbook uses site-prefixed files in pub/media and a Fastly VCL snippet that picks one by host header.
Should I add my Admin URL to robots.txt?
No. Adobe says to remove Admin path entries, because an exposed Admin path is a vulnerability. The file is public, and RFC 9309 says its rules do not authorize or restrict access.
Will adding AI crawler rules affect my search ranking?
Adobe's documentation does not say. Rules for OAI-SearchBot or Claude-SearchBot only decide whether those crawlers may fetch your pages, so check your Googlebot rules separately.
Check your store's rules
The free check reads one page and reports the robots.txt rules that apply to each named AI crawler, with no account needed. If you run several stores or domains, an audit can cover up to 500 pages with evidence and prioritized fixes.
Filed under Platform guides. Topics: adobe-commerce, magento, robots-txt, ai-crawlers, ecommerce