Shopware robots.txt: user-agent rules in 6.7 and older setups
Where Shopware robots.txt rules live in each version, when the Admin accepts user-agent groups for AI crawlers, and how to check the file each domain serves.
By Crawlito · 9 min read
In short
- Before Shopware 6.7.1 there is no generated robots.txt: you create a plain text file in the public directory, or one file per host in public/robots with a rewrite rule.
- From 6.7.1 Shopware serves /robots.txt itself and you add rules per domain under Settings > General > Basic information, in the Rules for robots.txt section.
- In Shopware's source, the Admin field only takes Allow and Disallow lines up to 6.7.4.0; separate User-agent groups and a switch to disable the default rules appear in 6.7.5.0.
- Shopware prefixes every Allow and Disallow path with the domain's path, so a rule on the example.com/en domain is written out as /en/... in the served file.
- On Apache with Shopware's default .htaccess, a leftover public/robots.txt is served before the generated one, so delete it after moving rules into the Admin.
On this page
- Which Shopware robots.txt setup applies to your version
- Before 6.7.1: a static file in the public directory
- From 6.7.1: the Admin field and how Shopware renders it
- AI crawler groups for a Shopware shop
- Adding rules from a plugin (6.7.5 and later)
- How to check the robots.txt each domain serves
- Frequently asked questions
- Check what crawlers get from your shop
How you set Shopware robots.txt rules depends on your version. Before 6.7.1 you write a static robots.txt file in the public directory. From 6.7.1 Shopware generates the file and you add rules per domain in the Admin under Settings > General > Basic information. Per-crawler User-agent groups, the ones you need to treat AI crawlers differently, are only accepted in that Admin field from 6.7.5 onwards, going by Shopware's source code.
This post covers the Shopware-specific parts: which era your shop is in, how the Admin field is turned into the served file, AI crawler groups that fit a shop, and how to check what each sales channel domain actually returns. For the general rules on how groups are matched, see robots.txt groups explained.
Which Shopware robots.txt setup applies to your version
Shopware documents two eras, and the source code adds a third step in between:
| Version | Where the rules live | Per-crawler groups |
|---|---|---|
| 6.0.0 to 6.7.0.0 | A file you create in public/ (or one file per host in public/robots/) | Yes, you write the whole file |
| 6.7.1.0 to 6.7.4.x | Admin field, rendered by the /robots.txt route | No: the field takes Allow and Disallow lines only |
| 6.7.5.0 and later | Admin field, plus plugin events | Yes: User-agent groups in the field, extendable by plugins |
The user guide Creation of a robots.txt is marked for versions 6.0.0 to 6.7.0.0 and says the file is not created automatically. The developer guide Extend robots.txt configuration says full support with user-agent blocks arrived in 6.7.1 and that its events need 6.7.5.
The source tells a slightly different story for 6.7.1 to 6.7.4. At the v6.7.1.0 tag, the template prints one User-agent: * group, the default rules, your domain rules and the sitemap links, and the Admin field is labelled "Domain specific robots.txt Allow/Disallow rules". The same is true at v6.7.4.0. Only at v6.7.5.0 does the template print your own User-agent groups. In 6.7.1 to 6.7.4 the field keeps only lines that start with Allow or Disallow and silently skips the rest, including User-agent lines. Pasting a User-agent: GPTBot group with Disallow: / into that field therefore adds Disallow: / to the User-agent: * group and blocks every crawler. If you are on 6.7.1 to 6.7.4 and need a separate group for GPTBot, update to 6.7.5 or later, or keep a static file (see the last section).
Before 6.7.1: a static file in the public directory
Shopware's user guide says to create robots.txt in the public directory and gives a sample that starts with User-agent: * and Allow: /, then disallows URLs with a query string, /account/, /checkout/, /widgets/, /navigation/, /bundles/ and the imprint, privacy and terms pages, and ends with a Sitemap line you have to fill in with your domain.
For several domains or sales channels, the guide uses a rewrite to pick a file by host. For Apache, add this at the start of public/.htaccess:
RewriteRule ^robots\.txt$ robots/%{HTTP_HOST}.txt [NS]For nginx, add the equivalent to the server block:
rewrite ^/robots\.txt$ /robots/$host.txt;Then create public/robots/ with one file per hostname, named exactly like the host, such as shop.example.com.txt. Shopware notes that the plain public/robots.txt is then no longer used. Remember that www.example.com and example.com are different hosts and need different files. Google's robots.txt documentation says the rules apply only to the host, protocol and port number where the file is hosted.
A community plugin, FroshRobotsTxt, also generates the file for older versions. Its README says it cannot distinguish user agents, so it does not help with AI crawler groups.
From 6.7.1: the Admin field and how Shopware renders it
Shopware's basic information settings page describes the Rules for robots.txt section as the place to define domain-specific rules. Because it is a sales channel setting, you can give each sales channel its own rules. On a 6.7.5 or later shop the section has two fields: Disable default robots.txt rules and Domain specific robots.txt rules.
Reading the template and the page loader at v6.7.16.0, the served file is built in this order:
- A
User-agent: *group with Shopware's defaults:Allow: /,Disallow: /*?andAllowlines for theme, media and thumbnail files, plus product feed tracking URLs. - Any
AlloworDisallowlines from your field that sit outside aUser-agentgroup. They are appended to the*group. - Your own
User-agentgroups. - A
Sitemapline for each matching sales channel domain.
Three behaviours matter for AI crawler rules:
- Paths get the domain path in front. Each path is prefixed with the path of the sales channel domain. With domains
shop.example.comandshop.example.com/enon the same host,Disallow: /checkout/set on the English one comes out asDisallow: /en/checkout/. - Staging switches everything off. If the
core.stagingsetting is on, the template printsDisallow: /underUser-agent: *and nothing else, not even your groups or the sitemap lines. Check this after copying a database between environments; staging noindex leaks covers the reverse problem. - Unknown lines are dropped. The parser skips any directive it does not recognise, and Shopware logs a warning for it when the configuration is saved. At v6.7.16.0 the recognised types are
Allow,Disallow,Crawl-delay,Sitemap,Request-rate,Visit-time,HostandUser-agent. So aContent-Signalline, as described in Content Signals in robots.txt, does not reach the served file in that version. Shopware's development branch addsContent-usageandContent-signal, but at the time of writing no released version includes them and Shopware has not said which release will.
If you tick Disable default robots.txt rules, the User-agent: * line disappears too. Any loose Allow or Disallow lines would then sit outside a group, so put every rule under a User-agent line yourself.
AI crawler groups for a Shopware shop
Under RFC 9309, a crawler obeys the group that names it and ignores the * group. A named group in Shopware therefore loses the defaults and anything you appended to *. Name a crawler only if you want it treated differently from everyone else.
Pick crawlers by purpose. OpenAI's crawler documentation says OAI-SearchBot surfaces sites in ChatGPT search, while GPTBot gathers content that may be used to train its models. Anthropic's crawler article lists ClaudeBot for training, Claude-SearchBot for search and Claude-User for user requests. If you want search access but no training, enter this in Domain specific robots.txt rules on the sales channel whose domain is the root of the host:
Disallow: /account/
Disallow: /checkout/
Disallow: /widgets/
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /On 6.7.5 or later the served file for https://shop.example.com/robots.txt then looks roughly like this (blank lines and the full default list trimmed):
User-agent: *
Allow: /
Disallow: /*?
Allow: /*theme/
Allow: /media/*?ts=
Disallow: /account/
Disallow: /checkout/
Disallow: /widgets/
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Sitemap: https://shop.example.com/sitemap.xmlOAI-SearchBot and Claude-SearchBot are not named, so they follow the * group with the same exclusions as every other crawler. Whether you opt out of training is a policy choice; allow AI search, block AI training sets out the trade-offs.
Keep the loose Allow and Disallow lines above the first User-agent line. Shopware's parser skips blank lines, so a rule placed after a User-agent line, even after an empty line, joins that crawler's group instead of the * group.
A few more details from the parser are worth knowing. Shopware's help text for the field says User-agent groups apply across all domains while loose rules apply to the domain path. In the source, groups for the same crawler from every domain on the host are merged into one, and their path rules still get each domain's prefix. Stacked User-agent lines are written out as separate groups with the same rules, which crawlers read the same way. A User-agent line with no rules under it is dropped. And because path rules are prefixed per domain, a Disallow: / set only on the /en sales channel becomes Disallow: /en/, which blocks the English section but not the rest of the host. If the root and /en domains belong to the same sales channel, they share one rules field, so every rule is written out once for each domain path.
If you give a search crawler its own group, for example to set a Crawl-delay, copy the exclusions it should keep. Be careful with Disallow: /*?: on its own it also blocks theme and media URLs that carry a cache-busting query string, which is why Shopware's defaults add Allow lines for them. Wide query rules on category pages need the same care; see faceted navigation crawl traps.
OpenAI says ChatGPT-User acts on a user's request and that robots.txt rules may not apply to it, so a Disallow for it is not a reliable control.
Note that the AI crawler example in Shopware's developer guide uses the token anthropic-ai. Anthropic's current article does not list it, so use ClaudeBot, Claude-SearchBot and Claude-User.
Adding rules from a plugin (6.7.5 and later)
The developer guide documents two events. RobotsDirectiveParsingEvent fires after the configured text is parsed and lets a plugin add or change groups, for example to append AI crawler groups for every shop it is installed in. RobotsUnknownDirectiveEvent fires for directives Shopware does not recognise and lets a plugin mark them as handled so the warning goes away. Plugins can also add their own validation messages with ParseIssue, which Shopware logs when the configuration is saved in the Admin.
Use the parsing event when an agency manages many shops and wants the same policy everywhere. For a single shop, the Admin field is simpler and easier for the next person to find.
How to check the robots.txt each domain serves
Every sales channel domain can return a different file, so check each host, and include www and non-www variants:
curl -s https://shop.example.com/robots.txt
curl -s https://www.shop.example.com/robots.txt
curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://shop.example.com/robots.txtLook for these signs:
- A
Sitemapline with your real domain. The generated file adds it for each domain. If you seeYOUR_DOMAINor no sitemap, a static file is being served instead. - No bare
Disallow: /underUser-agent: *. That is either the staging switch or a leftover development rule. - Your AI crawler groups are present. If they are missing on 6.7.1 to 6.7.4, the version does not render them, and their
Disallowlines may have landed in the*group. On 6.7.5 or later, check the Shopware log for parse warnings written when the rules were saved, such as an unknown directive or a malformed line. - The paths carry the right prefix for domains with a path such as
/enor/de.
If an Admin change does not show up, look for a static file first. Shopware's default .htaccess serves any file that exists on disk before it hands the request to index.php, so an old public/robots.txt or a robots/ rewrite from before 6.7.1 wins over the generated route. On nginx it depends on your server block. The route is also marked for Shopware's HTTP cache in the controller, so clear the cache and any reverse proxy in front of the shop, then fetch again. OpenAI says search changes can take about 24 hours to show in its systems.
Other platforms handle the same job differently: Adobe Commerce, for example, edits the file in its design configuration and serves it through a Fastly rewrite on cloud.
Frequently asked questions
Where is robots.txt in Shopware 6?
Before 6.7.1 it is a file you create in the public directory. From 6.7.1 Shopware generates it, and you add rules per domain under Settings > General > Basic information, in the Rules for robots.txt section.
Can I block GPTBot in the Shopware Admin?
On 6.7.5 or later, yes: add a User-agent: GPTBot group with Disallow: / to the domain rules field. In Shopware's source for 6.7.1 to 6.7.4, the field skips User-agent lines and puts every Allow and Disallow line into the User-agent: * group, so a GPTBot block there would block all crawlers.
Why does my Shopware robots.txt ignore the Admin settings?
Usually a static public/robots.txt or a per-host rewrite from an older setup is still in place and the web server serves it first. Delete it, clear the cache, and fetch the file again for each domain.
Does Shopware add the sitemap to robots.txt automatically?
The generated file adds a Sitemap line for each matching sales channel domain, such as https://shop.example.com/sitemap.xml. A static file needs the line written by hand.
Will allowing AI crawlers get my products into ChatGPT?
No setting can promise that. robots.txt only decides whether a crawler may fetch your pages; whether an assistant uses them depends on much more.
Check what crawlers get from your shop
After you change the rules, fetch /robots.txt on every domain and then run the free check on a product page to see whether the main AI crawlers can reach it. For a shop with several sales channels and domains, an audit reviews robots.txt per host together with rendering, status codes and markup.
Filed under Platform guides. Topics: shopware, robots-txt, ai-crawlers, ecommerce