Sitemap in robots.txt: multiple files, other hosts and errors
How the Sitemap line in robots.txt works with several files, sitemaps hosted on another domain, platform-generated files, and the errors that make crawlers skip it.
By Crawlito · 9 min read
In short
- A Sitemap line in robots.txt needs a fully qualified URL; Google accepts any number of them, and they apply to every crawler that reads the file, whatever group they sit next to.
- Under sitemaps.org cross-submission rules, a host can point its robots.txt at a sitemap stored on another host, but that sitemap may list only URLs from the host that references it.
- Google's Search Console Sitemaps report lists only sitemaps submitted in the report or through the API, not ones discovered through robots.txt.
- Google says a sitemap index cannot list other sitemap index files, and its referenced sitemaps must be on the same site unless cross-site submission is set up.
- RFC 9309 does not define the Sitemap record, so support depends on each crawler; Google, Bing and the sitemaps.org protocol document it, while OpenAI, Anthropic and Perplexity do not say whether their bots use it.
On this page
- Several Sitemap lines or one sitemap index?
- Can robots.txt point to a sitemap on another host?
- Why a Sitemap line is ignored
- Platform-generated robots.txt files that drop or duplicate the line
- How to check which crawlers fetched the sitemap
- Frequently asked questions
- Check the file crawlers actually receive
A Sitemap line in robots.txt tells any crawler that reads the file where your XML sitemap lives. You can list several sitemaps, point to a sitemap stored on a different host, or point to a single sitemap index, and all three are valid. Things go wrong when the URL is relative, when the sitemap lists URLs from a host whose robots.txt does not point to it (those URLs are dropped), when the sitemap itself is blocked or broken, or when a platform rewrites robots.txt and drops the line without telling you.
The basics fit in one paragraph. The value must be a full URL with protocol and host, the line is independent of User-agent groups so it can go anywhere in the file, and you can repeat it. What belongs in the sitemap, the 50,000-URL and 50 MB limits, lastmod and submission to Google and Bing are covered in XML sitemaps for AI discovery. This post covers the rules that guide leaves out.
Several Sitemap lines or one sitemap index?
Both work. Google's robots.txt specification says you can specify multiple sitemap fields with no limit on how many, and the sitemaps.org protocol also allows more than one per robots.txt file. The choice is about maintenance, not support.
# Option A: one line per sitemap
Sitemap: https://www.example.com/sitemaps/pages.xml
Sitemap: https://www.example.com/sitemaps/products.xml
Sitemap: https://www.example.com/sitemaps/blog.xml
# Option B: one line pointing to an index
Sitemap: https://www.example.com/sitemap_index.xml| Question | Several Sitemap lines | One sitemap index |
|---|---|---|
| Adding a new sitemap file | Edit robots.txt | Regenerate the index; robots.txt stays the same |
| Search Console submission | Submit each file | Submit the index once |
| Files on other hosts | Allowed with cross-submission (next section) | Google requires the same site unless cross-site submission is set up |
| Nesting | Not applicable | Google says an index cannot list other index files |
| Risk | robots.txt grows and drifts from what the CMS generates | One broken index hides every child sitemap |
For most sites, one line pointing to the index your CMS or SEO plugin already generates is the simplest setup. Several lines make sense when sitemaps come from different systems, for example a CMS for pages and a separate service for a product catalogue, and nobody can merge them into one index.
Two index rules are worth knowing. Google's Sitemaps report help says a sitemap index cannot list other sitemap index files, only sitemap files, so do not point an index at another index. Google's large sitemaps page says the sitemaps an index references must be hosted on the same site as the index, unless you set up cross-site submission.
Can robots.txt point to a sitemap on another host?
Yes. Google's specification says the sitemap URL does not have to be on the same host as the robots.txt file. The rule that matters is which URLs that sitemap may list.
The sitemaps.org protocol normally limits a sitemap to URLs on its own host and protocol, in its own directory or below. Its "Sitemaps & Cross Submits" section adds an exception: if www.example.com/robots.txt points to a sitemap stored on another host, you have shown that you control www.example.com, and that sitemap may list URLs for www.example.com. Every URL in it must belong to the host whose robots.txt points to it.
A common setup is a sitemap generated by a separate service or stored on a CDN host:
# https://www.example.com/robots.txt
User-agent: *
Allow: /
Sitemap: https://sitemaps.example-cdn.com/www-example-com/sitemap.xmlThat file may list https://www.example.com/... URLs. It may not also list https://shop.example.com/... URLs, because shop.example.com has not pointed to it. Each host needs its own robots.txt, as explained in robots.txt for subdomains, protocols and ports, and each one should reference a sitemap of its own URLs:
# https://shop.example.com/robots.txt
User-agent: *
Allow: /
Sitemap: https://sitemaps.example-cdn.com/shop-example-com/sitemap.xmlGoogle's build and submit a sitemap page describes the same two routes for sitemaps kept in one place. With Search Console, verify ownership of every site, then submit the sitemaps or an index. With robots.txt, make one sitemap per site containing only that site's URLs, store them where you like, and make sure each site's robots.txt references its own sitemap. Microsoft announced support for the robots.txt method in a 2008 Bing post, which says all URLs in the sitemap must be within the same domain as the robots.txt file. That post is old, so check Bing Webmaster Tools for current behaviour.
Why a Sitemap line is ignored
When a crawler never fetches your sitemap, the cause is usually one of these. Each has a quick check.
| Problem | Example | Fix |
|---|---|---|
| Relative path | Sitemap: /sitemap.xml | Use the full URL: https://www.example.com/sitemap.xml |
| Wrong protocol or host | http://example.com/sitemap.xml on an https://www site | Point to the exact URL that returns 200, on the canonical host and protocol |
| Sitemap lists another host's URLs | A www sitemap that includes shop. URLs | One sitemap per host, each referenced from that host's robots.txt |
| Sitemap URL is disallowed | Disallow: /sitemaps/ above a sitemap in /sitemaps/ | Remove the rule or move the file; Google says crawlers may follow the line unless the URL is disallowed |
| Misspelt field name | Sitmap: or Site map: | Use Sitemap:; Google's parser tolerates site-map, but other parsers may not accept any variant |
| Index inside an index | An index listing sitemap_index_2.xml | List only sitemap files in an index |
| Line beyond the size limit | A very long robots.txt with the line at the end | Google ignores content after 500 KiB; put the line near the top |
| Sitemap returns HTML or 404 | An app shell served for /sitemap.xml | See Sitemap could not be fetched |
Some details behind the table:
- Case and spelling. Google's specification says the field name is case-insensitive but the value is case-sensitive. Google's open-source robots.txt parser also accepts
site-mapas a typo and matches any key that starts withsitemap, but that tolerance is Google's, not a rule other crawlers share.sitemap:works;https://www.example.com/Sitemap.xmland/sitemap.xmlare different URLs on most servers. - Redirects. If the URL in the line redirects, for example from
httptohttps, you rely on each crawler following it. Neither sitemaps.org nor Google's robots.txt specification describes how redirects on the sitemap URL itself are handled, so list the final URL. - Support is not universal. RFC 9309 does not define the Sitemap record. It lets crawlers interpret records outside the standard and says a Sitemaps record must not end a group, but leaves its meaning to the sitemap protocol. A crawler that follows RFC 9309 to the letter can ignore the line. Google and Bing document support; OpenAI, Anthropic and Perplexity do not say in their crawler documentation whether their bots read sitemaps, at the time of writing.
- The robots.txt itself must be valid. A file that returns an HTML page, a redirect loop or an error never reaches the Sitemap line. Common robots.txt mistakes covers those failures.
Platform-generated robots.txt files that drop or duplicate the line
Many sites do not write robots.txt by hand. The line then depends on which piece of software generates the file, and changing that software can remove it.
WordPress. WordPress serves a virtual robots.txt when no physical file exists. The core sitemap feature appends a line pointing to the sitemap index only when the site is set to public, as the WP_Sitemaps::add_robots() source shows. Once someone uploads a physical robots.txt, standard server setups serve that file instead and WordPress no longer generates one, so the automatic line disappears and you must add it yourself. Editing the WordPress robots.txt walks through both cases.
Yoast SEO. Yoast's robots.txt help page says that when you create a robots.txt with the plugin, it replaces the WordPress default with a block between # START YOAST BLOCK and # END YOAST BLOCK that includes a line for sitemap_index.xml. If you later switch SEO plugins or turn off the plugin's sitemap, check that the line still points to a file that exists.
Shopify. Shopify generates a default robots.txt, and its customization guide says that adding a robots.txt.liquid template to the theme replaces that generated output. The guide's Liquid for the default rules prints each group's sitemap with {{ group.sitemap }} when it is not blank. A custom template that rebuilds the groups without that block drops the line. The guide's example for an extra sitemap is a plain Sitemap: rule added outside the Liquid for the default rules.
The duplicate case is the mirror image. When a hand-written line, a plugin line and a core line all survive, the file can point to two indexes, one of them no longer maintained. Two lines are not an error, but a line that points to a stale or empty index sends crawlers to old URLs. Keep only lines whose targets you know are generated and current.
To see what is actually served, fetch the live file rather than reading the template:
curl -s https://www.example.com/robots.txt | grep -i '^sitemap:'Then confirm that each URL it prints returns 200 with an XML content type:
curl -s https://www.example.com/robots.txt | grep -i '^sitemap:' \
| sed -E 's/^[Ss][Ii][Tt][Ee][Mm][Aa][Pp]:[[:space:]]*//' | tr -d '\r' \
| while read -r url; do
curl -s -o /dev/null -w "%{http_code} %{content_type} $url\n" "$url"
doneHow to check which crawlers fetched the sitemap
Declaring a sitemap does not tell you who used it. Two sources do.
Search Console shows only what you submitted. Google's Sitemaps report help says the report lists only sitemaps submitted through the report or the API, and does not show sitemaps discovered through a robots.txt reference. If you rely on the robots.txt line alone, that sitemap does not appear in the report even when Googlebot reads it. Submit the same URL in the report if you want Google's fetch status and error list. Bing's July 2025 post lists both a robots.txt reference and submission in Bing Webmaster Tools as ways to get a sitemap found.
Server logs show everyone else. Search your access logs for requests to the sitemap paths and count them by user agent:
grep -E '"GET /(sitemap[^ ]*|sitemaps/[^ ]*) HTTP' /var/log/nginx/access.log \
| awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -20Adjust the path pattern to match the names you use. A sitemap hosted on another host or a CDN is logged there, not on your origin, so check that host's logs too. User-agent strings can be faked, so confirm that a request claiming to be Googlebot, Bingbot or an AI crawler really came from that operator before drawing conclusions; verifying real AI crawlers explains the reverse DNS and IP range checks. If an AI crawler never requests your sitemap, that is not a fault on your side: no AI vendor has documented that its crawler must use one.
Frequently asked questions
Does the Sitemap line have to go at the bottom of robots.txt?
No. The sitemaps.org protocol says it is independent of the user-agent lines, and Google says to insert it anywhere in the file. Placing it near the top keeps it well inside Google's 500 KiB limit.
Can I put a Sitemap line inside a User-agent group to target one crawler?
No. Google says the sitemap field is not tied to any user agent, so a line placed under User-agent: Googlebot still applies to every crawler that reads the file. There is no documented way to show a sitemap to one crawler only.
Can my robots.txt point to a sitemap on a different domain?
Yes. Google's specification allows a sitemap URL on another host, and the sitemaps.org cross-submission rules accept it as long as the sitemap lists only URLs from the host whose robots.txt points to it.
Why does Search Console not show the sitemap from my robots.txt?
Google's Sitemaps report lists only sitemaps submitted in the report or through the API. Sitemaps discovered through robots.txt are not shown, so submit the URL there if you want to see its status.
Do I still need the robots.txt line if I submit in Search Console and Bing?
It is still useful. Submission covers only Google and Bing, while the robots.txt line is available to any crawler that reads your file, including ones with no webmaster tool.
Check the file crawlers actually receive
A Sitemap line only helps if robots.txt itself is reachable and the URL it names returns a valid sitemap. The free check looks at a single page with no account needed, so it is a quick way to confirm the main AI crawlers can reach it. For a site with several hosts, generated robots.txt files or sitemaps stored elsewhere, an audit reviews robots.txt, sitemaps and crawler access together across up to 500 pages.
Filed under Access and robots.txt. Topics: robots-txt, sitemaps, google, bingbot