XML sitemaps for AI discovery: what to list and how to submit

An XML sitemap is a file that lists the URLs on your site you want crawled, with an optional date for when each one last changed. It does not make a page indexable or citable on its own, but it hands crawlers a complete, current list of addresses so they do not depend on finding every page through links. For AI search it matters because Google and Bing, whose indexes feed AI Overviews and Microsoft Copilot, both read sitemaps, and a list of live, canonical URLs gives any other crawler that reads it fewer dead ends.

The rules are short. Use absolute URLs, list only canonical pages that return 200, keep lastmod honest, stay under the size limits, and point to the sitemap from robots.txt.

How the sitemap protocol works

The format is defined at sitemaps.org and supported by Google, Bing and others. A minimal file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/</loc>
  </url>
  <url>
    <loc>https://www.example.com/pricing</loc>
    <lastmod>2026-10-21</lastmod>
  </url>
  <url>
    <loc>https://www.example.com/blog/shipping-rates?region=eu&amp;currency=eur</loc>
    <lastmod>2026-11-02T09:30:00+00:00</lastmod>
  </url>
</urlset>

What each part does:

A few protocol rules catch people out:

• The file must be UTF-8, and five characters must be escaped inside values: & as &, ' as ', " as ", > as > and < as <. An unescaped & in a query string makes the whole file invalid XML.
• Same host, same protocol, same directory or below. A sitemap at https://www.example.com/sitemap.xml cannot list https://shop.example.com/ or http://www.example.com/ URLs. A sitemap at /blog/sitemap.xml can only list URLs under /blog/. Invalid URLs are dropped. Put the sitemap at the site root to cover everything.
• Absolute URLs only. Google says to use fully qualified URLs, because it will try to crawl them exactly as listed. /pricing on its own is not a valid entry.

A sitemap only says what you want crawled; leaving a URL out does not exclude it. The protocol points to robots.txt or a robots meta tag for content you do not want indexed, as covered in robots.txt for AI crawlers.

Which URLs belong in a sitemap

A sitemap is a list of the URLs you want treated as the real addresses of your content. Google notes that it generally shows canonical URLs in results, and advises listing only the URL you prefer when the same content is reachable at several addresses. Every entry should pass four tests:

1. It returns 200 directly. No redirects, no 404 or 410, no 5xx. A URL that redirects belongs in the sitemap only as its final destination. See status codes, redirects and AI crawlers for what each code tells a crawler.
2. It is the canonical version. If the page's <link rel="canonical"> points somewhere else, list that other URL instead. Pick one host (with or without www), one protocol (https) and one trailing slash convention, and use it everywhere. Canonical tags and duplicate URLs explains how the two signals interact.
3. It is indexable. A URL marked noindex or blocked in robots.txt sends a contradictory signal when it also appears in the sitemap.
4. It has real content. Leave out internal search results, filtered or sorted listing variants, cart and account pages, and thin tag archives.

What this looks like in practice:

There is a practical reason to be strict. In its December 2024 analysis of crawler traffic on nextjs.org and the Vercel network, Vercel and MERJ found that 34.82% of ChatGPT's fetches and 34.16% of Claude's hit 404 pages, against 8.22% for Googlebot, and recommended keeping sitemaps up to date alongside proper redirects and consistent URL patterns. The figures are a snapshot from one network, but a sitemap full of stale URLs adds to the problem rather than fixing it.

Keep lastmod accurate, or leave it out

lastmod is the only optional field Google and Bing say they use, and both say it must reflect when the content changed. The protocol itself says the date must be the date the page was last modified, not when the sitemap was generated.

• Google uses lastmod only if it is consistently and verifiably accurate. Its 2023 announcement gives changes to the main text, structured data or links as examples of significant changes, while a small edit to a sidebar or footer is not one. It also says it is fine to leave out lastmod on pages where the date is hard to pin down, such as a homepage or category pages.
• Bing, in a July 2025 post, calls lastmod a key signal, recommends ISO 8601 dates with a time component, and warns against setting it to the time the sitemap was generated.

The common failure is a generator that stamps every URL with the build time. On a site that rebuilds nightly, every page claims to have changed every day, so the field carries no information and Google learns to ignore it. Take lastmod from the content itself: the article's updated date, the product's last edit in the CMS, or the source file's commit date. Do not set it to today's date on every deploy.

changefreq and priority can stay or go. Google says it does not use either, and Bing says it ignores both.

Size limits and the sitemap index

The protocol limits are the same for every engine that follows it:

The old 10 MB limit is long gone, and gzip does not raise the 50 MB ceiling. Once you approach either limit, split the URLs across several sitemaps and list them in an index:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemaps/pages.xml</loc>
    <lastmod>2026-11-01</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemaps/products-1.xml.gz</loc>
    <lastmod>2026-11-08</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemaps/blog.xml</loc>
    <lastmod>2026-11-07</lastmod>
  </sitemap>
</sitemapindex>

In an index, lastmod is the date the child sitemap file changed. Splitting by section (pages, products, blog) is worth doing even on smaller sites: Search Console and Bing Webmaster Tools report on each submitted file, so a drop in indexed URLs points straight to the section that caused it.

Most platforms already produce a sitemap or an index. WordPress has shipped a core sitemap index at /wp-sitemap.xml since version 5.5 and adds it to the virtual robots.txt when the site is public; SEO plugins often replace it with their own. Shopify, Wix, Squarespace and Webflow generate one at /sitemap.xml. In Next.js, an app/sitemap.ts file returns the list. Whatever produces it, check the output against the rules above rather than trusting the defaults.

Declare it in robots.txt and submit it

There are two documented ways to tell crawlers where a sitemap is: a Sitemap line in robots.txt and direct submission in a webmaster tool. Do both.

Add a Sitemap line to robots.txt

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap_index.xml

The line takes a full URL and, per the protocol, is independent of the user-agent lines, so it can go anywhere in the file and applies to every crawler that reads it. You can list several. Google supports the sitemap field in its robots.txt specification, and Common Crawl says CCBot uses any sitemap announced in robots.txt. This is the one method that reaches crawlers without a webmaster tool. OpenAI, Anthropic and Perplexity do not say in their crawler documentation whether their bots read sitemaps, so robots.txt is the place to make the file available to anyone who looks.

Submit to Google and Bing

• Google Search Console: open the Sitemaps report, enter the sitemap or index URL and submit. Google says this lets you see when Googlebot accessed the sitemap and any processing errors. Google is clear that a sitemap is a hint and does not guarantee crawling.
• Bing Webmaster Tools: submit the same URL under Sitemaps. Bing's July 2025 post says it tries to fetch a submitted sitemap immediately and then rechecks it, typically at least once a day. Bing's index also grounds Microsoft Copilot, which is why Bingbot still matters for AI search.

Do not rely on the old ping URLs. Google deprecated its sitemap ping endpoint in June 2023, said it would stop working six months later, and pings to it return 404; Bing removed anonymous sitemap submission in 2022. Plugins that still ping do no harm, but they do nothing either.

IndexNow for changes between crawls

IndexNow lets you tell participating search engines that a URL was added, updated or deleted, instead of waiting for them to reread your sitemap. You host a key file at your site root, then send changed URLs:

curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "host": "www.example.com",
    "key": "a1b2c3d4e5f6a7b8",
    "keyLocation": "https://www.example.com/a1b2c3d4e5f6a7b8.txt",
    "urlList": [
      "https://www.example.com/pricing",
      "https://www.example.com/blog/new-post"
    ]
  }'

A 200 means the submission was accepted and 202 means it was received while the key is checked; 403 means the key is not valid. One POST takes up to 10,000 URLs, and engines that adopt IndexNow share submissions with each other, so one endpoint is enough. At the time of writing, the participant list includes Bing, Yandex, Seznam, Naver, Yep, Internet Archive and Amazonbot. Google is not on it.

Treat IndexNow as a complement. The IndexNow FAQ says a submission does not guarantee indexing and asks you to submit only URLs that actually changed. Keep the sitemap as the complete list and use IndexNow for the deltas.

How to check your sitemap in five minutes

Fetch the sitemap and confirm it returns a 200 with an XML content type (application/xml or text/xml), not an HTML error page:

curl -sI https://www.example.com/sitemap.xml

Count the entries and, for a gzipped file, check the uncompressed size:

curl -s https://www.example.com/sitemap.xml | grep -o '<loc>' | wc -l
curl -s https://www.example.com/sitemaps/products-1.xml.gz | gunzip | wc -c

Then list every entry that does not return 200:

curl -s https://www.example.com/sitemap.xml \
  | grep -oE '<loc>[^<]+</loc>' | sed -E 's#</?loc>##g; s#&amp;#\&#g' \
  | while read -r url; do
      code=$(curl -s -o /dev/null -w '%{http_code}' "$url")
      [ "$code" != "200" ] && echo "$code $url"
    done

Any output is a URL to fix or remove. If the file is a sitemap index, the <loc> entries are child sitemaps, so run the same check on each of them. For a sample of the remaining URLs, check that the page's canonical matches the listed address and that there is no noindex. Finally, compare the sitemap with what a crawl of your links finds: pages that are linked but missing from the sitemap, and sitemap URLs that nothing links to, both deserve a look. Internal links and broken links covers the link side. If you also publish an llms.txt file, keep in mind it is a curated overview, not a replacement for the full sitemap.

Frequently asked questions

Do AI crawlers like GPTBot or ClaudeBot read XML sitemaps?

OpenAI, Anthropic and Perplexity do not document whether their crawlers read sitemaps at the time of writing. Common Crawl says CCBot uses sitemaps declared in robots.txt, and Google and Bing, whose indexes feed AI Overviews and Copilot, both read them. A Sitemap line in robots.txt makes the file available to any crawler that looks.

Should I include noindex or redirected pages in my sitemap?

No. List only canonical URLs that return 200 and are meant to be indexed. Redirects, 404s, noindex pages and duplicates waste crawls and send signals that contradict the page itself.

Does lastmod need to be updated every time the sitemap is generated?

No. lastmod should change only when the page's main content, structured data or links change. Google ignores the field on sites where it is not consistently accurate, so a build timestamp on every URL makes it useless.

What happens if my sitemap has more than 50,000 URLs?

The file is outside the protocol limit, so do not rely on engines reading all of it. Split the URLs into several files, each under 50,000 URLs and 50 MB uncompressed, and list them in a sitemap index.

Does submitting a sitemap guarantee my pages get indexed?

No. Google calls a sitemap a hint, and IndexNow makes no promise of indexing either. A sitemap helps crawlers find your URLs; whether they are crawled, indexed or used in AI answers depends on access, content and many other factors.

Check what crawlers can actually reach

A sitemap only helps if the URLs in it can be fetched and read. The free check looks at a single page with no account needed. A full audit runs deterministic checks across up to 500 pages and delivers a PDF with the evidence and prioritized fixes.