Skip to content

Odoo robots.txt: edit it and control the website sitemap

Where to edit Odoo robots.txt, how your custom lines join the generated file, how the 12-hour sitemap cache works, and how the Indexed switch removes pages.

By Crawlito · 8 min read

In short

  • In Odoo 19 you edit robots.txt with the Edit robots.txt button in Website > Configuration > Settings, which the 19.0 source shows only in developer mode; your text is appended below a generated User-agent: * group and Sitemap line.
  • Lines you add without a User-agent line of their own join Odoo's User-agent: * group, so put shared Disallow rules first and named AI crawler groups after them.
  • Odoo generates /sitemap.xml automatically, caches it for 12 hours and splits it into a sitemap index at 45,000 URLs per file.
  • Turning off the Indexed switch in a static page's properties adds a noindex robots meta tag and removes the page from the sitemap; Odoo says the option is not yet available for dynamic pages.
  • If the website's Domain field is set and does not match the host a crawler uses, Odoo serves Disallow: / on that host, leaves out your custom lines and adds noindex to every page.
On this page

To edit robots.txt in Odoo, go to Website > Configuration > Settings, scroll to the SEO settings and click Edit robots.txt; in Odoo 19 the button only appears in developer mode. You are not replacing the file: Odoo keeps generating its own User-agent: * group and Sitemap line, and appends whatever you type below them. The sitemap is generated for you at /sitemap.xml, refreshed every 12 hours, and the way to take a page out of it is the Indexed switch in the page's properties.

This guide follows Odoo's SEO documentation for version 19.0 and the Odoo 19.0 source code on GitHub, read on 15 November 2026. Older versions may differ in labels and details. For the general rules of robots.txt and sitemaps, see robots.txt for AI crawlers and XML sitemaps for AI discovery; this post covers what is specific to Odoo.

Where to edit robots.txt in Odoo

Odoo's documentation says it creates one robots.txt automatically, served at /robots.txt on your website (for example mydatabase.odoo.com/robots.txt on Odoo Online). To add your own instructions:

  1. Activate developer mode: in the Settings app, scroll to Developer Tools and click Activate the developer mode, or add ?debug=1 to the URL.
  2. Open the Website app and go to Configuration > Settings.
  3. If you run several websites, pick the right one in the website selector at the top of the settings page.
  4. Scroll to the SEO settings and click Edit robots.txt.
  5. Type your lines in the pop-up and click Save.

Two details come from the 19.0 source rather than the documentation. The Robots.txt setting is restricted to the technical features group, so it is hidden until developer mode is on, and it sits in the block titled Tracking & SEO (the SEO page calls it the SEO section). Earlier versions such as 18.0 show the button without developer mode. The text is stored on the website record, which is why each website in a multi-website database has its own custom block, and only users with website designer rights can change it.

Odoo's documentation adds two warnings worth repeating. robots.txt stops crawling, but it does not guarantee a page stays out of search results, because a blocked URL can still be indexed from links elsewhere. And reputable bots follow robots.txt while others may not, so Odoo points to Cloudflare on a custom domain for blocking those.

What the generated file looks like

In the Odoo 19.0 source, the website.robots template builds the file. On a host that matches your website's domain, the output has this shape:

User-agent: *
Sitemap: https://www.example.com/sitemap.xml


##############
#   custom   #
##############

(your lines from Edit robots.txt)

The template can also print Allow: lines under User-agent: * for routes that installed modules declare, so your live file may have a few more lines at the top. The point is the order: Odoo's group comes first, the Sitemap line second, the comment banner third and your text last.

That order has one consequence you need to plan for. Under RFC 9309, a group runs from its User-agent lines until the next User-agent line, and a Sitemap line does not end it. Comment lines starting with # are ignored. So a rule you type with no User-agent line of its own becomes part of Odoo's User-agent: * group. That is exactly how the documentation's own example works: adding Disallow: /about-us keeps all crawlers that use the * group away from /about-us.

It also means a rule you type after a named group belongs to that named group, not to everyone. Put the rules meant for all crawlers at the top of your custom block, then the named groups.

AI crawler groups for an Odoo shop

Here is a custom block for an Odoo eCommerce site that keeps all crawlers out of cart, checkout and customer portal pages, allows AI search and user-triggered fetchers, and opts out of AI training. The paths are examples from Odoo's routes (/shop/cart, /shop/payment, /shop/confirmation and the /my portal); check the URLs on your own site before copying.

Disallow: /shop/cart
Disallow: /shop/payment
Disallow: /shop/confirmation
Disallow: /my

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /shop/cart
Disallow: /shop/payment
Disallow: /shop/confirmation
Disallow: /my

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Read it from the crawler's side:

  • The first four lines have no User-agent line, so they join Odoo's generated User-agent: * group. Googlebot, Bingbot and every crawler without a named group follow them.
  • The search and user-triggered group repeats the same Disallow lines. A crawler that finds a group naming it follows only that group and ignores *, so without the repeat these crawlers could fetch cart and portal URLs. OpenAI and Perplexity say robots.txt rules may not apply to their user-triggered fetchers (ChatGPT-User, Perplexity-User), because a person asked for the page. The robots.txt groups guide explains the selection rule.
  • The training group blocks the crawlers and opt-out tokens used for model training. The allow AI search, block AI training guide covers the reasoning and the trade-offs.

If you want AI crawlers treated like everyone else, you do not need named groups at all: the rules under * already apply to them.

How the Odoo sitemap works

Odoo's documentation says it generates /sitemap.xml with all URLs and, for performance reasons, caches the file and updates it every 12 hours. When a site has many pages, Odoo produces a sitemap index that follows the sitemaps.org protocol, with 45,000 URLs per file. The source code matches: LOC_PER_SITEMAP = 45000 and SITEMAP_CACHE_TIME of 12 hours in the website controller. That stays under the protocol's 50,000-URL limit per file.

Each entry gets fields computed automatically:

ElementHow Odoo fills it
<loc>The page URL on the host that served the sitemap
<lastmod>The last change to the related record, for example the product or the page
<priority>Set by modules; for static pages, the page's priority field normalized, with 16 as the default

A few behaviours from the 19.0 source are useful when a URL is missing or unexpected:

  • The cache is per website and per domain. Each host gets its own cached sitemap, stored as an attachment. A page you publish now may not appear until the cache expires, up to 12 hours later.
  • Static pages are listed only when published, public and indexed. The query filters on the Indexed flag, the published state, an empty visibility restriction and a publish date in the past.
  • Shop pages follow the eCommerce access setting. When the shop is limited to logged-in users, the /shop category and product URLs are left out of the sitemap.

The Sitemap line in robots.txt is written for you, using the host the request came in on, so you do not need to add one in the custom block. Adding a second one pointing to the same file does no harm, but it is redundant.

Remove a page from the sitemap with the Indexed switch

To keep a static page out of the sitemap and out of search results, open the page, go to Website > Site > Properties and turn off Indexed. Odoo's page properties documentation describes the switch as the way to keep a page out of search engine results. Two things happen at once:

html
<meta name="robots" content="noindex"/>

The page's HTML gets that tag, and the page drops out of /sitemap.xml at the next regeneration.

Odoo's SEO page notes that this option is not yet available for dynamic pages, meaning pages Odoo generates from records, such as products, /shop and blog posts. The other methods on the same page are a 404 through a redirection, a 403 by restricting the page's visibility or unpublishing it, and a removal request in Google Search Console.

Choose the tool by what you want:

GoalUse in Odoo
Stop crawlers fetching a sectionA Disallow line in Edit robots.txt
Keep a static page out of search results and the sitemapTurn off Indexed
Remove a product or blog postUnpublish it, or redirect it

Do not combine a Disallow with a noindex for the same page: a crawler that cannot fetch the page never sees the tag. The robots.txt vs meta robots guide covers why. The source also adds noindex to paginated listing pages after page 1, so /shop/page/2 and similar URLs are not meant to be indexed.

Multi-website setups and the Domain field

Odoo's multi-website documentation says most website settings are website-specific, and that you select the website at the top of the settings page before changing them. robots.txt is one of them: each website has its own custom block, and each host gets its own sitemap.

The Domain field in the website settings matters more than it looks. In the 19.0 website.robots template, if a domain is set and the requested host does not match it (Odoo ignores www. and http versus https when comparing), the file becomes:

User-agent: *
Disallow: /
Sitemap: https://www.example.com/sitemap.xml

On that host your custom lines are not printed, and the page template adds <meta name="robots" content="noindex"/> to every page. In practice this keeps a second hostname, such as mydatabase.odoo.com, out of search once your real domain is set, and Odoo's documentation uses the same mechanism to keep test databases out of search: enter a random value in the Domain field.

The risk is a wrong value on a live site. If the field holds a typo, an old domain or a staging address, every crawler, AI crawlers included, is told Disallow: / on your real domain. The staging noindex leaks guide covers the reverse case, where a test setting reaches production.

How to check your Odoo robots.txt and sitemap

  1. Fetch the live file from the exact host crawlers use:
bash
curl -s https://www.example.com/robots.txt
  1. Confirm you see the custom banner and your lines. If you see Disallow: / and no banner, compare the Domain field in the website settings with the host you fetched.
  2. Check that each named AI crawler group repeats the Disallow lines it should follow.
  3. Fetch /sitemap.xml, and if it is an index, one of the child files. Look for a page you just set to not indexed; allow up to 12 hours before expecting it to disappear.
  4. View the source of a page you turned off and search for noindex.
  5. In a multi-website database, repeat these steps on every domain.

Frequently asked questions

Can I replace Odoo's robots.txt completely?

Not from the settings screen. Edit robots.txt, which in Odoo 19 appears only in developer mode, adds lines below the generated User-agent: * group and Sitemap line; it does not remove them.

Why does my Odoo robots.txt say Disallow: /?

The host you fetched does not match the website's Domain field, so Odoo blocks that host and adds noindex to its pages. Fix the domain value, or fetch the file from the domain it names.

How do I force Odoo to rebuild the sitemap?

Odoo documents only that the file is cached and updated every 12 hours. There is no documented button to regenerate it, so wait for the cache to expire.

Does the Indexed switch work on product pages?

Odoo says the option is not yet available for dynamic pages such as products. Unpublish or redirect the product instead.

Do AI crawlers read Odoo's sitemap?

Crawlers that read the Sitemap line in robots.txt can find it, since Odoo writes that line automatically. Each operator decides how it uses sitemaps, and a sitemap does not guarantee crawling or citation.

Check what crawlers see on your Odoo site

After editing robots.txt or the Indexed switches, run the free check on a product or landing page to see whether the main AI crawlers can fetch it. For a shop with several websites or domains, an audit reviews robots.txt, the sitemap and noindex signals across the site with evidence for each finding.

Filed under Platform guides. Topics: robots-txt, sitemaps, ecommerce, ai-crawlers, odoo