Skip to content

Joomla robots.txt: edit the file and set AI crawler rules

Where Joomla robots.txt lives, what robots.txt.dist is for, how to add AI crawler groups without losing the defaults, and how the Robots setting differs.

By Crawlito · 8 min read

In short

  • Joomla's robots.txt is a real file in the site root that you edit by FTP or a file manager, not a setting in the administrator.
  • Joomla ships the file as robots.txt.dist, and the installer renames it to robots.txt only when no robots.txt exists, so a custom file is not replaced that way.
  • The default file has one User-agent: * group with 15 Disallow rules, and a crawler you give its own group ignores all 15 unless you copy them in.
  • If Joomla runs in a subfolder, robots.txt must move to the domain root and every path needs the folder prefix, such as /joomla/administrator/.
  • The Robots option in Global Configuration, menu items and articles writes a meta robots tag on pages, which controls indexing, not crawling.
On this page

Joomla's robots.txt is a physical file in the folder where Joomla is installed, so you change it by editing that file over FTP, SFTP or your host's file manager. To add AI crawler rules, add groups for the crawlers you name, and copy Joomla's default Disallow lines into each of them: a crawler that finds a group naming it ignores the User-agent: * group completely. The Robots option in Joomla's administrator is a different control. It writes a meta robots tag into pages and does not touch robots.txt.

This guide covers the parts that are specific to Joomla: the file and its .dist twin, the default rules, the subfolder trap, the Robots setting, and how to add AI crawler groups safely. For the general syntax and every crawler's token, see the robots.txt guide for AI crawlers. Facts about Joomla's behaviour come from the Joomla documentation and the joomla-cms repository, read on 15 November 2026 on the 6.2-dev branch.

Where Joomla keeps robots.txt and what robots.txt.dist is

Joomla's documentation says the file is included in the Joomla root and must be called robots.txt. There is no editor for it in the administrator, so open it with whatever you use to reach the site's files.

The source code does not contain a robots.txt at all. It contains robots.txt.dist, and the installer's final clean-up step renames it. The code in installation/src/Model/CleanupModel.php is explicit:

php
// Rename the robots.txt.dist file if robots.txt doesn't exist
if (!file_exists(JPATH_ROOT . '/robots.txt') && file_exists(JPATH_ROOT . '/robots.txt.dist')) {
    try {
        File::move(JPATH_ROOT . '/robots.txt.dist', JPATH_ROOT . '/robots.txt');
    } catch (FilesystemException $e) {
        return false;
    }
}

Two consequences follow:

  • The rename happens once, and only if no robots.txt is there. If you upload your own file before installing, Joomla leaves it alone.
  • Updates carry robots.txt.dist, not robots.txt. The release build script in build/build.php force-adds robots.txt.dist to the diff packages it builds for updates, and there is no robots.txt in the package to write. After an update you may find a fresh robots.txt.dist next to your file. That copy is a reference for comparing your rules with the new default; crawlers never read it.

This is the opposite of the Drupal problem, where a build step can regenerate the public file during a Composer update (see Drupal Composer overwrites robots.txt). On Joomla, the risk is usually the reverse: an old custom file that never picks up changes to the default, or a hand edit that someone made on the server and nobody tracked.

A quick way to see whether your file has drifted from the current default:

bash
diff robots.txt robots.txt.dist

Run it in the Joomla root after an update. Lines starting with > are in the new default but not in your live file.

What the default Joomla robots.txt blocks

At the time of writing, robots.txt.dist on the 6.2-dev branch contains a comment block about subfolder installs, then one group for all crawlers:

User-agent: *
Disallow: /administrator/
Disallow: /api/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

These are code and system folders, not content. Your articles live at the URLs Joomla's router builds, which do not start with any of these paths. Three folders are not in the list:

FolderHoldsIn the default Disallow list?
/media/CSS, JavaScript and other extension assetsNo
/images/Images uploaded through the Media ManagerNo
/templates/Template filesNo

That matters because crawlers that render pages need the CSS and JavaScript to see the layout. If an older file on your site still disallows /media/ or /images/, compare it with the current robots.txt.dist and decide whether the rule is still wanted. The guide to blocked CSS and JavaScript explains what crawlers lose when those paths are blocked.

The subfolder install trap

RFC 9309 says the rules must be in a file named /robots.txt in the top-level path of the service. A crawler only ever requests https://example.com/robots.txt. If Joomla is installed at https://example.com/joomla/, the file Joomla created at /joomla/robots.txt is never read.

The comment at the top of Joomla's own file says what to do: move the file to the site root and prefix every path with the folder name.

User-agent: *
Disallow: /joomla/administrator/
Disallow: /joomla/api/
Disallow: /joomla/cache/
Disallow: /joomla/tmp/

Repeat the prefix for all 15 default rules. If something else already lives at the root, such as another CMS or a landing page, it may already have a robots.txt. Merge the Joomla rules into that file instead of replacing it, because there can only be one per host.

How to check: request the root file and the subfolder file, and confirm the root one returns 200 with your Joomla rules in it.

bash
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/robots.txt
curl -s https://example.com/robots.txt | grep -i "joomla/administrator"

How to add AI crawler groups without losing Joomla's defaults

The usual mistake is to append a group like this to the default file:

User-agent: GPTBot
Disallow: /

That part works. The trouble starts when you write a named group to allow a crawler, or to block only part of the site. Under RFC 9309, a crawler that finds a group naming it uses that group, and only falls back to * when nothing names it. So this file lets OAI-SearchBot fetch /administrator/, /cache/ and the rest:

User-agent: *
Disallow: /administrator/
Disallow: /cache/
# ...the other default rules

User-agent: OAI-SearchBot
Allow: /

The fix is to give the named crawlers the same list. You can put several User-agent lines above one set of rules. Here is a complete Joomla file that blocks the main training crawlers and keeps the defaults for everyone else, including AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, which fall through to *:

# Training crawlers: no access
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including AI search crawlers: Joomla defaults
User-agent: *
Disallow: /administrator/
Disallow: /api/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

Sitemap: https://www.example.com/sitemap.xml

If you do want a named group for a search crawler, for example to keep it out of one section, copy all 15 default lines into that group and add your extra rule. Which crawlers to block is a separate decision; allow AI search, block AI training goes through each operator's tokens and their trade-offs. Only keep the Sitemap: line if your site actually publishes a sitemap at that URL.

After saving, read the file as each crawler would. The robots.txt testing guide shows how to run Google's open-source parser against your file one user agent and URL at a time.

What the Robots setting in Joomla actually does

Joomla has a Robots field in three places. The field and its options come from the form definitions in the repository:

WherePath in the administratorOptions
Site-wideSystem > Global Configuration > Site, Metadata sectionindex, follow (default); noindex, follow; index, nofollow; noindex, nofollow
Menu itemThe menu item's Metadata tabUse Global, plus the four values
ArticleThe article's Publishing tab, with the other metadata fieldsUse Global, plus the four values

These values become a <meta name="robots"> tag in the page's HTML. The default site-wide value is empty, shown as "index, follow", and Joomla's article view only sets the tag when a value is present, so an article page on a default site outputs no robots meta tag. The article view also applies the article's own metadata after the menu item's settings, so a value set on the article wins for that page. One more built-in case: the print view of an article always gets noindex, nofollow.

The difference from robots.txt is the one that trips people up:

  • robots.txt decides whether a crawler may fetch a URL.
  • The Robots meta tag tells a crawler that has already fetched the page whether to index it and follow its links.

A crawler blocked in robots.txt never sees the meta tag, so combining a Disallow with noindex on the same page does not remove it from an index. Google's noindex documentation says the page must not be blocked by robots.txt for the rule to work. If the goal is to keep a page out of search results, leave it crawlable and set the Robots field to noindex. The robots.txt vs meta robots guide covers this in more depth. Setting Global Configuration to noindex, nofollow on a live site, for example a value left over from a staging copy, affects every page that does not override it, so check that field after any migration.

How to check what crawlers get from your Joomla site

  1. Fetch the live file from every host you serve, with and without www, and confirm a 200 and text/plain:

    bash
    curl -sI https://www.example.com/robots.txt
  2. Compare it with the default using the diff command above, so you know which rules are yours and which are Joomla's.

  3. Look for named groups and check that each one contains the default Disallow lines, or deliberately does not.

  4. Check the meta robots tag on a few key pages:

    bash
    curl -s https://www.example.com/your-article | grep -i 'name="robots"'

    No output means no tag, which is what an article page shows with Joomla's default settings. Any noindex you did not expect points to Global Configuration, the menu item or the article.

  5. Fetch the file as an AI crawler to rule out a firewall or security extension that treats bots differently: curl -s -A "GPTBot" https://www.example.com/robots.txt. A block page or challenge instead of rules is a server problem, not a robots.txt one.

Frequently asked questions

Can I edit robots.txt from the Joomla administrator?

Not in core Joomla at the time of writing. The documentation describes it as a file in the Joomla root, so you edit it over FTP, SFTP or a hosting file manager. Whatever tool you use, the file crawlers read is the one in the root.

Will a Joomla update overwrite my robots.txt?

The update packages include robots.txt.dist, not robots.txt, so an update does not write over your file. The rename from .dist happens in the installer at a fresh install, and only when no robots.txt exists. Keep a backup of your file anyway, so an accidental overwrite during a manual file upload is easy to undo.

Should I delete robots.txt.dist?

Crawlers do not read it, so leaving it is harmless. It is useful as the current default to compare against after an update.

Does setting Robots to noindex block AI crawlers?

No. It asks crawlers that fetch the page not to index it, and it does not stop the fetch. To stop a crawler from requesting pages, use a robots.txt group for its token, or a server rule if you need enforcement.

Next step

After you change the file, run the free check on an important page to see whether the main AI crawlers are allowed by your robots.txt and whether a meta robots tag or a firewall gets in the way. For agencies running several Joomla sites, an audit reviews robots.txt, meta robots and edge settings together, with the evidence for each finding.

Filed under Platform guides. Topics: robots-txt, ai-crawlers, meta-robots, joomla