Skip to content

How to edit robots.txt in WordPress (and why it isn't a file)

How to edit robots.txt in WordPress: the virtual file, physical file precedence, Yoast and Rank Math editors, the robots_txt filter, and adding AI crawler rules.

By Crawlito · 6 min read

In short

  • WordPress generates robots.txt on the fly with do_robots() when no physical robots.txt exists, so there is no file to find in the site root until you create one.
  • A physical robots.txt in the web root takes precedence on standard Apache and nginx setups, because the server serves existing files before handing the request to WordPress.
  • Since WordPress 5.3, the Discourage search engines setting adds a noindex robots meta tag instead of writing Disallow: / to robots.txt.
  • Rank Math's robots.txt editor is virtual and does not work while a physical file exists; Yoast's file editor is under Yoast SEO, Tools, File editor.
  • Developers can change the virtual file with the robots_txt filter, which receives the output and whether the site is public.
On this page

WordPress robots.txt is usually virtual: when there is no robots.txt file in your site root, WordPress builds one on the fly each time it is requested. To edit it, either create a physical robots.txt file in the web root, use your SEO plugin's robots.txt editor, or change the output in code with the robots_txt filter. A physical file wins over the virtual one on standard server setups, which is the most common reason edits "don't show up".

This guide explains which version your site is serving, how each editing method works, and how to add rules for AI crawlers safely.

Why there is no robots.txt file in WordPress

When a request for /robots.txt reaches WordPress, core calls do_robots(), which prints a default file. In current WordPress the core output is:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

The /wp-admin/ path comes from admin_url(), so it differs if WordPress is installed in a subdirectory. Since WordPress 5.5, the built-in sitemap feature adds a line pointing to wp-sitemap.xml, but only when the site is public, as the WP_Sitemaps::add_robots() source shows. SEO plugins often replace that with their own sitemap line.

Because this output is generated, you will not find it over FTP or in your host's file manager. That surprises people who expect to open and edit a file.

Which robots.txt is your site serving?

Before changing anything, check what visitors and crawlers actually get:

bash
curl -si https://www.example.com/robots.txt

Then look in the web root (the folder that contains wp-config.php on most installs) for a file called robots.txt.

What you findWhat is served
No physical fileWordPress's virtual file, plus any plugin changes
A physical fileThe physical file, on standard setups; plugin editors that work on the virtual file have no effect
A physical file, but the output does not match itA CDN, host or caching layer is rewriting or caching robots.txt

The precedence comes from how WordPress is wired to the web server. WordPress's default .htaccess sends a request to index.php only if no matching file or folder exists:

apache
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

On nginx, the usual try_files $uri $uri/ /index.php?$args; line does the same. So if robots.txt exists on disk, the server returns it and WordPress never runs. Some managed hosts and plugins handle robots.txt differently, which is why the curl check comes first.

The Discourage search engines setting

Under Settings > Reading there is a checkbox labelled "Discourage search engines from indexing this site". Its effect changed in WordPress 5.3. The do_robots() changelog says that from 5.3 it no longer outputs Disallow: / when the setting is on; the job moved to a robots meta tag in the page HTML.

So today, with the box ticked:

  • The virtual robots.txt still shows the normal default rules.
  • Every page gets a noindex, nofollow robots meta tag.

Crawlers that honor noindex will drop your pages, and the robots.txt looks harmless while it happens. If pages are disappearing from search, check this setting before you check robots.txt. It is a common leftover after launching a site that was built on staging; see staging noindex leaks.

Note that a noindex meta tag only works if a crawler can fetch the page. If you also block a crawler in robots.txt, it never sees the noindex.

Method 1: create a physical robots.txt file

The simplest and most predictable option.

  1. Copy the current output from https://www.example.com/robots.txt into a text editor, so you keep anything plugins added.
  2. Make your changes.
  3. Save it as plain text named robots.txt, UTF-8 encoded.
  4. Upload it to the web root over SFTP or your host's file manager.
  5. Fetch the URL again and confirm the new content is served.

The trade-off: once a physical file exists, plugin editors that change the virtual file stop affecting the output, and WordPress no longer adds the sitemap line for you. Add Sitemap: lines yourself.

Method 2: use your SEO plugin's editor

Both major SEO plugins have an editor, and they behave differently.

Yoast SEO. Yoast's robots.txt help page puts the editor at Yoast SEO > Tools > File editor, with a "Create robots.txt file" button if none exists. Yoast notes the menu does not appear if file editing is disabled on your install, in which case you edit the file over FTP instead.

Rank Math. Rank Math's guide puts the editor at Rank Math SEO > General Settings > Edit robots.txt, with Advanced Mode enabled. Rank Math's editor works on the virtual file, and it says a physical robots.txt takes precedence: you have to delete the physical file before its editor has any effect.

If you run either plugin and your edits do not appear, the cause is almost always one of these: a physical file overriding a virtual editor, a page cache or CDN serving an old copy, or a second plugin also filtering robots.txt. Yoast and Rank Math AI crawler settings covers the plugins' other crawler options.

Method 3: change the virtual file with the robots_txt filter

For developers, the robots_txt filter receives the generated output and a boolean for whether the site is public. It has existed since WordPress 3.0. Put the code in a small must-use plugin rather than a theme, so a theme change does not remove your rules:

php
<?php
// wp-content/mu-plugins/robots-rules.php

add_filter( 'robots_txt', function ( $output, $public ) {
    if ( ! $public ) {
        return $output;
    }

    $output .= "\n# Opt out of AI model training, keep AI search\n";
    $output .= "User-agent: GPTBot\n";
    $output .= "User-agent: ClaudeBot\n";
    $output .= "User-agent: CCBot\n";
    $output .= "Disallow: /\n";

    return $output;
}, 99, 2 );

The priority of 99 runs it after most plugins, so your lines are appended to their output. The filter only affects the virtual file. If a physical robots.txt exists, this code does nothing.

Adding AI crawler rules without breaking the rest

Whichever method you use, the rules work the same way. Two points cause most mistakes.

A named group does not inherit the * rules. Under RFC 9309 a crawler obeys only the group that names it. If you add User-agent: GPTBot with its own rules, GPTBot stops reading your User-agent: * rules, including Disallow: /wp-admin/. Repeat every general rule inside each named group. robots.txt groups explained shows how this goes wrong.

Training and search crawlers are separate. Blocking GPTBot or ClaudeBot opts out of model training; it does not remove you from ChatGPT search or Claude's search, which use OAI-SearchBot and Claude-SearchBot. A complete file that opts out of training and keeps search open:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Sitemap: https://www.example.com/wp-sitemap.xml

Replace the sitemap URL with the one your SEO plugin generates, if you use one. Allow AI search but block AI training explains each token.

After you edit: clear caches and recheck

  1. Purge your page cache plugin and your CDN for /robots.txt. Some caching setups store it like any other URL.
  2. Fetch the file again with curl and compare it line by line with what you intended.
  3. Check that the response is 200 and the content type is text/plain.
  4. Allow for crawler caching. Google generally caches robots.txt for up to 24 hours, and OpenAI says its search systems can take about 24 hours to adjust.

If a security plugin or firewall blocks crawlers before they reach robots.txt at all, rules in the file do not matter. WordPress security plugins that block crawlers covers that case.

Frequently asked questions

Where is the robots.txt file in WordPress?

Usually nowhere. WordPress generates it virtually when no physical file exists. If someone created one, it is in the web root next to wp-config.php on most installs.

Why don't my Yoast or Rank Math robots.txt changes show up?

Check for a physical robots.txt in the web root, which overrides Rank Math's virtual editor on standard setups, and then purge your page cache and CDN. A second plugin filtering robots.txt can also overwrite your changes.

Does "Discourage search engines" block crawlers in robots.txt?

Not since WordPress 5.3. It adds a noindex robots meta tag to pages instead, and the virtual robots.txt keeps its normal rules.

Can I edit robots.txt without a plugin?

Yes. Upload a plain text robots.txt to the web root, or add a robots_txt filter in a must-use plugin if you want to keep the virtual file.

Check what your WordPress site serves

The robots.txt you think you have and the one crawlers get can differ because of plugins, caches and physical files. To see your live rules and how AI crawlers are treated on one of your pages, run the free check.

Filed under Platform guides. Topics: wordpress, robots-txt, ai-crawlers, troubleshooting