Skip to content

Yoast and Rank Math settings for AI crawlers and llms.txt

Which Yoast SEO and Rank Math settings affect AI crawlers: llms.txt generation, robots.txt editors, bot blocking and crawl options, with where to find each one.

By Crawlito · 7 min read

In short

  • Yoast SEO generates llms.txt from Settings, Site features, under AI tools, and Rank Math from its LLMS Txt module; both are switches you turn on, and neither changes how crawlers can access your pages.
  • Yoast's Block unwanted bots setting blocks only Google AdsBot in the free plugin; Premium adds toggles for Google-Extended, GPTBot and CCBot, which are training controls, not search crawlers.
  • Both plugins edit a virtual robots.txt: Rank Math says a physical robots.txt in the site root is what visitors and crawlers get, so you must remove it to use the plugin's editor.
  • Yoast hides its robots.txt file editor when WordPress file editing is disabled, and Rank Math's editor sits under General Settings with Advanced Mode turned on.
  • Rank Math excludes noindex posts from its llms.txt and describes llms.txt as a proposal with limited adoption, which matches the major AI operators' documentation.
On this page

Yoast SEO and Rank Math each have a handful of settings that change what AI crawlers see on a WordPress site: an llms.txt generator, a robots.txt editor, and in Yoast's case toggles that write AI bot blocks into robots.txt. None of them is on the critical path for AI search on its own. The settings that can actually stop an AI search crawler are the robots.txt rules, and those are easy to change by accident with these tools.

This guide lists where each setting lives, what it writes, and what to check afterwards. Menu names come from each plugin's help pages at the time of writing; plugins rename menus often, so if a path does not match, search the plugin's settings for the feature name.

Which Yoast and Rank Math settings touch AI crawlers?

SettingYoast SEORank MathAffects crawler access?
llms.txt generationSettings, Site features, AI tools, llms.txtDashboard, LLMS Txt moduleNo
robots.txt editorTools, File editorGeneral Settings, Edit robots.txt (Advanced Mode)Yes
AI bot blockingSettings, Advanced, Crawl optimization, Block unwanted botsNo equivalent toggle documentedYes, for training bots
Crawl cleanupSettings, Advanced, Crawl optimizationNot covered hereSometimes, via robots.txt
Structured dataSchema FrameworkJSON-LD outputNo, but affects what machines can parse
Social previewsOpen Graph, X card data, Slack sharingOpen Graph outputNo, but affects link previews

The rows marked "Yes" are the ones to review after any plugin change.

llms.txt in Yoast SEO

Yoast's help page on enabling llms.txt gives the path: in WordPress, go to Yoast SEO, then Settings, then under AI tools open llms.txt and switch it on. The Site features page lists it alongside the Yoast AI features and a Schema aggregation endpoint.

Points from Yoast's documentation:

  • It is free. Yoast's feature page says it is included in the free plugin and in Premium, WooCommerce SEO and Yoast SEO for Shopify.
  • Automatic or manual. By default Yoast selects content for you. You can switch to manual page selection and choose the pages yourself.
  • Not on multisite. Yoast says llms.txt is not available for multisite installs.
  • Write permissions. If you see "Your llms.txt file couldn't be auto-generated", Yoast says to check the web server's filesystem permissions.

Yoast's pages do not say how often the file is regenerated or what happens if you already have a hand-made llms.txt in the site root. Check the live file after enabling the feature, and keep a copy of any file you wrote yourself.

llms.txt in Rank Math

Rank Math's llms.txt guide calls the feature the LLMS Txt module. Turn it on from Rank Math SEO, then Dashboard, and open its settings from the module's settings icon or from General Settings, then Edit llms.txt.

The documented settings are:

SettingWhat it does
Post typesWhich types to include, listed with title, URL and short description
TaxonomiesCategories, tags or other taxonomies, listed by name and URL
Posts/Terms limitMaximum number of items, 100 by default
Additional contentCustom links or text, one entry per line

Rank Math says posts set to noindex are excluded from the file, which keeps it consistent with what you want indexed. The guide also describes llms.txt as a proposal rather than an official standard, with limited adoption across platforms.

What llms.txt does and does not do

That description is accurate. Google says Search does not use llms.txt, and OpenAI, Anthropic and Perplexity do not document reading it on other people's sites. Our llms.txt guide has the operator-by-operator detail. Turning the feature on is harmless and may help tools that a person points at the file, but it does not grant or block access to anything. If an AI search crawler cannot reach your pages, llms.txt will not fix it.

robots.txt editing in Yoast and Rank Math

WordPress serves a virtual robots.txt when there is no physical file in the site root. Both plugins edit that virtual file, and both are overridden by a physical one.

Yoast

Yoast's robots.txt help page puts the editor at Yoast SEO, then Tools, then File editor, where you click Create robots.txt file. Yoast notes that the menu does not appear if your WordPress install has disabled file editing, for example on sites that set DISALLOW_FILE_EDIT in wp-config.php. Creating the file through Yoast replaces the WordPress default directives with Yoast's block.

Rank Math

Rank Math's robots.txt guide says to turn on Advanced Mode, open General Settings and choose Edit robots.txt. Rank Math creates a virtual file, and says that to use its editor you must remove any real robots.txt from the site's root folder; otherwise the physical file is what visitors see.

This is the most common source of confusion: you edit rules in the plugin, save, and nothing changes at /robots.txt because a physical file, a host-level rule or a CDN-managed robots.txt takes precedence. Editing the WordPress virtual robots.txt covers how to tell which one is live.

Yoast's crawl optimization and AI bot blocking

Yoast's crawl settings live under Yoast SEO, then Settings, then Advanced, then Crawl optimization. Yoast's 20.4 changelog says these features moved into the free plugin, though some older help pages still label them Premium.

Block unwanted bots

Yoast's bot-blocking guide says that in the free plugin you can only block Google AdsBot. With Premium you can also block Google Gemini and Vertex AI (the Google-Extended token), OpenAI's GPTBot and Common Crawl's CCBot.

Two things to understand before switching these on:

  1. These are training controls. Blocking GPTBot does not block OAI-SearchBot, which is the crawler for ChatGPT search. Blocking Google-Extended does not remove you from Google Search or AI Overviews; see Google-Extended explained. If your aim is to opt out of training while staying in AI search, these toggles fit that aim, and allowing AI search while blocking training shows the full robots.txt.
  2. AdsBot is different. Yoast warns that blocking Google AdsBot can have serious consequences if you run Google Ads.

Other crawl options that write to robots.txt

Some crawl optimization toggles add robots.txt rules. Yoast says the WP-JSON toggle adds a disallow rule for the REST API, and that you can switch it off unless you use the REST API to output important content. If your theme or a headless front end loads page content through the REST API, a crawler that respects robots.txt will no longer be able to fetch it. Yoast also offers internal site search cleanup, including blocking crawling of internal search URLs, which it recommends only when search pages are crawled excessively or under attack.

Schema and social settings

Both plugins output JSON-LD structured data and Open Graph tags. These do not control access, but they shape what machines can read once they have the page. Yoast's site features include a Schema Framework toggle, Open Graph data, X card data and Slack sharing.

If you run both plugins, or one of them alongside a theme that prints its own tags, you can end up with duplicate structured data and duplicate Open Graph tags. Pick one source.

How to check your settings after a change

Run these from outside your network after saving any of the settings above:

bash
# What robots.txt is actually live
curl -s https://www.example.com/robots.txt

# Is llms.txt there and is it text?
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://www.example.com/llms.txt

# Does a request carrying a search crawler token get a normal page?
curl -s -o /dev/null -w '%{http_code}\n' -A 'OAI-SearchBot' \
  https://www.example.com/blog/sample-post

Then read the robots.txt output line by line:

  1. Check for a group for each AI search crawler you want and confirm it is not disallowed.
  2. Look for unexpected Disallow lines, such as /wp-json/, and decide whether you rely on that path.
  3. Confirm the plugin's block is there. If your edits are missing, a physical file or another layer is serving robots.txt.
  4. Purge caches. Page and CDN caches can keep serving the old robots.txt.

If the robots.txt looks right but crawlers still get 403 responses, the cause is usually a security plugin or firewall, not the SEO plugin. Security plugins that block crawlers covers the usual suspects.

Frequently asked questions

There is no documented effect on AI search visibility. The major operators do not document reading llms.txt on publisher sites, and Rank Math itself calls it a proposal with limited adoption. It does no harm to enable it.

Not on their own. The Premium toggles block GPTBot, CCBot and Google-Extended, which are training controls. ChatGPT search uses OAI-SearchBot, which those toggles do not name.

Why do my robots.txt changes in Rank Math not show up?

Rank Math edits a virtual file. If a physical robots.txt exists in the site root, that file is served instead, and Rank Math says you must remove it to use the plugin's editor. Caches and CDN-managed robots.txt can also hide the change.

Why can't I see Yoast's file editor?

Yoast says the File editor menu does not appear when WordPress file editing is disabled. Edit the file over SFTP or your host's file manager instead, or ask your host to allow file editing.

Check your WordPress site

The free check reads your live robots.txt and fetches a page as an automated client, so you can see whether your plugin settings say what you meant. For a full WordPress review, including plugins, caching and firewall rules, see the audit.

Filed under Platform guides. Topics: wordpress, robots-txt, llms-txt, ai-crawlers