Yoast and Rank Math settings for AI crawlers and llms.txt
Which Yoast SEO and Rank Math settings affect AI crawlers: llms.txt generation, robots.txt editors, bot blocking and crawl options, with where to find each one.
By Crawlito · 7 min read
In short
- Yoast SEO generates llms.txt from Settings, Site features, under AI tools, and Rank Math from its LLMS Txt module; both are switches you turn on, and neither changes how crawlers can access your pages.
- Yoast's Block unwanted bots setting blocks only Google AdsBot in the free plugin; Premium adds toggles for Google-Extended, GPTBot and CCBot, which are training controls, not search crawlers.
- Both plugins edit a virtual robots.txt: Rank Math says a physical robots.txt in the site root is what visitors and crawlers get, so you must remove it to use the plugin's editor.
- Yoast hides its robots.txt file editor when WordPress file editing is disabled, and Rank Math's editor sits under General Settings with Advanced Mode turned on.
- Rank Math excludes noindex posts from its llms.txt and describes llms.txt as a proposal with limited adoption, which matches the major AI operators' documentation.
On this page
- Which Yoast and Rank Math settings touch AI crawlers?
- llms.txt in Yoast SEO
- llms.txt in Rank Math
- robots.txt editing in Yoast and Rank Math
- Yoast's crawl optimization and AI bot blocking
- Schema and social settings
- How to check your settings after a change
- Frequently asked questions
- Check your WordPress site
Yoast SEO and Rank Math each have a handful of settings that change what AI crawlers see on a WordPress site: an llms.txt generator, a robots.txt editor, and in Yoast's case toggles that write AI bot blocks into robots.txt. None of them is on the critical path for AI search on its own. The settings that can actually stop an AI search crawler are the robots.txt rules, and those are easy to change by accident with these tools.
This guide lists where each setting lives, what it writes, and what to check afterwards. Menu names come from each plugin's help pages at the time of writing; plugins rename menus often, so if a path does not match, search the plugin's settings for the feature name.
Which Yoast and Rank Math settings touch AI crawlers?
| Setting | Yoast SEO | Rank Math | Affects crawler access? |
|---|---|---|---|
| llms.txt generation | Settings, Site features, AI tools, llms.txt | Dashboard, LLMS Txt module | No |
| robots.txt editor | Tools, File editor | General Settings, Edit robots.txt (Advanced Mode) | Yes |
| AI bot blocking | Settings, Advanced, Crawl optimization, Block unwanted bots | No equivalent toggle documented | Yes, for training bots |
| Crawl cleanup | Settings, Advanced, Crawl optimization | Not covered here | Sometimes, via robots.txt |
| Structured data | Schema Framework | JSON-LD output | No, but affects what machines can parse |
| Social previews | Open Graph, X card data, Slack sharing | Open Graph output | No, but affects link previews |
The rows marked "Yes" are the ones to review after any plugin change.
llms.txt in Yoast SEO
Yoast's help page on enabling llms.txt gives the path: in WordPress, go to Yoast SEO, then Settings, then under AI tools open llms.txt and switch it on. The Site features page lists it alongside the Yoast AI features and a Schema aggregation endpoint.
Points from Yoast's documentation:
- It is free. Yoast's feature page says it is included in the free plugin and in Premium, WooCommerce SEO and Yoast SEO for Shopify.
- Automatic or manual. By default Yoast selects content for you. You can switch to manual page selection and choose the pages yourself.
- Not on multisite. Yoast says llms.txt is not available for multisite installs.
- Write permissions. If you see "Your llms.txt file couldn't be auto-generated", Yoast says to check the web server's filesystem permissions.
Yoast's pages do not say how often the file is regenerated or what happens if you already have a hand-made llms.txt in the site root. Check the live file after enabling the feature, and keep a copy of any file you wrote yourself.
llms.txt in Rank Math
Rank Math's llms.txt guide calls the feature the LLMS Txt module. Turn it on from Rank Math SEO, then Dashboard, and open its settings from the module's settings icon or from General Settings, then Edit llms.txt.
The documented settings are:
| Setting | What it does |
|---|---|
| Post types | Which types to include, listed with title, URL and short description |
| Taxonomies | Categories, tags or other taxonomies, listed by name and URL |
| Posts/Terms limit | Maximum number of items, 100 by default |
| Additional content | Custom links or text, one entry per line |
Rank Math says posts set to noindex are excluded from the file, which keeps it consistent with what you want indexed. The guide also describes llms.txt as a proposal rather than an official standard, with limited adoption across platforms.
What llms.txt does and does not do
That description is accurate. Google says Search does not use llms.txt, and OpenAI, Anthropic and Perplexity do not document reading it on other people's sites. Our llms.txt guide has the operator-by-operator detail. Turning the feature on is harmless and may help tools that a person points at the file, but it does not grant or block access to anything. If an AI search crawler cannot reach your pages, llms.txt will not fix it.
robots.txt editing in Yoast and Rank Math
WordPress serves a virtual robots.txt when there is no physical file in the site root. Both plugins edit that virtual file, and both are overridden by a physical one.
Yoast
Yoast's robots.txt help page puts the editor at Yoast SEO, then Tools, then File editor, where you click Create robots.txt file. Yoast notes that the menu does not appear if your WordPress install has disabled file editing, for example on sites that set DISALLOW_FILE_EDIT in wp-config.php. Creating the file through Yoast replaces the WordPress default directives with Yoast's block.
Rank Math
Rank Math's robots.txt guide says to turn on Advanced Mode, open General Settings and choose Edit robots.txt. Rank Math creates a virtual file, and says that to use its editor you must remove any real robots.txt from the site's root folder; otherwise the physical file is what visitors see.
This is the most common source of confusion: you edit rules in the plugin, save, and nothing changes at /robots.txt because a physical file, a host-level rule or a CDN-managed robots.txt takes precedence. Editing the WordPress virtual robots.txt covers how to tell which one is live.
Yoast's crawl optimization and AI bot blocking
Yoast's crawl settings live under Yoast SEO, then Settings, then Advanced, then Crawl optimization. Yoast's 20.4 changelog says these features moved into the free plugin, though some older help pages still label them Premium.
Block unwanted bots
Yoast's bot-blocking guide says that in the free plugin you can only block Google AdsBot. With Premium you can also block Google Gemini and Vertex AI (the Google-Extended token), OpenAI's GPTBot and Common Crawl's CCBot.
Two things to understand before switching these on:
- These are training controls. Blocking
GPTBotdoes not blockOAI-SearchBot, which is the crawler for ChatGPT search. BlockingGoogle-Extendeddoes not remove you from Google Search or AI Overviews; see Google-Extended explained. If your aim is to opt out of training while staying in AI search, these toggles fit that aim, and allowing AI search while blocking training shows the full robots.txt. - AdsBot is different. Yoast warns that blocking Google AdsBot can have serious consequences if you run Google Ads.
Other crawl options that write to robots.txt
Some crawl optimization toggles add robots.txt rules. Yoast says the WP-JSON toggle adds a disallow rule for the REST API, and that you can switch it off unless you use the REST API to output important content. If your theme or a headless front end loads page content through the REST API, a crawler that respects robots.txt will no longer be able to fetch it. Yoast also offers internal site search cleanup, including blocking crawling of internal search URLs, which it recommends only when search pages are crawled excessively or under attack.
Schema and social settings
Both plugins output JSON-LD structured data and Open Graph tags. These do not control access, but they shape what machines can read once they have the page. Yoast's site features include a Schema Framework toggle, Open Graph data, X card data and Slack sharing.
If you run both plugins, or one of them alongside a theme that prints its own tags, you can end up with duplicate structured data and duplicate Open Graph tags. Pick one source.
How to check your settings after a change
Run these from outside your network after saving any of the settings above:
# What robots.txt is actually live
curl -s https://www.example.com/robots.txt
# Is llms.txt there and is it text?
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://www.example.com/llms.txt
# Does a request carrying a search crawler token get a normal page?
curl -s -o /dev/null -w '%{http_code}\n' -A 'OAI-SearchBot' \
https://www.example.com/blog/sample-postThen read the robots.txt output line by line:
- Check for a group for each AI search crawler you want and confirm it is not disallowed.
- Look for unexpected Disallow lines, such as
/wp-json/, and decide whether you rely on that path. - Confirm the plugin's block is there. If your edits are missing, a physical file or another layer is serving robots.txt.
- Purge caches. Page and CDN caches can keep serving the old robots.txt.
If the robots.txt looks right but crawlers still get 403 responses, the cause is usually a security plugin or firewall, not the SEO plugin. Security plugins that block crawlers covers the usual suspects.
Frequently asked questions
Does turning on llms.txt in Yoast or Rank Math help AI search?
There is no documented effect on AI search visibility. The major operators do not document reading llms.txt on publisher sites, and Rank Math itself calls it a proposal with limited adoption. It does no harm to enable it.
Will Yoast's AI bot toggles remove my site from ChatGPT search?
Not on their own. The Premium toggles block GPTBot, CCBot and Google-Extended, which are training controls. ChatGPT search uses OAI-SearchBot, which those toggles do not name.
Why do my robots.txt changes in Rank Math not show up?
Rank Math edits a virtual file. If a physical robots.txt exists in the site root, that file is served instead, and Rank Math says you must remove it to use the plugin's editor. Caches and CDN-managed robots.txt can also hide the change.
Why can't I see Yoast's file editor?
Yoast says the File editor menu does not appear when WordPress file editing is disabled. Edit the file over SFTP or your host's file manager instead, or ask your host to allow file editing.
Check your WordPress site
The free check reads your live robots.txt and fetches a page as an automated client, so you can see whether your plugin settings say what you meant. For a full WordPress review, including plugins, caching and firewall rules, see the audit.
Filed under Platform guides. Topics: wordpress, robots-txt, llms-txt, ai-crawlers