WordPress AI crawlers: the settings that decide what they see

On WordPress, whether AI crawlers can read your site comes down to six things: the search visibility checkbox, the robots.txt that WordPress or a plugin generates, your SEO plugin's settings, any security or firewall plugin, how your theme or page builder renders text, and what your cache serves. Each one can keep GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot out, or hide your pages from search, while the site looks fine in your browser. This guide gives you a check and a fix for each, in the order that usually finds the problem fastest.

Facts about WordPress and its plugins come from their own documentation, read on 8 November 2026. Plugin menus move between versions, so confirm the path in your own dashboard.

Step 1: check "Discourage search engines from indexing this site"

Go to Settings > Reading > Search engine visibility. If the box is ticked, untick it unless the site really should stay hidden.

Since WordPress 5.3, the box adds a noindex, nofollow robots meta tag to every page. Up to 5.2 it served Disallow: / in robots.txt instead, according to the Reading settings documentation. The core team made the change because a robots.txt block can still let a linked URL appear in results.

For AI search the tag matters most in Google. A page must be indexed and eligible to show a snippet before it can appear as a supporting link in AI Overviews or AI Mode, as Google's AI features page explains. A noindexed page fails that test.

Check it from a terminal:

curl -s https://www.example.com/ | grep -io '<meta[^>]*robots[^>]*>'

A result such as <meta name='robots' content='noindex, nofollow' /> on a live site means the box is ticked, or an SEO plugin is setting noindex on that page. Check a post, a page and a category archive too, because SEO plugins can set noindex per post type. The setting is stored in the database, so it travels with any database copy from staging to production. The leftover noindex covers that problem in detail.

Step 2: find out which robots.txt WordPress is actually serving

WordPress has no robots.txt file by default. It generates one on request through do_robots(), which outputs User-agent: *, a Disallow for the admin path and an Allow for admin-ajax.php, as the function reference shows. Since WordPress 5.5 it also appends a Sitemap: line for the core sitemap, but only when the site is public.

A typical virtual file looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/wp-sitemap.xml

The catch is precedence. The virtual file applies only when WordPress is installed in the site root and no physical robots.txt exists there. A physical file uploaded by a developer, a host or an old plugin replaces the virtual one completely. Every rule a plugin writes through the robots_txt filter then disappears.

Check it:

curl -s https://www.example.com/robots.txt

Then, on the server, look for a physical file in the WordPress root:

ls -la robots.txt

Read the output for groups that name AI crawlers. A group such as User-agent: GPTBot with Disallow: / is a training opt-out and does not affect ChatGPT search. A group naming OAI-SearchBot, Claude-SearchBot or PerplexityBot with Disallow: / is a search opt-out: OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers, apart from navigational links, and Anthropic says blocking Claude-SearchBot may reduce your visibility in Claude's search results. Remember that a crawler with its own named group ignores the * group entirely, as robots.txt groups explains.

Fix it: pick one source of truth. If you keep a physical file, edit it directly and switch off plugin toggles that you assume are working. If you want plugins to manage the file, delete the physical one. Developers can also add rules in a small plugin or theme through the filter:

add_filter( 'robots_txt', function ( $output, $public ) {
    if ( $public ) {
        $output .= "\nUser-agent: GPTBot\nDisallow: /\n";
    }
    return $output;
}, 10, 2 );

This example opts out of OpenAI training while leaving OAI-SearchBot, the crawler behind ChatGPT search, under the * group. OpenAI's crawler documentation confirms the two controls are independent. For the full pattern across operators, see how to allow AI search but block AI training.

Step 3: review what your SEO plugin changes

Three widely used SEO plugins, Yoast SEO, Rank Math and All in One SEO, all edit robots.txt and all print structured data. Their AI-related settings at the time of writing:

Two things to check after any change:

1. What a toggle actually blocks. Yoast's Premium toggles cover training tokens, which do not decide search visibility. An "all AI crawlers" option can be broader, and its list is not documented in the help article. Read the resulting /robots.txt and look for search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot before you save and walk away.
2. Whether the rules reach crawlers. If a physical robots.txt exists, plugin rules written to the virtual file are not served. If Cloudflare's managed robots.txt is on, Cloudflare places its own rules before yours, so your file is only part of what crawlers read.

Structured data from SEO plugins

Yoast, Rank Math and AIOSEO each output JSON-LD for the site, the page and the content type. The common faults are duplicates and stale facts: a theme or a second plugin printing its own Organization or Article block, or an organization name and logo set years ago. Check a page:

curl -s https://www.example.com/sample-post/ | grep -c 'application/ld+json'

A count above one or two often means a theme or second plugin adds its own markup, which is a reason to look closer with a validator such as the Schema Markup Validator. Keep expectations realistic: Google says no special schema.org structured data is needed to appear in AI Overviews or AI Mode. Structured data helps machines confirm what a page is, as structured data for AI search explains.

Step 4: test whether a security plugin or firewall blocks AI crawlers

robots.txt is a request. A security plugin, a host firewall or a CDN makes the final decision, and it can block a crawler that robots.txt allows. Common causes on WordPress:

• Rate limiting. Wordfence's rate limiting settings set a separate request limit for crawlers, with a special option only for Google's crawlers. Other crawlers, AI crawlers included, fall under the general crawler limit, and a low limit can throttle or block them.
• User-agent blocklists. Some security and "bad bot" plugins ship lists that include AI crawler names. Check the plugin's blocked bots list.
• Challenge pages. A host or CDN challenge returns an HTML page instead of your content. A crawler that fetches raw HTML gets the challenge, not your text, and Anthropic states that it will not attempt to bypass CAPTCHAs.

Check it by sending the same request with a browser user agent and an AI crawler user agent, and comparing the status codes:

URL=https://www.example.com/sample-post/

curl -s -o /dev/null -w "browser: %{http_code}\n" \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "$URL"

curl -s -o /dev/null -w "GPTBot: %{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" "$URL"

curl -s -o /dev/null -w "PerplexityBot: %{http_code}\n" \
  -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" "$URL"

A 200 for the browser and a 403, 429 or 503 for a crawler points to a user-agent rule. Read the result with care in both directions. Some CDNs block requests that claim a crawler's user agent but come from an IP outside its published ranges, so a 403 here can be a fake-bot rule that the real crawler would pass. And the test cannot reproduce rules that block the real crawler's IP addresses, because your request comes from your own network. For those, read the plugin's blocked-requests log or your server logs and look for the crawler tokens. Why a WAF challenge page means AI crawlers read nothing covers how to read those responses.

Fix it: raise crawler rate limits to something your server can handle, remove AI search crawlers from blocklists, and allowlist by published IP ranges rather than user agent alone, since user agents are easy to fake. OpenAI recommends allowing OAI-SearchBot's published IP ranges, and Perplexity recommends firewall rules that match both the user agent and PerplexityBot's published ranges. Fetch those lists automatically instead of copying them, because they change.

Step 5: confirm your theme or page builder puts text in the HTML

Most classic WordPress themes and page builders build the page in PHP on the server, so the text is in the HTML that crawlers download. That matters, because the December 2024 Vercel and MERJ analysis found that none of the major AI crawlers it measured, including OpenAI's crawlers, ClaudeBot and PerplexityBot, rendered JavaScript at the time.

The risk is in the parts that load later:

• posts, reviews or prices fetched by a widget after the page loads
• "load more" buttons and infinite scroll on archive pages
• headless WordPress with a client-rendered front end that calls the REST API from the browser
• text placed inside images, sliders or video

Check it by searching the raw HTML for a sentence you can see on the page:

curl -s https://www.example.com/services/ | grep -c "a sentence from the middle of the page"

Pick a short phrase without apostrophes or quotes, because WordPress often converts them to curly characters or HTML entities. A count of 0 then means a crawler that does not run JavaScript never receives that text. The curl guide shows how to repeat the test with crawler user agents. Fix it by switching the widget to a server-rendered option, rendering the first page of results in PHP, or, for headless builds, using server-side or static rendering.

Step 6: clear caches and recheck

Page caching plugins, host caches and CDNs serve stored copies of your pages. After you untick the search visibility box or change a plugin setting, a cached page can keep serving the old noindex tag, and a cached robots.txt can keep serving old rules.

After every change in steps 1 to 5:

1. Purge the caching plugin, the host cache and the CDN cache.
2. Re-run the curl checks above against the live URLs.
3. Check the response headers for cache status and an X-Robots-Tag you did not expect:

curl -sI https://www.example.com/ | grep -iE '^(x-robots-tag|cache-control|age|x-cache|cf-cache-status):'

Crawlers also keep their own copies. OpenAI says its search systems can take about 24 hours to adjust after a robots.txt change, and Perplexity says robots.txt changes can take up to 24 hours to take effect.

Frequently asked questions

Does WordPress block AI crawlers by default?

No. The default virtual robots.txt blocks only the admin area for all user agents, and the search visibility box is unticked on a normal install. Blocks usually come from a plugin, a physical robots.txt file, a firewall or a CDN setting.

Should I use my SEO plugin's AI bot toggle?

Use it if you want to opt out of AI training, but read the robots.txt it produces afterwards. Training tokens such as GPTBot, CCBot and Google-Extended do not decide whether you appear in ChatGPT search or Google Search, but blocking a search crawler such as OAI-SearchBot keeps you out of ChatGPT search answers.

Why do my robots.txt changes in the plugin not show up?

Usually a physical robots.txt file in the site root, which replaces the virtual file that plugins edit. A cached copy at your CDN or caching plugin is the other likely cause.

Do page builders like Elementor hurt AI crawler access?

Not usually, because most builder pages are generated in PHP on the server. The exceptions are widgets that load content with JavaScript after the page arrives, so test the pages that use them with curl.

Check your WordPress site

If you would rather not run every curl command by hand, start with the free single-page check, which needs no account. For a whole site, the audit covers up to 500 pages and returns evidence with prioritized fixes.