Skip to content

All in One SEO robots.txt editor and Block AI Crawlers explained

What the All in One SEO robots.txt editor and its Block AI Crawlers option actually write, which bots they cover, and how to keep AI search crawlers allowed.

By Crawlito · 8 min read

In short

  • All in One SEO does not create a robots.txt file: it adds rules to the virtual file WordPress generates, through the robots_txt filter.
  • In the free plugin's source at version 5.0.5.1, Block AI Crawlers covers four tokens: AdsBot, Google-Extended, GPTBot and CCBot, each written as Disallow: /.
  • Choosing All AI Crawlers also writes a Disallow group for Google AdsBot, an ads quality crawler rather than an AI crawler, so leave it off if you run Google Ads.
  • None of the built-in options name OAI-SearchBot, Claude-SearchBot or PerplexityBot; if one is blocked, the rule came from somewhere else, such as a custom or imported rule.
  • The llms.txt generator writes a real file to the site root and is off by default; llms-full.txt and Markdown output need the Pro plan.
On this page

All in One SEO (AIOSEO) edits the virtual robots.txt that WordPress builds on each request. It does not write a file. Its Block AI Crawlers option adds a Disallow: / group for each bot you tick, and in the free plugin that list has four entries: Google AdsBot, Google-Extended, OpenAI's GPTBot and Common Crawl's CCBot. These are training and ads controls. The search crawlers behind ChatGPT search, Claude and Perplexity are not on the list, and Google says Google-Extended does not affect Google Search, so the option does not block those search crawlers. (Google-Extended does cover grounding in Gemini Apps and Vertex AI.) The two real risks are ticking All AI Crawlers on a site that runs Google Ads, and custom or imported rules that block more than you meant.

This guide covers what each AIOSEO setting writes, how to read the result, and where the plugin's llms.txt file fits. The general WordPress picture, including physical files and the robots_txt filter, is in editing the WordPress virtual robots.txt, and the Yoast and Rank Math equivalents are in Yoast and Rank Math AI crawler settings.

Where the AIOSEO robots.txt settings live

AIOSEO's robots.txt help article puts the editor under All in One SEO > Tools > Robots.txt Editor. Some of the same options also appear elsewhere in the plugin.

SettingWhereWhat it writes
Enable Custom Robots.txtTools, Robots.txt EditorTurns on your own rules from the rule builder
Block AI CrawlersRobots.txt Editor, and Search Appearance, Advanced, Crawl CleanupDisallow: / for each ticked bot
Block Crawling of Internal Site Search URLsSame two placesDisallow lines for search result URLs, for all crawlers
ImportRobots.txt EditorCopies rules from a URL or pasted text into the builder
Network robots.txtNetwork Admin, All in One SEO, Network ToolsRules for every site in a multisite network

The plugin's interface text says the bot setting "will continue to work even if Crawl Cleanup is disabled". In the plugin's code, the bot and site search rules are added to robots.txt whether or not Enable Custom Robots.txt is on. So a site can block GPTBot through AIOSEO without the custom editor ever being switched on, which is worth knowing when you audit someone else's site.

What Block AI Crawlers actually writes

The help article says you can pick crawlers one by one or choose All AI Crawlers, but it does not name them. The plugin source on WordPress.org does. In version 5.0.5.1, the free plugin's options and labels are:

Label in AIOSEOToken writtenWhat the token controls
Google AdsBotAdsBotGoogle Ads landing page quality checks
Google Gemini & Vertex AI BotsGoogle-ExtendedUse of content for Gemini training and grounding
OpenAI GPTBotGPTBotUse of content for OpenAI model training
Common Crawl CCBotCCBotCommon Crawl's open dataset, widely used for training

Ticking all of them produces groups like these, after the WordPress defaults:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: AdsBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

Three points follow from the documentation of each operator:

  • GPTBot is not ChatGPT search. OpenAI's crawler documentation says disallowing GPTBot means content should not be used for training foundation models, that OAI-SearchBot is the crawler for ChatGPT search, and that "each setting is independent of the others".
  • Google-Extended is not Google Search. Google's common crawlers page says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal".
  • AdsBot is not an AI crawler. It checks ad landing pages. Google's special-case crawlers page documents the token as AdsBot-Google and says it ignores the * group, so it only follows rules that name it. AIOSEO writes the shorter AdsBot. Do not rely on that difference either way: if you run Google Ads, leave the box unticked. AdsBot-Google and robots.txt explains what a blocked AdsBot does to ads.

The practical rule: All AI Crawlers is a training opt-out plus an ads block. If you want the training opt-out alone, tick Google-Extended, GPTBot and CCBot individually. If you also want to opt out of Anthropic or Apple training, add ClaudeBot and Applebot-Extended as custom rules, as described in allow AI search, block AI training.

Using the rule builder and import

With Enable Custom Robots.txt on, click Add Rule and fill in three fields:

  1. User Agent: the crawler token, or * for all crawlers.
  2. Directive: Allow, Block (written as Disallow), Clean-param or Crawl-delay. AIOSEO notes that Clean-param is Yandex-only and that Google does not use Crawl-delay.
  3. Value: the path, with * wildcards allowed.

Click Save Changes, then Open Robots.txt to see the live result. Rules can be reordered by dragging and deleted with the trash icon.

The Import button accepts a robots.txt URL or pasted text. Two behaviours in the plugin's code matter when you import:

  • Only five directives survive. The importer keeps User-agent, Allow, Disallow, Clean-param and Crawl-delay lines. Comments, Sitemap lines and anything else, such as a Content-Signal line, are dropped. Add sitemaps through the plugin's sitemap settings instead.
  • Shared groups are split. A group that lists several user agents above one set of rules is stored as one copy of the rules per user agent. The meaning is the same under RFC 9309, but the output looks longer than the file you pasted.

Imported rules land in the same list as your custom rules. In the plugin's code that list is only printed while Enable Custom Robots.txt is on, so an import has no visible effect with the toggle off.

The same parsing applies to the live file. AIOSEO hooks the WordPress robots_txt filter at priority 10000, reads the output that WordPress and other plugins produced, and prints it again from its own parsed rules. If another plugin adds a comment or a non-standard line to robots.txt, expect AIOSEO's output not to include it.

AIOSEO also says the default WordPress rules cannot be removed from its interface. To change those, use the robots_txt filter in code.

How to keep AI search crawlers out of a blanket block

The built-in options never name OAI-SearchBot, Claude-SearchBot or PerplexityBot. If one of them is blocked on an AIOSEO site, the cause is a rule someone added or imported, a network rule on multisite, or a line from WordPress or another plugin that AIOSEO passes through. Look for these patterns in the output:

PatternEffect on AI search crawlers
User-agent: * with Disallow: /Blocks every crawler that has no group of its own, including AI search crawlers
A group naming OAI-SearchBot, Claude-SearchBot or PerplexityBot with Disallow: /Blocks that crawler outright
An imported "block all AI bots" listOften mixes training and search tokens in one group

To keep search open while a broad block stays in place, give each search crawler its own group. A crawler follows only the most specific group that names it, so a named group lets it ignore Disallow: / under *. Because it then ignores everything under *, repeat any rules you still want it to follow. In the rule builder, that means rules like these:

User AgentDirectiveValue
OAI-SearchBotAllow/
OAI-SearchBotBlock/wp-admin/
Claude-SearchBotAllow/
Claude-SearchBotBlock/wp-admin/
PerplexityBotAllow/
PerplexityBotBlock/wp-admin/

robots.txt groups explained covers how crawlers pick a group. If you want the blanket block gone entirely, delete the * rule rather than adding exceptions around it. OpenAI says its search systems can take about 24 hours to adjust after a robots.txt change.

When a physical file or the server overrides the plugin

AIOSEO only works on the virtual file. If a real robots.txt exists in the web root, the server returns it and WordPress never runs. The plugin checks for this and shows an admin notice titled Physical Robots.txt File Detected, with Import and Delete and Delete buttons. Import and Delete copies the file's rules into the editor, with the limits described above, and then removes the file.

Two cases the notice can miss:

  • No direct file access. In the plugin's code the check only runs when WordPress can write to the filesystem directly. On hosts that need FTP credentials for file operations, look for the file yourself over SFTP.
  • nginx not passing the request to WordPress. AIOSEO's nginx article says a rewrite should no longer be needed, but if /robots.txt returns a 404, add this rule or ask your host to:
nginx
rewrite ^/robots.txt$ /index.php?aioseo_robots_path=root;

On multisite, AIOSEO's network article says the network editor only works for subdomain networks, and that network-level rules apply to every site and cannot be overridden at site level. The free plugin's built-in feature list marks the network robots.txt tool as an Elite plan feature at the time of writing. If a subsite blocks a crawler you did not expect, check the network rules first.

Where the AIOSEO llms.txt file fits

The llms.txt generator sits apart from robots.txt, on the LLMs.txt tab of the plugin's Sitemaps settings. In the plugin's code it is off by default, writes a real llms.txt file to the site root once enabled, and regenerates it on a daily schedule. You choose which post types and taxonomies to include. AIOSEO's feature page says the free version generates llms.txt, while llms-full.txt and Markdown conversion need Pro.

llms.txt does not grant or block access to anything. A crawler blocked in robots.txt is still blocked, and none of the major AI search operators document reading llms.txt on publisher sites at the time of writing. See the llms.txt guide for each operator's position. Because it is a physical file, a page cache or CDN may serve an old copy after changes.

How to check what AIOSEO is serving

Run these after any change, and purge page and CDN caches first:

bash
# The robots.txt crawlers actually get
curl -s https://www.example.com/robots.txt

# Which bot groups the plugin wrote
curl -s https://www.example.com/robots.txt | grep -i -A2 'user-agent'

# Is llms.txt there, and is it plain text?
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://www.example.com/llms.txt

Then check four things:

  1. AdsBot: if you run Google Ads, no group should disallow AdsBot or AdsBot-Google.
  2. Search crawlers: no rule that applies to OAI-SearchBot, Claude-SearchBot or PerplexityBot should be Disallow: /.
  3. Plugin changes are visible: if your edits are missing, a physical file, the network rules or a cache is serving robots.txt.
  4. Sitemap line: the Sitemap URL you expect is present.

Frequently asked questions

No. In the free plugin it blocks AdsBot, Google-Extended, GPTBot and CCBot. ChatGPT search uses OAI-SearchBot, which none of the built-in options name.

Is it safe to tick All AI Crawlers?

It is a training opt-out plus a Disallow group for Google AdsBot. If you run Google Ads, tick Google-Extended, GPTBot and CCBot individually instead.

Why does my AIOSEO robots.txt look different from the file I imported?

The importer keeps only User-agent, Allow, Disallow, Clean-param and Crawl-delay lines, and splits shared groups into one group per user agent. Comments and Sitemap lines are dropped.

Why are my AIOSEO robots.txt changes not showing?

A physical robots.txt in the web root overrides the plugin, network rules apply on multisite, and page or CDN caches can serve an old copy. Check with curl after purging caches.

There is no documented effect. The major AI search operators do not say they read llms.txt on publisher sites, and the file does not change what crawlers may access.

Check what your plugin settings produce

Plugin screens show what you selected, not what crawlers receive. Run the free check on an important page to see your live robots.txt and how the main AI crawlers are treated. For a full review of a WordPress site's plugins, caching and firewall together, see the audit.

Filed under Platform guides. Topics: wordpress, robots-txt, ai-crawlers, llms-txt