Seven robots.txt mistakes that block AI crawlers by accident

The robots.txt mistakes that block AI crawlers by accident are rarely deliberate choices. They are leftovers and side effects: a staging Disallow: / that reached production, a copied "block AI bots" list that includes search crawlers, a named group that wipes out your general rules, a typo in a token, a robots.txt that returns HTML or a server error, blocked CSS and JavaScript, and subdomains nobody checked. Each one is below with the broken version, the fixed version and a way to confirm it.

If you want the basics of the file first, start with robots.txt for AI crawlers: a complete guide.

The seven mistakes at a glance

Mistakes in the rules themselves

1. A leftover Disallow: / from staging

Staging sites are often told to stay out of search with a blanket block. The problem starts when that file, or the setting that generates it, ships to production.

Before: copied from staging.

User-agent: *
Disallow: /

After:

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

On WordPress this is less common than it used to be. Since WordPress 5.3, the "Discourage search engines from indexing this site" box adds a noindex, nofollow robots meta tag instead of writing Disallow: /, according to the Reading settings documentation. But a physical robots.txt file in the site root replaces WordPress's virtual one entirely, so a file uploaded from staging still wins. The noindex version of the same leak is covered in the leftover noindex.

On frameworks that generate the file, make the block depend on the environment rather than on someone remembering to change it. In Next.js on Vercel, for example:

// app/robots.ts
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  if (process.env.VERCEL_ENV !== "production") {
    return { rules: [{ userAgent: "*", disallow: "/" }] };
  }
  return {
    rules: [{ userAgent: "*", disallow: "/admin/" }],
    sitemap: "https://example.com/sitemap.xml",
  };
}

VERCEL_ENV is a Vercel system environment variable set to production, preview or development (Vercel). It is not set on other hosts, so if you self-host, switch the check to a variable your own deployment sets, or this file will block everything in production.

2. A copied or plugin-made AI block list

Many "block AI bots" lists treat every AI-related token the same. They block training crawlers, which is a reasonable choice, but they also block the crawlers that put pages into AI search results.

Before: blocks training and search alike.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: CCBot
Disallow: /

OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links, and Perplexity says PerplexityBot is how sites are surfaced and linked in its results (OpenAI, Perplexity). Anthropic says disabling Claude-SearchBot may reduce visibility in its users' search results (Anthropic). If your goal was a training opt-out, keep only the training tokens:

After: training opt-out, search allowed.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /admin/

Platform switches deserve the same review. Squarespace's "Block known artificial intelligence crawlers" checkbox, off by default, lists 26 user agents, including DuckAssistBot, which DuckDuckGo says fetches pages for its cited AI answers and does not use for training (Squarespace, DuckDuckGo). On Cloudflare, Bot Preference Sync (managed robots.txt) places its own Disallow: / groups for training crawlers such as GPTBot, ClaudeBot and CCBot before your file once any of your AI access policies uses a block or disallow action, and those policies can also block at the edge whatever robots.txt says (Cloudflare). Cloudflare and AI crawlers walks through those settings.

3. A named group that drops your * rules

Groups do not merge across user agents. Under RFC 9309, a crawler obeys the group that matches its token and ignores *; it falls back to * only when no group names it. Bing says the same of Bingbot: if a group is addressed to it, the general rules are ignored and must be repeated (Bing).

The accidental block usually comes from stacking. Consecutive User-agent lines form one group and share the rules below them:

Before: meant to block training only.

User-agent: GPTBot
User-agent: OAI-SearchBot
Disallow: /

That blocks OAI-SearchBot too. Split them, and repeat any general rules inside each named group, because the named crawler no longer reads *:

After:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /admin/
Disallow: /cart/

User-agent: *
Disallow: /admin/
Disallow: /cart/

One fallback is easy to miss. Apple says that when robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules (Apple). A User-agent: Googlebot group with Disallow: / blocks Applebot as well. How robots.txt groups work covers group selection in depth.

4. Typos in tokens and wrong case in paths

User-agent tokens are matched case-insensitively under RFC 9309, so oai-searchbot and OAI-SearchBot are the same. Spelling is a different matter. A misspelled token matches nothing, and the crawler you meant to address falls back to *. On a site that blocks by default and allows named crawlers, that is a block:

Before: the allow group matches no crawler.

User-agent: OAISearchBot
Allow: /

User-agent: *
Disallow: /

After:

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Disallow: /

Paths are the opposite: the RFC says path matching should be case-sensitive, and rules match from the start of the path. Disallow: /Blog/ does not block /blog/, and Disallow: /search also blocks /search-engine-guide. End a directory rule with a slash, or use $ to mark the end of a pattern.

5. Blocking CSS and JavaScript

Older templates often blocked asset folders:

Before:

User-agent: *
Disallow: /wp-content/
Disallow: /_next/
Disallow: /*.js$

Googlebot renders pages with a headless Chromium, and Google's AI Overviews and AI Mode are part of Search, controlled by Googlebot rules (Google). Apple warns that blocking JavaScript or CSS can stop Applebot rendering a page properly (Apple). Blocking /wp-content/ also blocks images in /wp-content/uploads/. Remove those lines. On WordPress, the result looks like this:

After:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

In Vercel's December 2024 measurements, none of the major AI crawlers it named (OpenAI's crawlers, ClaudeBot, Meta-ExternalAgent, Bytespider, PerplexityBot) executed JavaScript; only Gemini, through Googlebot, and Applebot rendered pages. For the non-rendering crawlers, the raw HTML matters more than the assets. That is a separate problem, but it is worth knowing which one you have.

Mistakes in how and where the file is served

6. robots.txt served as HTML, or as a 5xx error

Two serving problems look harmless in a browser.

HTML with a 200 status. Single-page apps and some static hosts rewrite every unknown path to index.html. A crawler then receives your app shell as robots.txt. Parsers skip lines they cannot read, and an HTML page usually has no lines that parse as rules, so your blocks, your allows and your Sitemap: line all disappear.

A 5xx or a timeout. This is the serious one. RFC 9309 says that when robots.txt is unreachable because of a server or network error, crawlers must assume complete disallow. Google stops crawling the site for the first 12 hours, then uses its last good copy for up to 30 days (Google), and it treats a 429 as a server error too (Google). A 4xx such as 404 or 403 means the opposite: no restrictions.

Check it:

curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.com/robots.txt
curl -s https://example.com/robots.txt | head -5

You want 200 with text/plain (often followed by a charset) and plain rules in the first lines, not <!DOCTYPE html>. Fix it by serving a real file before any catch-all rewrite, and by making sure your CDN or firewall does not challenge or rate-limit requests for /robots.txt. Status codes, redirects and AI crawlers covers the same rules for ordinary pages.

7. Forgotten subdomains

A robots.txt file applies only to the host, protocol and port it is served from (Google). Anthropic tells site owners to add rules to every subdomain they want covered. So example.com/robots.txt says nothing about shop.example.com, blog.example.com or docs.example.com, which often run on a different platform with its own defaults.

The typical failure is a subdomain built on another service whose file was never reviewed, or a staging subdomain's blanket block copied to a new production host.

for host in example.com www.example.com shop.example.com blog.example.com docs.example.com; do
  printf '%-22s ' "$host"
  curl -s -L -o /dev/null -w '%{http_code} %{content_type}\n' "https://$host/robots.txt"
done

Then read each 200 response with the same eyes as your main file. Google Search Console's robots.txt report lists the files Google found for your top 20 hosts, with their fetch status and errors (Google).

How to check all seven in five minutes

1. Fetch robots.txt on every host you serve and confirm 200 text/plain (mistakes 6 and 7).
2. List the groups: curl -s https://example.com/robots.txt | grep -inE '^(user-agent|disallow|allow):'.
3. For each crawler you care about (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot), find the one group that applies to it, or * if none does, and read only that group (mistakes 1 to 4).
4. Search for asset paths and wildcard rules that end in .js or .css (mistake 5).
5. Test real URLs against each token with a parser that follows the RFC, such as Google's open-source robots.txt parser. How to test your robots.txt against each AI crawler shows the commands.

After a fix, allow for caching. OpenAI and Perplexity both say changes can take about 24 hours to take effect, and RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable.

Frequently asked questions

Does a missing robots.txt block AI crawlers?

No. Under RFC 9309 a 404 or other 4xx response means the crawler may access everything. A 5xx error is what makes compliant crawlers assume the whole site is disallowed.

Is robots.txt case-sensitive?

User-agent tokens are not: crawlers match them case-insensitively. Paths are: Disallow: /Blog/ does not block /blog/.

Do I need to repeat my * rules inside a named group?

Yes. A crawler that finds a group naming it ignores the * group completely, so any general rule you want it to follow must be written again inside its own group.

Does blocking GPTBot keep my site out of ChatGPT search?

No. OpenAI says GPTBot and OAI-SearchBot are controlled independently. GPTBot covers training; OAI-SearchBot decides whether pages can be shown in ChatGPT search answers.

How quickly do crawlers pick up a fixed robots.txt?

For most major crawlers, within about a day: Google generally caches the file for up to 24 hours, OpenAI and Perplexity say to allow about 24 hours, and Bing says at least a few hours. Some take longer; DuckDuckGo says a DuckAssistBot opt-out takes about 72 hours.

Next step

The free check reads your robots.txt and reports the rule that applies to each named AI crawler for one page, along with the status code and robots meta tags, with no account needed. Run it on one page from each host you serve to catch mistakes 6 and 7 as well.