robots.txt for AI crawlers: a complete guide
How robots.txt works for AI crawlers: the RFC 9309 matching rules, how GPTBot, ClaudeBot and others read the file, and three templates you can copy.
9 min read
10 guides
Access and robots.txt
Most AI search problems begin before a crawler reads a single word: a robots.txt rule, a noindex left over from staging, or a redirect chain that ends in an error. These guides cover the rules that decide whether a page can be fetched and used.
Crawl-delay for AI bots: who honors it and what to use instead
Crawl-delay is not in the robots.txt standard and AI crawler support varies. See which bots honor it and how to slow the rest with 429 and Retry-After.
9 min read
How to allow AI search but block AI training in robots.txt
A copy-paste robots.txt that blocks AI training (GPTBot, ClaudeBot, Google-Extended, CCBot) and allows AI search crawlers, with every line explained.
10 min read
How to test robots.txt against each AI crawler
Test robots.txt the way AI crawlers read it: fetch it raw with curl, check every host, run Google's parser and trace the group each AI token follows.
9 min read
noindex, nosnippet and X-Robots-Tag: what they mean for AI search
How nosnippet, max-snippet, data-nosnippet and noindex control AI Overviews and AI Mode, what other AI tools document, and how to set X-Robots-Tag for PDFs.
8 min read
robots.txt user-agent groups: why a named group overrides *
How robots.txt user-agent groups work under RFC 9309: a crawler obeys one group, rules under * are not inherited, and the longest matching rule wins.
8 min read
robots.txt vs meta robots: blocking crawling vs blocking use
robots.txt stops crawling, noindex stops listing. Why a Disallow can hide your noindex, when to use each, and how the same logic applies to AI crawlers.
9 min read
Seven robots.txt mistakes that block AI crawlers by accident
Seven robots.txt mistakes that quietly block AI crawlers, from a leftover staging Disallow to a 5xx error, each with a before-and-after fix you can test.
9 min read
Site noindex by mistake: how staging settings hide live sites
Five staging leftovers that keep a live site out of Google and AI search, how to spot each one in two minutes with curl, and a launch checklist to prevent them.
9 min read
Status codes and redirects: what AI crawlers do with them
Redirect chains, soft 404s and wrong status codes can stop AI crawlers reaching a page. How 3xx, 4xx and 5xx are read, robots.txt errors, and curl checks.
10 min read