Do AI crawlers respect robots.txt? What the evidence says
Do AI crawlers respect robots.txt? What OpenAI, Anthropic and Perplexity document, which user-triggered fetchers are exempt, and what the research found.
By Crawlito · 8 min read
In short
OpenAI and Anthropic document that their crawlers follow robots.txt, but OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
RFC 9309 says robots.txt rules are not a form of access authorization, so the file is a request to well-behaved crawlers, not a lock.
A 2025 measurement study (Kim et al., ACM IMC 2025) found that bots were less likely to comply with stricter robots.txt directives, and that AI search crawlers rarely checked the file at all.
The only way to know what happens on your site is to read your own server logs; a user agent string alone proves nothing, because anyone can send it.
If you need a hard block, use a firewall rule, rate limit or authentication, and accept that blocking search crawlers can remove your pages from AI search answers.
Mostly yes for the declared crawlers, and no for the fetchers that act on a user's request. OpenAI and Anthropic document that their crawlers honor robots.txt, while OpenAI, Perplexity and Google all document that user-triggered fetchers may or do ignore it. Independent measurement is less comforting: one 2025 study found that compliance drops as rules get stricter. Treat robots.txt as a request that most large operators say they follow, and use a firewall when the answer has to be no.
What robots.txt is, and what it is not
robots.txt is a plain text file at the root of a host that tells crawlers which paths they may fetch. The standard behind it is RFC 9309, and the RFC is direct about its limits: it says these rules are not a form of access authorization, and that the protocol is not a substitute for real content security measures.
That has two practical meanings:
A crawler that chooses to ignore the file faces no technical barrier. Nothing in robots.txt stops a request; the server still answers it.
The file only works on crawlers that fetch it, parse it and act on it. Each of those is a step the crawler's operator has to build.
The RFC also fixes how a polite crawler behaves. It treats a 4xx response for robots.txt as unavailable, meaning the crawler may access any resources, and a 5xx as unreachable, meaning the crawler must assume complete disallow. It also says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable. For the matching rules themselves, see our robots.txt guide for AI crawlers.
What each operator documents about honoring robots.txt
The cleanest evidence is what operators say in public. At the time of writing:
Operator and agent
Purpose
What the operator documents
OpenAI OAI-SearchBot
ChatGPT search
Governed by robots.txt; sites that opt out do not appear in ChatGPT search answers
OpenAI GPTBot
Model training
Governed by robots.txt; a disallow signals content should not be used for training
Anthropic also says it supports the non-standard Crawl-delay directive "where appropriate" and that its bots respect anti-circumvention technologies and will not bypass CAPTCHAs. It adds that blocking its IP ranges may not reliably achieve an opt-out, and points to robots.txt as the method. Anthropic's page does not carve out a separate exemption for Claude-User.
OpenAI says its search systems can take about 24 hours to adjust after a robots.txt change. It does not give a timeline for GPTBot or ChatGPT-User.
Perplexity says each of its settings works independently and that changes can take up to 24 hours.
A user-triggered fetcher does not crawl. A person asks an assistant a question or hands it a URL, and the assistant requests that page once, much as a browser would. Operators argue that a robots.txt rule written for automatic crawlers does not describe that situation.
This is stated plainly in documentation. Perplexity says that because a user requested the fetch, Perplexity-User generally ignores robots.txt rules. Google's documentation says the same about its user-triggered fetchers: because a user requested the fetch, they generally ignore robots.txt rules. OpenAI uses softer wording for ChatGPT-User: robots.txt rules "may not apply."
You can disagree with that position, and some site owners do. But it is the documented behavior, so a Disallow for a user-triggered agent is not a dependable control. If you want one, you need enforcement at the server or edge. Our post on training, search and user-triggered crawlers explains what each category costs you to block.
What independent evidence found
The Kim et al. study
The most-cited academic evidence is "Scrapers selectively respect robots.txt directives: evidence from a large-scale empirical study" by Taein Kim and co-authors, posted on arXiv in May 2025 and published in the proceedings of ACM IMC 2025. According to the paper's abstract, the authors analyzed web logs from their institution covering 40 days, with 130 self-declared bots and many anonymous ones, and ran controlled robots.txt experiments. They report two findings:
Bots were less likely to comply with stricter robots.txt directives.
Some bot categories, including AI search crawlers, rarely check robots.txt at all.
In the paper's body, the authors report that AI assistants and AI search crawlers had the lowest re-check rates, with fewer than 40% of bots in those categories checking robots.txt within a 168-hour window. The test is strict: a bot only counted as checking at that interval if every 168-hour window in the dataset contained a robots.txt request. They conclude that relying on robots.txt to prevent unwanted scraping is risky and that alternatives are needed.
Read this with its limits in mind. The data comes from one institution's sites. The authors describe it as a snapshot, and bot behavior changes. A bot that skips a window may still be using a cached copy; the authors say investigating robots.txt caching was out of scope. It tells you what happened to those sites in early 2025, not what any specific crawler does on yours today.
A contested case: Perplexity and Cloudflare
In August 2025, Cloudflare published a report alleging that when Perplexity's declared crawlers were blocked, requests continued from an undeclared crawler using a generic browser user agent and unlisted IP addresses. Cloudflare said it tested this on new domains with robots.txt set to disallow all crawling, and that ChatGPT-User stopped when disallowed. It said it removed Perplexity from its verified bot list as a result.
Perplexity denied the claims, according to press coverage, with a spokesperson reportedly saying the bot Cloudflare described was not Perplexity's. This is one party's measurement against the other party's response, so treat it as an allegation, not a settled finding. It does show why a robots.txt rule that names a user agent only works if the traffic announces itself honestly.
How to check compliance in your own logs
You do not need to settle the debate in general. You need to know what reaches your own server. Our guide to finding AI crawlers in server logs covers the basics; here is the compliance check.
Pick a path you disallow for the crawler in question, and note the date and time you changed robots.txt. Allow for the documented delay: about 24 hours for OpenAI's search systems.
Count robots.txt fetches by that agent. A crawler that never requests the file cannot be reading your rules.
Check the source IP. Anyone can send a user agent string, so confirm that suspicious hits come from the operator's published ranges. Verifying real AI crawlers shows how. Requests that claim to be a bot but come from elsewhere are not evidence about that bot.
Look at what the response was. A 200 on a disallowed path means your server answered it; a 403 or 429 means a rule you set refused it.
If a verified request reaches a path you disallowed, after the delay, from an operator that documents robots.txt support, you have a specific finding. Report it to the operator and enforce at the edge in the meantime. If the agent is a user-triggered fetcher, the visit may be exactly what its documentation describes.
When to enforce with a WAF, rate limits or authentication
Use robots.txt for preferences you are comfortable having honored voluntarily. Use something stronger when a request must not succeed.
Authentication. Content that must stay private belongs behind a login. robots.txt does not protect it, and a listed path can even point to it. Anthropic says its bots will not attempt to bypass CAPTCHAs, but that is a statement about one operator, not a control you can rely on for everything else.
Firewall or WAF rules. Match the user agent token and, where the operator publishes ranges, the source IP. Matching only the user agent stops honest bots and nothing else. Cloudflare's settings are covered in Cloudflare and AI crawlers.
Rate limits. If the problem is load rather than permission, return 429 with a Retry-After header instead of blocking. See rate limiting AI crawlers.
Enforcement has a cost. OpenAI says sites that opt out of OAI-SearchBot do not appear in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot or Claude-User may reduce visibility in search results. Before you block a search or user agent at the firewall, decide whether you are willing to lose that access.
What you cannot verify
Be honest about the gaps before you write a policy:
Current compliance of any named crawler on your site. Published studies are snapshots. Only your logs show today's behavior.
Undeclared crawlers. A bot that hides its identity is invisible to a user-agent rule, and robots.txt cannot name it.
What happens to content already collected. The documentation covered here describes how crawlers treat your robots.txt going forward. It does not describe retroactive removal.
Operators with no published position. If a crawler's operator documents nothing about robots.txt, you cannot assume it reads the file.
Whether a visit is a crawl or a user action. From the logs alone, ChatGPT-User and a crawler can look similar; the operator's documentation is the only guide to intent.
Frequently asked questions
Do AI crawlers respect robots.txt?
The ones whose operators document it, such as OAI-SearchBot, GPTBot and Anthropic's bots, say they do. User-triggered fetchers such as ChatGPT-User and Perplexity-User are documented as not reliably following it. Independent measurement found compliance varies by bot and drops as rules get stricter.
Does ChatGPT-User obey robots.txt?
OpenAI says robots.txt rules may not apply to ChatGPT-User, because the visits happen at a user's request. Use OAI-SearchBot rules for ChatGPT search, and a firewall rule if you need to stop ChatGPT-User requests.
Is robots.txt legally binding?
Not as a technical matter: RFC 9309 says the rules are not access authorization. Whether ignoring it has legal consequences depends on your jurisdiction and facts, and this post does not give legal advice.
How do I know a bot really ignored my rules?
Check the logs for requests to a disallowed path after the documented delay, then confirm the source IP belongs to the operator. A request that only carries the right user agent could be anyone.
Should I stop using robots.txt?
No. It is still the documented opt-out for training and search crawlers at the major operators. Keep it for honest crawlers and add enforcement for anything that matters.
Check what crawlers can reach
The free check reads one page the way AI crawlers do, with no account needed, and shows the robots.txt rules that apply to each named AI crawler. For a whole site, an audit covers up to 500 pages with evidence and prioritized fixes. Your own server logs remain the proof of what actually visited.
Filed under AI crawlers. Topics: · robots-txt, ai-crawlers, bot-protection, server-logs
User agents are easy to fake. Verify GPTBot IPs and other AI crawlers with published IP lists, reverse DNS or Cloudflare verified bots, then deal with fakes.
PerplexityBot indexes pages for Perplexity search; Perplexity-User fetches pages on request. How to allow, block and verify each, and what blocking changes.
Which Cloudflare settings block AI bots: AI crawler policies, AI Crawl Control, Bot Fight Mode, managed robots.txt and WAF rules, and how to allow the ones you want.