How long robots.txt changes take to affect AI crawlers
How long robots.txt changes take to reach AI crawlers: the delays OpenAI, Meta, Perplexity and DuckDuckGo document, and how caching and errors change it.
By Crawlito · 10 min read
In short
OpenAI says its search systems need about 24 hours to adjust after a robots.txt update, Perplexity and Meta say up to 24 hours, and DuckDuckGo says a DuckAssistBot opt-out takes effect after 72 hours.
RFC 9309 says crawlers should not use a cached robots.txt for more than 24 hours unless the file is unreachable, and that SHOULD NOT is not a hard limit.
A bot can keep showing up in your logs after you block it: it may be using a cached copy, it may be a user-triggered fetcher that does not follow robots.txt, or the user agent may be spoofed.
A 4xx on robots.txt means no restrictions under RFC 9309, and a 5xx means assume everything is disallowed, so a deploy that briefly returns 503 can change what crawlers do.
Amazon says its bots may use a copy of robots.txt cached within the last 30 days, which is a reason to plan changes days ahead rather than minutes.
Where operators give a figure, it is usually about a day. OpenAI says about 24 hours for its search systems, Perplexity and Meta say up to 24 hours, Amazon says about 24 hours per setting, and DuckDuckGo says 72 hours for DuckAssistBot. Those are the operators' own statements at the time of writing. Anthropic's public help article does not give a figure at all, so treat a day as a planning number, not a promise.
The delay mostly comes from caching. Crawlers store a copy of your robots.txt so they do not fetch it before every page, and the copy they hold is the one they obey until it expires. The rest of this post covers the documented timings, why a bot can still appear in your logs after you block it, what an error on robots.txt does, and how to roll out a change without locking yourself out.
How long do robots.txt changes take for each AI crawler?
Here is what each operator's own documentation says. None of these are guarantees, and some pages cover several crawlers with one sentence.
Operator
What it documents
Source
OpenAI
"~24 hours" for its systems to adjust, stated for OAI-SearchBot search results. The page gives no timing for GPTBot. For ChatGPT-User it says robots.txt rules may not apply.
Each user agent setting may take about 24 hours to reflect. It also says its listed bots (Amazonbot, Amzn-SearchBot and Amzn-User) may use a copy cached within the last 30 days.
Two things are worth reading closely. Amazon's page gives both a 24-hour figure and a 30-day cache window, and it does not reconcile them, so for Amazon's bots a longer wait is possible. And OpenAI's "about 24 hours" is stated for search results only.
What RFC 9309 says about caching robots.txt
RFC 9309, the robots.txt standard, has a short caching section. Crawlers may cache the file, and they should not use the cached version for more than 24 hours unless the file is unreachable.
That wording is a SHOULD NOT, not a MUST NOT, and the unreachable exception is explicit: if a crawler cannot fetch a fresh copy, it may keep using the old one. The operator pages above are the better guide for any one bot.
Google adds a detail. It may adjust how long it caches the file based on Cache-Control: max-age headers, and its cache may be shared across its crawlers. So a long max-age on your own robots.txt can lengthen the delay for Google. The other operators above do not say whether they read that header.
Why does a bot still show up in logs after you block it?
You edit robots.txt, check the file, and the next morning the bot is in your access log. That is often normal. Work through these causes in order.
The crawler is still on its cached copy. Within the windows above, a bot can legitimately act on the old rules. Compare the time of your change with the documented delay first.
The request is for robots.txt itself. A bot that fetches /robots.txt is doing what it should. Filter your logs to requests for other paths before counting hits.
It is a user-triggered fetcher. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Amazon says Amzn-User may not follow all directives. A visit from one of these after a Disallow is documented behaviour, not a bug. Allow AI search, block AI training explains how the three crawler purposes differ.
The user agent is spoofed. Anyone can send a request that says it is GPTBot. Check the source IP against the operator's published list before you conclude a real crawler ignored your rules. AI crawlers in server logs shows how.
The rule does not match the host or the token. Amazon says its bots check robots.txt per host, so blog.example.com has its own file. A typo in a token, or a named group that does not include the paths you meant, leaves the bot allowed. Test robots.txt against each AI crawler walks through group selection.
robots.txt did not return what you think. A cache or a 4xx can mean the bot never saw your rule. The next two sections cover those.
The operator pages cited here do not say whether URLs already queued for a crawl are dropped when the rules change. If a bot keeps requesting disallowed paths well beyond its documented window, from verified IP addresses and for a bot that says it follows robots.txt, block it at your firewall or server instead of waiting. robots.txt is a request, not access control.
Is your own cache delaying the change?
The crawler is not the only place a stale copy can sit. Your CDN, reverse proxy or hosting platform may serve an old robots.txt even after you updated the origin.
Cloudflare's default cache behaviour caches robots.txt by default, with a 120-minute default Edge TTL for 200 responses when the origin sends no cache-control or expires header. Other CDNs and platforms have their own rules, so check yours. Look at the response headers of the live file:
A large Age value or cf-cache-status: HIT after your edit means a cache is serving the old file. Purge that URL on your CDN, then fetch again. Do this first, because a crawler cannot read a file your edge is not serving.
Also set a sensible lifetime. A short Cache-Control: max-age on robots.txt, an hour or so, keeps your own edge from adding a long delay on top of the crawler's. Google says it may adjust its cache by max-age, so avoid a multi-day value.
What does a 4xx or 5xx on robots.txt do to crawlers?
The status code of the file changes how a crawler reads your whole site, and it interacts with caching. Status codes and redirects for AI crawlers has the longer treatment. The short version:
robots.txt response
RFC 9309
Google
Amazon
2xx
Follow the parseable rules
Process the file
Not stated on its page
3xx
Follow at least five consecutive redirects
Follows at least five hops, then treats it as a 404
Not stated on its page
4xx
Unavailable: crawler may access any resource
Treated as no robots.txt, except 429
Not stated on its page
5xx or unreachable
Assume complete disallow
Stops crawling for 12 hours, then uses the last good copy for up to 30 days
If the file cannot be fetched, behaves as if it does not exist (no status codes named)
Amazon's row for errors comes from its statement that if a file cannot be fetched, its bots behave as if it does not exist. It does not break that down by status code. For crawlers whose pages do not describe error handling, the RFC is the only written baseline, and it is a baseline, not a statement of what each bot does.
Two consequences follow, and they are why status codes matter for timing:
A 403 or 404 on robots.txt restricts nothing. If your firewall returns 403 to bots on /robots.txt, the RFC reads that as "no restrictions", which removes every rule you wrote.
A 5xx can block everything, briefly or not. Under the RFC, a crawler that gets a 500 or 503 must assume complete disallow. Google's own behaviour is gentler, but a deploy that returns 503 on the file at the wrong moment can still stall a crawler. A crawler may also keep a cached copy if the file stays unreachable, which means an outage can freeze the old rules in place instead of applying your new ones.
How do you confirm which robots.txt version a crawler fetched?
Your server log records that a crawler requested /robots.txt, when, and what you returned. That is enough to confirm a fetch happened. It does not by default tell you which version of the file was served. Three ways to close the gap:
Compare the response size. The log's bytes-sent field changes when the file changes. Note the size of the old and new versions, then match it against the logged requests after the edit.
Add a version header and log it. Set a header on the file that names the version, and write it into the log. This is a technique, not something operators document. In nginx:
Bump the string each time you change the file. The header is public, so keep it to a date and a letter.
Check the first fetch after the edit. Find the earliest /robots.txt request from the bot after your deploy and read its status and version. A fetch before your deploy tells you nothing about the new rules.
Match on the token, not the whole user agent string, because version numbers change. For Google, the robots.txt report in Search Console shows the fetch status and when Google last checked each file, and lets you request a recrawl of it. Google notes that a recrawl request does not guarantee that newly unblocked URLs are recrawled immediately. The other operator pages cited here do not mention an equivalent button.
How to avoid lockouts when you change rules
Because the old rules can stay in force for a day or more, a mistake is slow to undo. A wrong Disallow: / cached by a crawler stays cached after you fix it. A few habits keep that from costing you.
Test before you deploy. Run the new file through the checks in Test robots.txt against each AI crawler, including group selection for each token you care about.
Keep staging out of production. The classic lockout is a staging Disallow: / that ships with a release. Make the file a separate, reviewed artifact, and see staging noindex leaks for how these get out.
Deploy it atomically. Replace the file in one step so it never returns a 5xx, an empty body or an HTML error page mid-deploy. Under the RFC, an unreachable file can mean "disallow all".
Change one thing at a time. If you allow one bot and block another in one release, you cannot tell which change caused a visit pattern.
Keep the previous version. Store each file in version control so rollback is one commit. Expect the rollback to take as long to reach crawlers as the original change did.
Order allow changes correctly. If you are letting a bot in, change robots.txt first, then loosen the firewall rule, so the bot is not briefly allowed in one place and denied in the other. If you are keeping a bot out, do not rely on waiting: robots.txt is voluntary, and a server or CDN rule does not depend on the crawler's cached copy.
robots.txt for AI crawlers has copy-ready templates if you need a starting file.
Frequently asked questions
How long does OpenAI take to notice a robots.txt change?
OpenAI says about 24 hours for its search systems to adjust after a robots.txt update. Its page gives no timing for GPTBot, and it says robots.txt rules may not apply to ChatGPT-User.
Can I force a crawler to re-read robots.txt right away?
For Google, you can request a recrawl of the file in Search Console, though that does not guarantee immediate recrawling of newly unblocked URLs. The AI crawler operators cited here do not document a way to force a refetch, so wait out the cache and watch your logs.
Does a 404 on robots.txt block AI crawlers?
No. Under RFC 9309 a 4xx means the file is unavailable and the crawler may access any resource. Amazon also says its bots behave as if the file does not exist when they cannot fetch it.
Why is a bot still crawling after I disallowed it?
It may still hold a cached copy, it may be a user-triggered fetcher that does not follow robots.txt, or the user agent may be spoofed. Check the documented delay, then verify the IP address before treating it as a violation.
Should I wait for the crawler or block it at the firewall?
If you need the traffic to stop now, use the firewall or server. robots.txt is a request that crawlers apply on their own schedule, so use it for policy and your edge for enforcement.
Check what your site serves now
The free check fetches the live robots.txt for one page and shows which rule applies to each named AI crawler, with no account needed. It tells you what the file says today; it cannot tell you what a crawler has cached. For a whole site, an audit checks robots.txt, status codes and firewall behaviour across up to 500 pages and lists the fixes in priority order.
Filed under AI crawlers. Topics: · robots-txt, ai-crawlers, caching, troubleshooting
Redirect chains, soft 404s and wrong status codes can stop AI crawlers reaching a page. How 3xx, 4xx and 5xx are read, robots.txt errors, and curl checks.