GPTBot vs OAI-SearchBot vs ChatGPT-User: OpenAI's crawlers explained
GPTBot vs OAI-SearchBot vs ChatGPT-User: what each OpenAI agent does, how robots.txt applies to each, and how to verify them by IP.
By Crawlito · 7 min read
In short
GPTBot collects content that may be used to train OpenAI's foundation models; your OAI-SearchBot rule decides whether your pages can be shown in ChatGPT search answers.
OpenAI says the GPTBot and OAI-SearchBot robots.txt controls are independent, so you can disallow GPTBot and still allow OAI-SearchBot to stay eligible for ChatGPT search.
ChatGPT-User fetches pages when a person asks ChatGPT to, and OpenAI says robots.txt rules may not apply to it because the request is user-initiated.
OpenAI publishes a JSON list of IP ranges for each of its agents, so you can check whether a request claiming to be GPTBot or OAI-SearchBot comes from an OpenAI address.
OpenAI says its search systems can take about 24 hours to react to a robots.txt change.
OpenAI runs separate agents for separate jobs. GPTBot crawls content that may be used to train OpenAI's foundation models, OAI-SearchBot crawls so pages can be shown in ChatGPT search, and ChatGPT-User fetches a page when a person in ChatGPT asks for it. Blocking GPTBot does not remove you from ChatGPT search: only OAI-SearchBot controls that, and OpenAI documents the two settings as independent.
Everything below comes from OpenAI's Overview of OpenAI Crawlers, read on 8 November 2026. The page used to live at platform.openai.com/docs/bots, which now redirects there, so update any old bookmarks or citations.
What each OpenAI agent does
OpenAI documents four agents. Most sites only need to think about the first three.
Agent
Job
Governed by robots.txt?
Affects ChatGPT search?
IP list
OAI-SearchBot
Crawls so sites can be surfaced in ChatGPT's search features
Yes
Yes: opted-out sites are not shown in search answers
A few details from OpenAI's page matter in practice:
Shared crawl. If you allow both OAI-SearchBot and GPTBot, OpenAI says it may use the results of one crawl for both purposes so it does not crawl twice. So a low number of GPTBot hits in your logs is not, on its own, evidence that your content is excluded from training.
Navigational links. Sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, but OpenAI says they can still appear as navigational links.
OAI-AdsBot only visits pages submitted as ads, OpenAI says, and its data is not used to train foundation models. It may use the page content for ad relevance. Unless you run ads on ChatGPT, it has no reason to visit you.
How robots.txt applies to GPTBot and OAI-SearchBot
Both crawlers read robots.txt and match the group that names them. Their product tokens are GPTBot and OAI-SearchBot. Under RFC 9309, a crawler that finds a group naming it follows that group and ignores the User-agent: * group. That catches people out: if you add a named OAI-SearchBot group to allow it, any general Disallow rules you had under * no longer apply to OAI-SearchBot unless you repeat them. The robots.txt groups guide walks through this in detail.
The reverse also matters. A file that only says this blocks every OpenAI crawler that obeys robots.txt, search included:
User-agent: *
Disallow: /
If that is on your live site by accident (a staging leftover is a common cause), OAI-SearchBot is opted out and your pages will not be shown in ChatGPT search answers, whatever you did about GPTBot.
Why ChatGPT-User is different
ChatGPT-User is not an automatic crawler. OpenAI says it is "not used for crawling the web in an automatic fashion" and that, because its actions are initiated by a user, "robots.txt rules may not apply." OpenAI also says ChatGPT-User is not used to decide whether content may appear in Search, and tells site owners to use OAI-SearchBot in robots.txt to manage search opt-outs and automatic crawling.
So a line like User-agent: ChatGPT-User with Disallow: / is not a dependable way to stop ChatGPT from opening your pages, even though many older templates include it. If you truly need to refuse those requests, that is a server or firewall decision based on the published IP list, and it means people who paste your URL into ChatGPT get nothing back.
Example robots.txt combinations
Here are the configurations most sites choose between. Each one is a complete file you can adapt; add your own Sitemap: line and any rules for other crawlers.
Allow search, block training
This is the combination OpenAI itself describes as an example: allow OAI-SearchBot, disallow GPTBot. Your pages stay eligible for ChatGPT search, and you signal that your content should not be used for training.
Note that OAI-SearchBot here ignores the /admin/ rule, because it has its own group. If /admin/ should be off limits to it too, add Disallow: /admin/ under User-agent: OAI-SearchBot. The same pattern for other vendors is covered in how to allow AI search but block AI training.
Allow both
You do not need any OpenAI-specific lines for this. With no group naming them, GPTBot and OAI-SearchBot follow your * group. Being explicit does no harm and makes intent obvious to the next person who edits the file:
Your pages will not be shown in ChatGPT search answers, though they can still show up as navigational links, and ChatGPT-User may still fetch a page a user asks about.
Block training for one section only
Rules are path-based, so you can opt part of a site out of training while leaving the rest open to GPTBot:
Because GPTBot now has its own group, it ignores your * group. Repeat any general rules here that should apply to it as well.
How long changes take
For search results, OpenAI says its systems can take about 24 hours to adjust after you update robots.txt. Wait a day before concluding a change did not work.
How to verify a request really comes from OpenAI
Anyone can put GPTBot in a user-agent header. To know a request is genuine, check its source IP against the JSON file OpenAI publishes for that agent. Each file has a creationTime and a prefixes array of ipv4Prefix CIDR ranges:
The example above is the start of searchbot.json as it stood on 8 November 2026. The files change often: at the time of writing, chatgpt-user.json had a creationTime of 6 November 2026, two days earlier. Do not copy the ranges into a config file by hand: fetch them on a schedule instead.
This one-liner checks whether an IP from your logs falls inside OAI-SearchBot's published ranges. Swap the file name for gptbot.json or chatgpt-user.json as needed:
bash
IP=104.210.140.130
curl -s https://openai.com/searchbot.json \
| python3 -c "import json,sys,ipaddress; ip=ipaddress.ip_address(sys.argv[1]); d=json.load(sys.stdin); print(any(ip in ipaddress.ip_network(p['ipv4Prefix']) for p in d['prefixes'] if 'ipv4Prefix' in p))" "$IP"
It prints True for an IP inside the ranges and False otherwise. Treat a request that claims to be GPTBot from an IP outside the current gptbot.json as not coming from OpenAI's GPTBot. For the same check across other vendors, see how to verify a real AI crawler.
What the user-agent strings look like
OpenAI's example strings, whose version numbers may change:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Match on the token (GPTBot, OAI-SearchBot, ChatGPT-User), not the full string. Older guides quote /1.0, /1.1 or /1.3 versions, and OAI-SearchBot's string now starts with a full Chrome-on-macOS prefix, so exact-match rules break. OpenAI also says that when fetching robots.txt it may add a robots.txt; marker to the user agent (for example ...OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot), which lets you tell those fetches apart in your logs.
How to check whether OpenAI's crawlers can reach you
robots.txt is only half of access. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges. A CDN or WAF rule that challenges or blocks those IPs can keep you out of ChatGPT search even when robots.txt is correct.
Read your live robots.txt, not the one in your repository: curl -s https://example.com/robots.txt. Look for a group naming GPTBot or OAI-SearchBot, and for a blanket Disallow: / under *.
Look for OpenAI agents in your access logs. With the standard combined log format, this counts requests per IP and status code:
The output columns are count, IP and status code. A run of 403s, or of challenge responses, to IPs that pass the check above usually points at a firewall or bot rule. More on reading logs in finding AI crawler visits in server logs.
Send a request with the crawler's user agent.curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://example.com/ shows whether a rule keyed on the user agent blocks it. A 200 from your own machine is not proof, because firewalls often judge by IP too, but a 403, or a cf-mitigated: challenge header on Cloudflare, is a clear sign of a problem.
Check your CDN's bot settings. On Cloudflare, AI crawler policies and Bot Fight Mode can block or challenge these agents; the Cloudflare AI crawler settings guide lists where to look.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT search?
No. GPTBot only signals whether your content may be used for training. Search eligibility is controlled by OAI-SearchBot, and OpenAI documents the two controls as independent.
Does ChatGPT-User obey robots.txt?
Not reliably. OpenAI says ChatGPT-User acts on behalf of a user and that "robots.txt rules may not apply." It is also not used to decide search eligibility, so manage search with OAI-SearchBot instead.
If I block OAI-SearchBot, does my site disappear from ChatGPT entirely?
Not entirely. OpenAI says opted-out sites are not shown in ChatGPT search answers but can still appear as navigational links, and ChatGPT-User may still open a page a user asks about.
How long after a robots.txt change does ChatGPT search react?
OpenAI says to allow about 24 hours for its search systems to adjust after you update robots.txt.
How do I know a GPTBot request is real?
Check the source IP against the current gptbot.json. If the IP is not in any listed range, treat the request as not coming from OpenAI's GPTBot, whatever its user agent says.
Next step
If you are not sure what your live robots.txt, CDN and firewall say to each OpenAI agent, run the free single-page check on one of your pages. It needs no account, and it is a quick way to see where to start with the checks above.
Filed under AI crawlers. Topics: · gptbot, oai-searchbot, chatgpt, robots-txt, ai-crawlers
AI training crawlers vs search crawlers vs user-triggered fetchers: what each does, what blocking costs, and how to opt out of training but stay in AI search.