How ChatGPT search finds sources: what OpenAI documents
How ChatGPT search finds and fetches pages: OAI-SearchBot, ChatGPT-User, search providers, robots.txt and noindex, and what OpenAI does not document.
By Crawlito · 6 min read
In short
- OpenAI says OAI-SearchBot is the crawler used to surface websites in ChatGPT's search features, and sites that disallow it will not be shown in ChatGPT search answers, though they can still appear as navigational links.
- ChatGPT-User fetches pages for certain user actions, is not used for automatic crawling, and OpenAI says it is not used to decide whether content may appear in search.
- OpenAI's help centre says ChatGPT search typically rewrites a prompt into one or more targeted queries and may send them to third-party search providers.
- GPTBot is the training crawler; OpenAI says each robots.txt setting is independent, so blocking GPTBot does not remove a site from ChatGPT search.
- OpenAI does not document a submission form, sitemap handling, ranking factors or how it chooses which sources to cite.
On this page
ChatGPT search finds sources in two documented ways. It rewrites your prompt into one or more search queries, which it may send to third-party search providers, and it relies on OpenAI's own crawler, OAI-SearchBot, to surface websites in its search results. When a user asks ChatGPT to open a specific page, a separate agent, ChatGPT-User, fetches it on their behalf. OpenAI documents which robots.txt rules control each part, but not how it ranks or chooses the sources it cites.
This guide separates what OpenAI has published from what it has not, so you can check the parts you control.
The three OpenAI agents and what each does
OpenAI's crawler documentation lists its user agents and says each robots.txt setting is independent of the others:
| Agent | Purpose according to OpenAI | robots.txt |
|---|---|---|
OAI-SearchBot | Surfacing websites in search results in ChatGPT's search features | Obeyed; OpenAI recommends allowing it if you want to appear |
ChatGPT-User | Certain user actions in ChatGPT and Custom GPTs; not automatic crawling | May not apply, because a user initiates the request |
GPTBot | Making OpenAI's generative AI foundation models more useful and safe | Obeyed; disallowing it signals content should not be used for training |
OpenAI also lists OAI-AdsBot, which only visits pages submitted as ads on ChatGPT to check their safety. It is not part of how search finds organic sources.
Two lines in the documentation matter most for search visibility:
- Sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, but can still appear as navigational links.
- ChatGPT-User is not used to determine whether content may appear in search. OpenAI says to use OAI-SearchBot in robots.txt to manage search opt-outs.
If your site allows both GPTBot and OAI-SearchBot, OpenAI says it may use the results of one crawl for both purposes.
How a ChatGPT search answer is put together
OpenAI's help article on ChatGPT search describes the request side. When ChatGPT searches the web, it typically rewrites the prompt into one or more targeted queries and may send those to third-party search providers. Its example turns a broad question into a narrower query with a topic and a year. With memory turned on, the rewrite may use relevant saved memories. OpenAI says it does not share the user's IP address or account information with search providers to run the search, though a general location may be shared.
The answer then links to its sources, and a Sources panel lists the cited pages and other relevant links. OpenAI warns that search results and citations can be incomplete or incorrect.
Put together, a page can be reached in a few documented ways:
- It is in the search providers' results for one of the rewritten queries.
- It is in OpenAI's own crawled data, gathered by OAI-SearchBot.
- A user asks about it directly, and ChatGPT-User fetches it.
What OpenAI does not publish is how these combine: which provider or providers it uses, how results from its own crawl and from providers are merged, or how it decides which pages to cite. The help article does not name the search providers. Treat any claim that ChatGPT search "uses only Bing" or "uses only its own index" as unverified unless OpenAI publishes it.
The idea of rewriting one prompt into several searches is often called query fan-out; query fan-out explained covers what that means for content.
What you control: robots.txt
robots.txt is the main documented control. To be eligible for ChatGPT search while opting out of training:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /To opt out of ChatGPT search entirely, disallow OAI-SearchBot. Remember that your pages may still appear as plain navigational links, according to OpenAI.
Two cautions:
- Groups do not inherit. A named
OAI-SearchBotgroup replaces yourUser-agent: *rules for that crawler. If you only need OAI-SearchBot to follow the same rules as everyone else, you do not need a named group at all. - Changes take time. OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust.
GPTBot vs OAI-SearchBot vs ChatGPT-User has more combinations and the user-agent strings.
What you control: noindex and access
OpenAI's publishers and developers FAQ describes a noindex robots meta tag as a way to keep a page out of ChatGPT search, and notes that the crawler must be allowed to fetch the page to see the tag. That has a practical consequence: if you disallow a page in robots.txt and also add noindex, the crawler never sees the noindex.
The same FAQ says ChatGPT adds utm_source=chatgpt.com to referral URLs, which lets you identify visits from ChatGPT search in analytics.
Beyond those rules, the crawler has to be able to read the page. Nothing OpenAI publishes says its crawlers run JavaScript, so content that only appears after scripts run is at risk. Firewalls and bot protection that challenge unknown clients can also block OAI-SearchBot. OpenAI publishes IP ranges for each agent, linked from its crawler page, so you can allow the real crawler and verify traffic that claims to be it.
When a user pastes your URL
If someone asks ChatGPT about a specific page, the fetch comes from ChatGPT-User, not OAI-SearchBot. OpenAI says robots.txt may not apply to these user-initiated requests, so a robots.txt block on OAI-SearchBot does not stop them. A firewall rule that blocks the ChatGPT-User user agent or OpenAI's published ChatGPT-User IP ranges does, and the user then gets an error instead of your content. Decide deliberately whether that is what you want, and check your firewall for broad "AI bot" rules that catch it by accident.
What OpenAI does not document
Being clear about the gaps avoids wasted work:
| Question | Documented? |
|---|---|
| A form or console to submit a site to ChatGPT search | No |
| Whether OAI-SearchBot reads XML sitemaps | Not on OpenAI's crawler page |
| Which third-party search providers are used | Not named in the help article |
| Ranking or citation factors | No |
| Whether structured data affects inclusion | No |
| Whether llms.txt is read | No |
| How often OAI-SearchBot recrawls a page | No |
A sitemap and good internal links still help any crawler discover pages, so keep them. Just do not expect a documented ChatGPT-specific effect. Submitting a website to ChatGPT, Perplexity and Copilot covers what each platform does and does not offer.
How to check that ChatGPT can reach your pages
- Read your live robots.txt and find the group OAI-SearchBot obeys:
curl -s https://www.example.com/robots.txt- Fetch a page with OAI-SearchBot's user agent and confirm you get
200and real content, not a challenge page:
curl -s -o /dev/null -w "%{http_code}\n" \
-A "compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
https://www.example.com/your-pageThe string above is the part OpenAI documents; real requests prefix it with a browser-like string, and the version number changes over time. This only tests user-agent rules. Firewalls that check IP addresses treat your machine differently from OpenAI's.
- Look for the crawler in your logs. Search for
OAI-SearchBotandChatGPT-User, check the status codes they received, and verify the IPs against OpenAI's published lists. - Search for a noindex in the page's HTML and in the
X-Robots-Tagresponse header. - Check referrals for
utm_source=chatgpt.comand thechatgpt.comreferrer.
If any step fails, why ChatGPT can't see your website works through the usual causes one by one.
Frequently asked questions
Does ChatGPT search use Bing?
OpenAI's help article says ChatGPT search may send rewritten queries to third-party search providers, but it does not name them. Treat claims about a single provider as unverified.
Can I submit my website to ChatGPT search?
OpenAI does not document a submission form. Allow OAI-SearchBot in robots.txt, make sure pages are reachable and readable without JavaScript, and keep a sitemap and internal links for discovery.
If I block GPTBot, will I disappear from ChatGPT search?
No. OpenAI says GPTBot and OAI-SearchBot settings are independent. ChatGPT search inclusion depends on OAI-SearchBot.
Does ChatGPT-User obey robots.txt?
OpenAI says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. It also says ChatGPT-User is not used to decide whether content appears in search.
How does ChatGPT decide which sources to cite?
OpenAI does not publish its ranking or citation criteria. Removing technical barriers makes a page eligible; it does not decide whether it is cited.
Check your page against OpenAI's crawlers
The parts of ChatGPT search you control are access and readability. To see how OAI-SearchBot and other AI crawlers are treated by your robots.txt and whether your content is in the HTML they receive, run the free check on a page you care about.
Filed under Strategy and audits. Topics: chatgpt, oai-searchbot, ai-search, robots-txt, ai-crawlers