Why ChatGPT can't see your website: a troubleshooting guide

If ChatGPT can't see your website, the cause is usually one of six things: robots.txt blocks OpenAI's search crawler, a CDN or firewall challenges it, the content only exists after JavaScript runs, a noindex rule opts the page out, the page is too new or not linked from anywhere, or the content sits behind a login. Work through them in that order. Each section below gives the symptom, a way to confirm it in a few minutes, and the fix.

One honest caveat before you start. Fixing these removes technical barriers. It makes your pages possible to read, not certain to be quoted. Which sources ChatGPT picks for an answer depends on many things outside your site.

Which OpenAI crawler matters for ChatGPT search?

OpenAI documents four agents on its crawler overview page. Only one of them decides whether you can show up in ChatGPT search results.

OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links. The GPTBot and OAI-SearchBot settings are independent, so blocking GPTBot to opt out of training does not remove you from search. If you want the full breakdown, read GPTBot, OAI-SearchBot and ChatGPT-User explained.

Cause 1: robots.txt blocks OAI-SearchBot

This is a frequent cause, and often an accident: a copied "block all AI bots" list that includes every OpenAI token, or a blanket rule that catches everything.

Symptom: ChatGPT search never lists your pages as sources, and your server logs show OAI-SearchBot requesting /robots.txt but few or no pages.

How to confirm: fetch the file and look for any group that applies to OAI-SearchBot.

curl -s https://example.com/robots.txt

Three patterns block it. A named group that disallows everything:

User-agent: OAI-SearchBot
Disallow: /

A blanket rule with no OpenAI group to override it:

User-agent: *
Disallow: /

A named group that shuts out the paths you care about:

User-agent: OAI-SearchBot
Disallow: /blog/

Remember how groups work under RFC 9309: a crawler obeys the group that names it and ignores *. So if you have an OAI-SearchBot group, the * rules do not apply to it at all, for better or worse.

Also check the status code. Under RFC 9309, a robots.txt that returns a 5xx error tells crawlers to assume everything is disallowed, while a 4xx such as 404 means no restrictions apply.

curl -s -o /dev/null -w "%{http_code}\n" https://example.com/robots.txt

Fix: allow OAI-SearchBot explicitly. If you want to keep opting out of training, block GPTBot separately. This is the same split OpenAI shows in its own example.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

OpenAI says its systems can take about 24 hours to adjust after you change robots.txt, so give it a day before testing again. Our robots.txt guide for AI crawlers covers the syntax in more depth.

A Disallow for ChatGPT-User is not a reliable switch either way (see the FAQ below). Manage search through OAI-SearchBot.

Cause 2: a CDN or firewall challenges the crawler

robots.txt can say yes while your edge network says no. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges. A bot rule that challenges or blocks those IPs keeps you out even when robots.txt is correct.

Symptom: robots.txt looks fine, but logs at your origin show no OpenAI requests, or your CDN's security log shows them challenged or blocked.

How to confirm: request a page with OAI-SearchBot's user agent and look at the status and headers.

curl -s -o /dev/null -D - -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://example.com/

A 403 or 429 suggests a firewall or rate-limit rule. The header cf-mitigated: challenge means Cloudflare served a challenge page: Cloudflare documents that every challenge page response carries it, though it does not mark outright blocks. Note the limit of this test: it sends the user agent from your IP, not OpenAI's, so rules based on IP or reputation can treat the real crawler differently. Your CDN's security events log is the better evidence.

Fix on Cloudflare: check these settings, all described in Cloudflare's AI crawler policy docs:

• AI Crawler and Agent access policies (Security > Settings, filtered by Bot traffic). Each policy applies to verified bots with that behavior, so if the Search policy is set to Block, Cloudflare blocks the bots it classifies as search crawlers. Set Search to Allow; you can still set Training to Block or Disallow.
• AI Crawl Control. Check that OAI-SearchBot is set to Allow in the crawler list, since that is where you can see how your zone treats it.
• Bot Fight Mode (Free plan). It challenges traffic that matches bot patterns across the whole domain, and Cloudflare says it cannot be bypassed with WAF custom rules. If it is catching crawlers you want, turn it off or move to a plan with Super Bot Fight Mode, where a WAF Skip rule works.

On other WAFs, write an allow rule that matches both the OAI-SearchBot token in the user agent and the IP ranges OpenAI publishes at openai.com/searchbot.json. That file changes, so fetch it automatically rather than pasting the ranges into a config by hand. Cloudflare and AI crawlers walks through each setting.

Cause 3: the content only exists after JavaScript runs

If your page is a client-side rendered app, the HTML the server sends may be little more than an empty <div id="root"></div> and some script tags. A browser fills it in. A crawler that does not run JavaScript sees the empty shell.

OpenAI does not document whether OAI-SearchBot renders JavaScript. The best public measurement is Vercel and MERJ's December 2024 analysis, which found that the major AI crawlers it observed, including OpenAI's, Anthropic's and Perplexity's, did not execute JavaScript at the time. ChatGPT's crawler downloaded script files but did not run them. The exceptions it reported were Gemini, which uses Googlebot's infrastructure, and AppleBot. Treat that as a snapshot, not a permanent rule, but plan for it.

Symptom: your site looks complete in a browser, but ChatGPT describes it vaguely, gets facts wrong, or seems to know only your page title.

How to confirm: fetch the raw HTML and search it for a sentence you can see on the page.

curl -s -A "OAI-SearchBot/1.4" https://example.com/pricing | grep -c "Plans start at"

A count of 0 means the sentence is not in the HTML the server sent. You can also open view-source: in your browser (not the Elements panel, which shows the rendered page).

Fix: serve the main content in the initial HTML with server-side rendering or static generation. Most frameworks support this: Next.js App Router pages are server components by default, and Nuxt, SvelteKit and Astro produce HTML on the server or at build time. Content you load in a useEffect or after a click is still missing, so move product details, prices and key copy into the server render. In Next.js, content inside a Suspense boundary that resolves at request time is streamed in a hidden element near the end of the response, so keep key copy outside those boundaries. Client-side rendering and the empty HTML problem shows the patterns.

Cause 4: a noindex rule opts the page out

A leftover noindex from a staging site, a CMS setting or a plugin can hide pages without anyone noticing.

Symptom: some pages show up as sources and others never do, often a whole section or template.

How to confirm: check both the HTML and the response headers.

curl -s https://example.com/page | grep -io '<meta[^>]*robots[^>]*>'
curl -sI https://example.com/page | grep -i x-robots-tag

OpenAI's publishers and developers FAQ describes a noindex meta tag as the way to keep a page out of ChatGPT search, and notes the crawler has to be able to fetch the page to see the tag. So noindex is the deliberate opt-out; if you see it on pages you want found, it is your culprit. The FAQ talks about the meta tag, so check the X-Robots-Tag header as well, since many servers set noindex that way. OpenAI's crawler page does not say how it treats nosnippet. Google does document that nosnippet keeps content from being used as a direct input to AI Overviews and AI Mode, so check for it too if Google's AI features matter to you.

On WordPress, the "Discourage search engines from indexing this site" box under Settings > Reading adds a noindex, nofollow robots meta tag to every page (since WordPress 5.3; older versions wrote Disallow: / to robots.txt instead). It is easy to leave ticked after a launch.

Fix: remove the tag or header from pages you want found, then check the template or plugin that added it so it does not come back. See noindex, nosnippet and X-Robots-Tag for how each directive behaves.

Cause 5: the page is too new or not discoverable

A crawler has to find a URL before it can read it. OpenAI does not document how OAI-SearchBot discovers new pages or how often it recrawls, so there is no OpenAI-specific setting to submit a URL.

Symptom: older pages appear as sources, but a page you published last week does not, or a page that nothing else links to never does.

How to confirm: check whether the page is linked from your navigation or other crawled pages, whether it is in your XML sitemap, and whether the sitemap is listed in robots.txt with a Sitemap: line. Then search your logs for any OpenAI request to that URL.

grep "OAI-SearchBot" access.log | grep "/new-page" | tail -5

Fix: link the page from relevant existing pages, add it to your sitemap, and give it time. These are the standard ways crawlers find URLs; none of them forces a recrawl on a schedule you control.

Cause 6: the content is behind a login or paywall

Crawlers read what an anonymous visitor gets. If the server sends a login form, a paywall stub or a sign-up wall, that is all the crawler reads.

Symptom: ChatGPT can describe your landing page but not your documentation, member articles or product catalogue.

How to confirm: open the page in a private window with no cookies, or fetch it with curl. If you see a login form or a teaser paragraph, so does the crawler.

Fix: decide what should be public. If you want a summary to be readable, publish a public abstract or overview page and keep the full content gated. Do not serve crawlers different content from users based on user agent; Google treats serving materially different content to crawlers and people as cloaking.

How to tell whether OpenAI is reaching you now

Once you have made fixes, look for evidence rather than asking ChatGPT about yourself.

1. Search your logs for the three tokens, and match on the token rather than the full string, because version numbers change.

   bash

   grep -E "OAI-SearchBot|ChatGPT-User|GPTBot" access.log | awk '{print $1, $7, $9}' | tail -20

2. Verify the IP of a request against OpenAI's published list, since anyone can fake a user agent.

   bash

   python3 -c 'import ipaddress,json,sys,urllib.request; ip=ipaddress.ip_address(sys.argv[1]); d=json.load(urllib.request.urlopen("https://openai.com/searchbot.json")); print(any(ip in ipaddress.ip_network(p["ipv4Prefix"]) for p in d["prefixes"] if "ipv4Prefix" in p))' 203.0.113.7

   Use chatgpt-user.json or gptbot.json for the other agents.

3. Check referral traffic. OpenAI's publisher FAQ says ChatGPT adds utm_source=chatgpt.com to links in search results. Look for that parameter and for chatgpt.com as the referrer, since not every visit may carry the tag.

How to measure AI crawler visits and AI referral traffic covers both in detail.

Frequently asked questions

Does blocking GPTBot stop my site appearing in ChatGPT?

No. GPTBot controls whether content may be used for training. ChatGPT search eligibility is controlled by OAI-SearchBot, and OpenAI says the two settings are independent.

How long after fixing robots.txt will ChatGPT notice?

OpenAI says its search systems can take about 24 hours to adjust after a robots.txt change. Crawling the pages themselves and using them in answers can take longer, and OpenAI does not publish a schedule.

Can I block ChatGPT-User with robots.txt?

Not reliably. OpenAI says ChatGPT-User acts on a user's request and that robots.txt rules may not apply to it. It also says ChatGPT-User does not decide search inclusion, so use OAI-SearchBot rules for that.

Everything checks out and ChatGPT still doesn't mention my site. Why?

Being reachable makes your pages eligible to be read, nothing more. Which sources ChatGPT uses depends on the question, the other pages available and factors OpenAI does not publish. Clear, specific content on the topic is the part you control.

Does ChatGPT read llms.txt?

OpenAI's crawler documentation does not say that any of its crawlers read llms.txt on other sites. Fix access, rendering and noindex first; they are documented and they matter.

Check your own page

The free check reads one page the way AI crawlers do, with no account needed: the robots.txt rules for each named AI crawler, the robots meta tag and headers, and whether the main text is in the raw HTML. It runs from Crawlito's servers, not OpenAI's IP ranges, so your CDN's security log stays the evidence for Cause 2. If the problem spans a whole site, an audit runs the checks across up to 500 pages with evidence and prioritized fixes.