AI search optimization myths: eight that waste your time

Most AI search optimization myths share one flaw: they treat a guess as a requirement. When you check the claims against what OpenAI, Google, Anthropic, Perplexity and the robots.txt standard actually publish, the list of real technical requirements is short: let the right crawlers in, serve your content in the HTML, and keep the page indexable. This post takes eight common claims, shows what is documented at the time of writing, and says what to do instead.

Myth 1: you need an llms.txt file to appear in ChatGPT

The claim: add /llms.txt and ChatGPT, Claude and Perplexity will start using your site.

What is documented: llms.txt is a proposal written by Jeremy Howard in September 2024: a Markdown file that gives language models a curated overview of a site. It is not a standard, and it does not control access. OpenAI's crawler documentation says that to help a site appear in ChatGPT search results you should allow OAI-SearchBot in robots.txt and allow requests from its published IP ranges. It says nothing about OpenAI crawlers reading llms.txt on third-party sites. Perplexity's and Anthropic's crawler pages do not say so either. OpenAI, Perplexity and Anthropic each publish an llms.txt for their own developer docs, which is a different thing from their crawlers reading yours.

Google is explicit. Its AI optimization guide says Google Search does not use llms.txt and that maintaining one for other services will neither harm nor help your visibility in Google Search.

What to do: write one if it costs you little and you have a docs-style site where a curated index helps, but do not treat it as a fix. If ChatGPT cannot see you, check OAI-SearchBot access first. Our llms.txt guide covers the format if you want one anyway.

Myth 2: there is special "AI schema" you must add

The claim: a new structured data type, or a particular set of schema.org properties, makes pages eligible for AI answers.

What is documented: Google's AI features page says you do not need new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode, and that no special schema.org structured data is required. To be eligible as a supporting link, a page must be indexed and eligible to be shown in Google Search with a snippet. Google adds that there are no additional technical requirements, and that inclusion is not guaranteed.

Structured data still has real uses: it describes your organization, products and articles in a form machines parse reliably, and it powers rich results that still exist. But some of those results are gone. Google stopped showing FAQ rich results on May 7, 2026, and removed HowTo documentation in September 2023, according to its Search Central changelog.

What to do: keep JSON-LD that is valid and matches what is visible on the page. Skip anything sold as "AI schema". See structured data for AI search for what helps and what is hype.

Myth 3: blocking GPTBot removes you from ChatGPT search

The claim: if you disallow GPTBot to keep your content out of training, ChatGPT stops showing your site.

What is documented: OpenAI runs separate agents with separate jobs. GPTBot crawls content that may be used to train its foundation models. OAI-SearchBot surfaces sites in ChatGPT's search features. OpenAI says the two robots.txt settings are independent, and its own example allows OAI-SearchBot while disallowing GPTBot. The same split exists elsewhere: Anthropic separates ClaudeBot (training) from Claude-SearchBot and Claude-User, and Google-Extended controls Gemini training and grounding without affecting Google Search inclusion or ranking.

This configuration opts out of OpenAI training and stays eligible for ChatGPT search:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

OpenAI says changes can take about 24 hours to affect search results.

What to do: decide per purpose, not per company. How to allow AI search but block AI training has a full file for each operator.

Myth 4: AI crawlers render JavaScript the way Google does

The claim: Googlebot runs JavaScript, so AI crawlers must too, and a client-side rendered app is fine.

What is documented: Google renders pages with an evergreen headless Chromium after queueing them, and it still calls server-side rendering or pre-rendering a great idea. For AI crawlers, the best public measurement is Vercel and MERJ's analysis from December 2024. It found that none of the major AI crawlers it observed rendered JavaScript, naming OpenAI's crawlers, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. ChatGPT's and Claude's crawlers downloaded JavaScript files but did not execute them. That is a snapshot from one network, not a permanent rule, so test your own pages rather than relying on it.

You can see what a non-rendering crawler receives in one command:

curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://example.com/pricing | grep -i "your main heading"

If your heading or main text is missing from that output, a crawler that does not run JavaScript cannot read it.

What to do: serve the main content in the initial HTML with server-side or static rendering. Do AI crawlers run JavaScript? explains how to test each page type.

Myth 5: AI answers need a minimum word count or an FAQ section

The claim: pages under a certain length, or without a question-and-answer block, are ignored by AI search.

What is documented: nothing. No operator page we have read sets a word count, a reading level or a required section format. Google's stated requirements for AI Overviews and AI Mode are indexing and snippet eligibility, with no additional technical requirements. Google also says AI features may use "query fan-out", running several related searches across subtopics and data sources. That describes how answers are gathered; it sets no length or format for the pages they draw on.

FAQ sections are a fine way to answer real questions. They are not a ticket into AI answers, and since May 2026 FAQPage markup no longer produces FAQ rich results in Google Search.

What to do: write for the question the reader has, at the length it needs. Make sure the answer is in plain text in the HTML, under a clear heading, and not hidden behind a tab that only loads on click.

Myth 6: one score predicts whether AI tools will cite you

The claim: a tool scores your "AI visibility" and the number tells you how likely you are to be cited.

What is documented: no AI search operator publishes a citation-likelihood score or the formula behind its source selection. What does exist is measurement after the fact. Bing Webmaster Tools added an AI Performance report in February 2026, as a public preview, that counts how often a site is cited across Microsoft Copilot, AI-generated summaries in Bing and select partner integrations. It reports citations, not clicks, and it does not cover ChatGPT, Perplexity or Google.

A readiness score can honestly measure something narrower: whether published, checkable conditions pass, such as crawler access, raw HTML content and indexability. The Crawlito score works that way. It is computed by fixed rules and measures checks, not the chance of being cited, as how the Crawlito readiness score works explains.

What to do: treat any score as a list of findings to fix. Be wary of a tool that cannot tell you which check produced which point.

Myth 7: robots.txt stops crawlers from reading your pages

The claim: a Disallow rule blocks a bot, so sensitive or premium content is protected.

What is documented: RFC 9309, the robots.txt standard, says in its introduction that "these rules are not a form of access authorization", and its security section says the protocol is not a substitute for valid content security measures. Compliance is voluntary, and operators document exceptions for user-triggered fetches:

The RFC also warns that listing paths in robots.txt exposes them publicly, so a Disallow line for a private path advertises it.

What to do: use robots.txt to state your preferences to well-behaved crawlers. Protect content that must stay private with authentication, or block at the edge with a WAF rule that matches the user agent and the operator's published IP ranges. Edge rules cut the other way too: OpenAI recommends allowing requests from OAI-SearchBot's published IP ranges as well as allowing it in robots.txt, so a CDN or firewall rule that blocks those addresses can undo a correct robots.txt.

Myth 8: your analytics already shows all your AI traffic

The claim: if AI search mattered for your site, it would show up in your analytics dashboard.

What is documented: there are two kinds of AI traffic, and most dashboards see at most one.

• Crawler visits. Tag-based analytics records a visit when its script runs in the visitor's browser. The December 2024 Vercel data says the major AI crawlers did not execute JavaScript, so their fetches leave no trace there. They do appear in server and CDN logs.
• Referral visits. People who click a link in an AI answer are real visitors and can be counted. OpenAI's publishers and developers FAQ says ChatGPT adds utm_source=chatgpt.com to referral URLs from its search results. Check the referrer host (chatgpt.com) as well as the UTM tag, as a cross-check. Google counts clicks from AI Overviews and AI Mode inside the Search Console Performance report under the Web search type, with no separate AI filter.

A quick way to count AI crawler requests in an nginx or Apache access log:

grep -Eo "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" access.log \
  | sort | uniq -c | sort -rn

User-agent strings can be faked, so confirm important findings against each operator's published IP list before acting on them.

What to do: measure crawler visits in logs and referral visits in analytics, and report them separately. How to measure AI crawler visits and AI referral traffic walks through both.

Frequently asked questions

Does llms.txt hurt my site?

Not in Google Search, which ignores the file and says it will neither harm nor help your visibility there. No major AI search operator documents that its crawlers read llms.txt on other sites, so it does not replace crawler access or readable HTML.

Can I block AI training and still appear in AI search?

Usually, yes. OpenAI, Anthropic, Google and Apple each separate training controls (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) from search crawling, so you can disallow the training token and keep the search crawler allowed.

What are the actual technical requirements for AI search?

The documented ones are short: the operator's search crawler must be allowed in robots.txt and not blocked by your CDN or firewall, the content must be in the HTML it receives, and for Google the page must be indexed and eligible for a snippet. Whether an assistant then cites you depends on factors outside your site.

Is AI search optimization different from SEO?

Mostly in which crawlers you have to let in and whether they render JavaScript. The basics of crawlable, indexable, clearly structured pages carry over, and Google states no extra requirements for its AI features.

Check the facts on your own site

The quickest way past these myths is to look at what crawlers actually receive from your pages. As a next step, run the free single-page check on one important URL to see its crawler access, raw HTML and indexability findings; it needs no account.