AI search readiness audit: what it covers and what to expect

An AI search readiness audit checks whether AI search tools such as ChatGPT search, Perplexity, Claude and Google AI Overviews can reach your pages, read the text on them and understand how they fit together. A good one runs repeatable, rule-based checks across your whole site, shows the evidence for each problem and tells you exactly how to fix and re-test it. It does not predict whether an assistant will cite you, because nobody outside those companies can measure that.

This post explains what to expect from an audit, what the report should contain, and the warning signs that an audit is selling guesswork. If you are new to the topic, start with what AI search readiness means.

What does an AI search readiness audit measure?

Readiness is about technical barriers: things on your side that stop a crawler from fetching a page or from finding the text in it. A thorough audit groups its checks into four areas.

Access matters most because nothing else counts if the request fails. A page that returns a 403 or a challenge page gives the crawler nothing to read, and a noindex asks search engines to leave it out of results, however good its content is.

Each check should be deterministic: given the same site, it gives the same result every time. "The raw HTML for this URL contains 41 words of main text" is a deterministic finding. "This page feels thin to an LLM" is not.

Why an audit has to compare raw and rendered HTML

Many AI crawlers read only the HTML your server sends and do not run JavaScript. In its December 2024 analysis with MERJ, Vercel reported that none of the major AI crawlers it measured rendered JavaScript at the time, naming OpenAI's crawlers, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. Googlebot, by contrast, renders pages with a headless Chromium, as Google's JavaScript SEO guide describes.

So an audit needs two versions of every page:

1. The raw response, fetched the way a non-rendering crawler fetches it.
2. The rendered DOM, after a real browser has run the scripts.

The gap between the two is the finding. If a product page shows 900 words in the browser and 30 in the raw HTML, a crawler that does not render sees a near-empty page. You can spot-check this yourself:

This strips scripts, styles and tags from the raw HTML and counts the words left, as a non-rendering crawler would receive them:

curl -sL https://example.com/products/blue-kettle \
  | perl -0777 -pe 's/<script\b.*?<\/script>//gis; s/<style\b.*?<\/style>//gis; s/<[^>]+>/ /g' \
  | wc -w

The count includes navigation and footer text, so treat it as a rough figure and compare it with what you see in the browser. A single page tells you little; the audit's job is to run that comparison on every page and group the failures by template. Our guide to raw vs rendered HTML covers the manual method in more detail.

Why robots.txt has to be evaluated per crawler

Checking that robots.txt "exists and looks fine" is not enough. Under RFC 9309, a crawler obeys the group that names it and falls back to the * group only when no group matches. Some operators add their own fallback rules on top, as Apple does. Take this file:

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /

User-agent: Googlebot
Disallow: /search

Here is what each crawler actually does with it:

Two results surprise most site owners. Googlebot can crawl /admin/, because its own group replaces the * group rather than adding to it (see how robots.txt groups work). And Applebot follows the Googlebot rules, because Apple documents that Applebot follows Googlebot's instructions when robots.txt mentions Googlebot but not Applebot. Apple's page does not say how that fallback interacts with a * group, so if Applebot matters to you, give it a group of its own and remove the doubt.

A good audit also labels each crawler by purpose. Blocking GPTBot is a training opt-out: OpenAI says its GPTBot and OAI-SearchBot settings are independent, so a site can block GPTBot and still be eligible for ChatGPT search. An audit that marks a deliberate training opt-out as a search problem is wrong.

User-triggered agents need their own note. OpenAI says that for ChatGPT-User "robots.txt rules may not apply", and Perplexity states that Perplexity-User generally ignores robots.txt. A report that lists Disallow rules for those agents as an effective block is giving you false comfort.

Access problems that robots.txt does not show

A clean robots.txt can sit in front of a firewall that blocks the same crawlers. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges, so a CDN rule that challenges those IPs can keep you out of results even when robots.txt is correct. On Cloudflare, every challenge page response carries a cf-mitigated: challenge header, which makes challenges easy to detect. The header marks challenges only; an outright block shows up as an error status such as 403 without it.

curl -s -o /dev/null -D - \
  -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  https://example.com/ | grep -iE "^HTTP|cf-mitigated"

A spoofed user agent from your laptop only tests rules keyed on the user agent. Rules keyed on IP addresses or bot scores can treat the real crawler differently, so the audit should say which kind of test produced each access finding. More on this in why a WAF challenge page means AI crawlers read nothing.

How much of the site should an audit cover?

Problems in AI readiness usually live in templates: every product page, every blog post, every location page. Testing the home page and a few landing pages misses them. Ask any provider:

• How many pages are crawled, and how they are chosen (sitemap, link discovery or both).
• Whether every crawled page is fetched both raw and rendered.
• What happens when the crawl is blocked partway through.

The last question matters. If the audit's own crawler hits a challenge page on 60% of URLs, the honest result is "incomplete" with the reason, not a score calculated from the 40% it could read. A score built on partial coverage looks precise and means little.

What should an AI search readiness audit report contain?

Each finding should be something you can act on without calling the auditor. Look for five parts:

1. What was found, in one plain sentence.
2. Evidence: the response status, the robots.txt line, the header, the word counts, the exact HTML snippet.
3. Affected URLs: how many pages are affected, with example URLs, so a problem that one template causes is reported once rather than hundreds of times.
4. A prioritized fix: what to change, where, and why it comes before other fixes.
5. A verification step: the command or tool you can run to confirm the fix worked.

An illustrative finding in that format:

Finding:   Main text missing from the raw HTML
Severity:  High (Readable content)
Affected:  212 URLs, for example /products/blue-kettle
Evidence:  /products/blue-kettle
           Raw HTML: 31 words of main text
           Rendered: 948 words of main text (raw/rendered ratio 3%)
           Product description loads from /api/products after hydration
Fix:       Render the description and specifications on the server,
           or include them in the initial HTML response
Verify:    curl -s https://example.com/products/blue-kettle | grep -c "Blue kettle"

Prioritization should follow impact on reach, not the number of findings. A site-wide challenge page outranks fifty missing alt attributes. If the report sorts findings alphabetically or by category, you will spend your first week on the wrong things.

Red flags in an AI search audit

Some audits sell certainty that does not exist. Be wary of these:

• A score that claims to measure your chance of being cited. Whether ChatGPT, Perplexity or Google's AI features cite a page depends on the query, the competition and each operator's ranking systems. A readiness score can measure published checks; it cannot measure citation probability.
• Findings generated by an AI model. A language model asked "is this page AI-ready?" can give different answers on different runs, and can describe problems that are not there. Findings should come from rules you can re-run. A model can help explain a verified finding in plain language; it should not decide what the findings are.
• "AI schema" or special markup for AI Overviews. Google's page on AI features and your website says no special schema.org structured data, AI text files or new machine-readable files are needed to appear in AI Overviews or AI Mode. Structured data is still useful when it is valid and matches the page; see what helps and what is hype.
• llms.txt presented as required. It is a proposal from llmstxt.org, and at the time of writing none of OpenAI, Anthropic, Perplexity or Google documents that its crawlers read llms.txt on third-party sites. Google's AI optimization guide says Google Search ignores these files. An audit can report whether the file is present and parses, but it should not treat a missing file as a barrier.
• Penalties for training opt-outs. Blocking GPTBot, ClaudeBot, CCBot, Google-Extended or Applebot-Extended is a decision about training, not search access. Google-Extended also covers grounding in Gemini Apps and Vertex AI, but Google says it does not affect inclusion in Google Search.
• Guarantees of traffic, rankings or citations. No one can promise them.

How Crawlito approaches the audit

Crawlito's full-site audit crawls up to 500 pages twice, once as a browser and once as a non-rendering crawler, and compares them page by page. It runs deterministic checks across access, readable content, structure and discovery. The score is computed by rules, weighted towards access, and how the score works is published. When the crawl cannot fetch enough of the site, the report shows Incomplete or Access blocked instead of a score, with what blocked it. On a site with more than 500 pages, the report says how many of the discovered pages were checked and never calls the crawl complete.

Claude writes the plain-language explanations and fix steps from findings the rules have already verified; it cannot add, remove or change a finding. A person reviews every report before delivery. You get a private report page and a PDF with evidence, prioritized fixes and a developer appendix grouped by where each fix is made. The report link is valid for 30 days, and we issue a new one on request. One recheck is included: you request it from the report page within 14 days, and it is compared with the first report. Audits start from $300 per site. Opting out of AI training never lowers the search-readiness score.

Frequently asked questions

How long does an AI search readiness audit take?

It depends on the number of pages and whether a person reviews the report; the review and write-up usually take longer than the crawl. Ask for a delivery date before you pay. For our audits, the crawl starts once the invoice is paid and the delivery date is confirmed with the invoice.

Do I need to give the auditor access to my server or CMS?

No. A readiness audit should read what any public visitor or crawler can read. Server or CDN logs are only needed if you also want a report of which AI crawlers actually visited.

Will fixing the findings get my site cited in ChatGPT or Perplexity?

No one can promise that. Fixing the findings removes technical barriers that stop AI tools from reaching and reading your pages; whether an assistant cites you depends on many things outside your site.

Is an audit different from an SEO audit?

It overlaps on status codes, canonicals and sitemaps, but it adds checks most SEO audits skip: robots.txt evaluated for each AI crawler by purpose, raw versus rendered text for crawlers that do not run JavaScript, and challenge pages served to AI user agents.

What happens after the recheck?

The recheck report lists which findings are resolved, which are new and which are still open, with the change in score. If you want ongoing monitoring, a monthly retainer adds a monthly recheck and a verified AI-crawler visit report from your server or Cloudflare logs.

Where to start

If you want to see the kind of evidence an audit produces before committing, run the free single-page check on one of your key templates. If it shows problems, or you need the whole site covered, request a full-site audit.