The AI search readiness checklist: 20 checks you can run today

This AI search readiness checklist is 20 checks in four groups: access, readable content, structure and discovery. Work through them in order, because each group depends on the one before it: a crawler that is blocked never reads your text, and text it cannot read gains nothing from good headings. Most checks take a minute or two with curl, view-source and your browser's developer tools.

If you want the reasoning behind the four groups first, read what AI search readiness means. Otherwise, pick one important page, replace https://example.com/page in the commands below with its URL, and start.

The checklist at a glance

Run the checks on one page per template (home, a product or service page, an article, a category page), not just the homepage. Templates fail together.

Which crawlers to test

Each AI search tool reaches your site through a named crawler. Testing "bots" in general misses the point: the rule that matters is the one for that name.

GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended control training use (Google-Extended also covers grounding in Gemini Apps and Vertex AI), not inclusion in search results. You can block them and still pass this checklist: see how to allow AI search but block AI training.

Access: can AI crawlers fetch the page?

• 1. The page returns 200 after a short redirect chain.
• 2. robots.txt returns 200, or a deliberate 404, and never a 5xx.
• 3. robots.txt allows each search crawler you care about.
• 4. No CDN or WAF challenge page is served to crawlers.
• 5. No unintended noindex or nosnippet in meta tags or headers.

Check 1 shows every hop and its status:

curl -sIL https://example.com/page | grep -iE '^(HTTP|location|x-robots-tag)'

Pass: the last HTTP line shows 200, with one or two redirects at most (for example http to https). A 4xx, a 5xx, or a redirect loop is a fail. A few servers answer the HEAD request that -I sends differently from a normal GET, so if the result looks odd, repeat it as a GET with curl -sL -o /dev/null -w '%{http_code}\n' https://example.com/page.

Check 2 matters more than it looks, because crawlers react to the robots.txt status before they read a single rule:

curl -s -o /dev/null -w '%{http_code}\n' https://example.com/robots.txt

Under RFC 9309, a 4xx means no restrictions apply, while a 5xx or network error means the crawler must assume everything is disallowed. Google's robots.txt specification says that on a 5xx it stops crawling the site for the first 12 hours, then falls back to the last good copy for up to 30 days. It excludes 429 from its "4xx means no restrictions" rule and counts a 429 as a server error. Pass: 200 (or a 404 you chose on purpose). A 401 or 403 is not a block: it also means no restrictions.

Check 3 means reading the file group by group:

curl -s https://example.com/robots.txt

Find the group that names each crawler in the table above. A crawler that finds a group naming it obeys that group and ignores User-agent: * entirely. In this file, GPTBot is blocked everywhere, every unnamed crawler is kept out of /admin/, and OAI-SearchBot can fetch /admin/ because its own group never mentions it:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Disallow: /admin/

Repeat shared rules inside every named group. Two more traps: Applebot follows your Googlebot rules when robots.txt does not name Applebot, and OpenAI says changes take about 24 hours to apply to search. Within a group, the longest matching rule wins, whatever order the lines are in. The robots.txt groups guide covers the matching rules in detail. Pass: the group each search crawler obeys allows the page.

Search Console's old robots.txt Tester was retired in November 2023, when Google introduced the robots.txt report. The report shows which files Google fetched for your top hosts, when, and any errors, but it cannot test single URLs. For that, use URL Inspection or Google's open-source robots.txt parser.

Check 4 catches the block robots.txt cannot show you. Send a request with a crawler's user agent and look at the status and headers:

UA='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot'
curl -s -D - -o page.html -A "$UA" https://example.com/page | grep -iE '^(HTTP|cf-mitigated|server)'

Fail: a 403, a 503, or a cf-mitigated: challenge header, which Cloudflare adds to every challenge page. A pass here is a good sign but not proof. Bot protection often checks the source IP, so a real crawler from its published ranges can be treated differently from your laptop. OpenAI, for example, recommends allowing requests from its published IP ranges as well as allowing OAI-SearchBot in robots.txt. Confirm in your CDN's security events, and note that Cloudflare's Bot Fight Mode cannot be bypassed with WAF custom rules. The cf-mitigated header marks challenges only, so check for plain 403 blocks too. More in why a WAF challenge page means AI crawlers read nothing.

Check 5 looks in two places:

curl -sI https://example.com/page | grep -i '^x-robots-tag'
curl -s https://example.com/page | grep -ioE '<meta[^>]+name="(robots|googlebot)"[^>]*>'

Pass: nothing printed, or only values you intended. noindex keeps a page out of search, and Google says nosnippet also stops the content being used as a direct input for AI Overviews and AI Mode.

Optional: llms.txt. If you publish one, make sure /llms.txt returns 200 and is valid Markdown under the llmstxt.org proposal. It is not required: Google Search says it ignores llms.txt, and no major AI search operator documents reading it on third-party sites.

Readable content: is your text in the raw HTML?

• 6. A sentence from the main content appears in the raw HTML.
• 7. The raw HTML holds roughly the same text as the rendered page.
• 8. Text behind tabs, accordions and "load more" is in the HTML.
• 9. Important text is not locked in images, canvas or video.
• 10. The page declares its language.

This group exists because many AI crawlers do not run JavaScript. In Vercel's December 2024 analysis, none of the major AI crawlers it measured, including OpenAI's crawlers, ClaudeBot and PerplexityBot, rendered JavaScript at the time. Googlebot does render, but Google's JavaScript SEO guide still calls server-side rendering or pre-rendering a great idea.

Check 6 is the fastest test on this list. Copy a distinctive sentence from the middle of the page and search the raw response for it:

curl -s https://example.com/page | grep -c "a distinctive sentence from your page"

Pick a sentence without quotes, apostrophes or ampersands, because the HTML may encode them as entities. Pass: 1 or more. A 0 usually means the text arrives through JavaScript after load, the empty HTML problem.

Check 7 compares word counts. Raw:

curl -s https://example.com/page | python3 -c "import sys,re,html; t=sys.stdin.read(); t=re.sub(r'(?is)<(script|style)[^>]*>.*?</\1>',' ',t); print(len(html.unescape(re.sub(r'<[^>]+>',' ',t)).split()))"

Rendered, in the DevTools console on the same page:

document.body.innerText.trim().split(/\s+/).length

Pass: the two numbers are close. The raw count can be a little higher, since it includes hidden text and the title. If it is a small fraction of the rendered one, most of the page is built in the browser. The curl guide shows more ways to see what a crawler sees.

Check 8: repeat check 6 with a sentence from a closed tab, a collapsed accordion or the second page of a "load more" list. Content that is in the HTML but hidden with CSS passes; content fetched only on click fails.

Check 9: look for headlines, prices or opening hours that exist only as pixels. Pass: every fact you want quoted is real text, with alt text on images that carry meaning.

Check 10:

curl -s https://example.com/page | grep -ioE '<html[^>]*lang="[^"]+"'

Pass: a value such as lang="en" that matches the content.

Structure: can machines tell what the page is?

• 11. One meaningful title and one H1 that matches it.
• 12. A logical heading order, main and nav landmarks, and descriptive link text.
• 13. A canonical tag that points to the page itself, or deliberately elsewhere.
• 14. JSON-LD that parses without errors.
• 15. Structured data that matches the visible facts.

Check 11:

curl -s https://example.com/page | grep -ioE '<(title|h1)[^>]*>[^<]*'

Pass: one title, one H1, both describing the same thing. A title of "Home" or an H1 that is a logo image fails.

Check 12: in the DevTools Elements panel, confirm headings run H1, H2, H3 without using headings for styling, that a <main> element wraps the content, and that links say where they go ("Pricing for agencies", not "Click here").

Check 13:

curl -s https://example.com/page | grep -ioE '<link[^>]+rel="canonical"[^>]*>'

Pass: one canonical, an absolute URL, returning 200, normally the URL you fetched. A canonical pointing at the homepage from every page is a common template bug.

Check 14 counts the JSON-LD blocks and prints the types in each, or stops with an error at the first block that does not parse:

curl -s https://example.com/page | python3 -c "import sys,re,json; blocks=re.findall(r'(?is)<script[^>]+application/ld\+json[^>]*>(.*?)</script>', sys.stdin.read()); print(len(blocks), 'JSON-LD blocks'); [print([i.get('@type') for i in (d if isinstance(d, list) else d.get('@graph', [d]))]) for d in map(json.loads, blocks)]"

Pass: types such as Organization, Product or Article, and no traceback. Then run the page through the Schema Markup Validator. Keep expectations honest: Google says no special schema is needed for AI Overviews, and FAQ rich results stopped appearing in Google Search in May 2026. Structured data is still useful: it states names, prices and dates in a form a parser does not have to guess at.

Check 15: compare the name, price, address and dates in the JSON-LD with what the page shows. Pass: they agree. Markup that contradicts the page is worse than none.

Discovery: can crawlers find every page?

• 16. An XML sitemap exists and robots.txt points to it.
• 17. The sitemap lists only canonical, indexable URLs that return 200.
• 18. No broken internal links.
• 19. Duplicate URLs are consolidated.
• 20. Key pages are indexed in Google and Bing.

Checks 16 and 17:

curl -s https://example.com/robots.txt | grep -i '^sitemap:'
curl -s https://example.com/sitemap.xml | grep -o '<loc>' | wc -l

The second command counts <loc> entries even when the whole sitemap sits on one line. If the file is a sitemap index, the count is the number of child sitemaps, so repeat it on each. Pass: a Sitemap: line, a sitemap that returns 200, and a URL count close to the number of pages you want found. The sitemaps protocol caps each file at 50,000 URLs and 50 MB uncompressed, and every URL must use the same protocol and host as the sitemap. Spot-check a few entries with check 1: redirects, 404s and noindex pages do not belong in a sitemap.

Check 18: run a link checker or crawler over the site. Pass: zero internal links to 404 pages, and no internal links that redirect when they could point straight at the final URL.

Check 19: fetch the page with a trailing slash, without one, with www and with a tracking parameter. Pass: each variant redirects to one URL or declares the same canonical.

Check 20: use URL Inspection in Google Search Console and Bing Webmaster Tools. Google says a page must be indexed and eligible to show a snippet to appear as a supporting link in AI Overviews or AI Mode, and Microsoft says Copilot grounds web answers through the Bing search service, so these two indexes matter for several AI surfaces at once.

Frequently asked questions

How long does the AI search readiness checklist take?

Most checks take a minute or two once the commands are ready, so one page is a short session rather than a project. Repeat it for each page template, since pages built from the same template usually pass or fail together.

Do I need an llms.txt file to pass?

No. It is optional. Google Search says it ignores llms.txt, and OpenAI, Anthropic and Perplexity do not document their crawlers reading it on other sites.

Does blocking GPTBot or ClaudeBot fail the checklist?

No. GPTBot and ClaudeBot are training crawlers. OpenAI says its GPTBot and OAI-SearchBot settings are independent, so a site can block training and stay eligible for ChatGPT search; Anthropic likewise runs Claude-SearchBot and Claude-User separately from ClaudeBot.

If I pass all 20 checks, will AI assistants cite my site?

Not necessarily. The checklist removes technical barriers to reaching and reading your pages. Whether an assistant cites a page depends on the question, the competition and many other things outside your site.

How often should I re-run it?

After any change to robots.txt, CDN or firewall settings, rendering or templates, and at least once a quarter. robots.txt changes are not instant: OpenAI says to allow about 24 hours, and Perplexity says up to 24 hours.

Run the checklist across the whole site

Twenty checks on one page is manageable by hand. Across hundreds of pages it is not. When you need every page covered, a Crawlito full audit runs deterministic checks on up to 500 pages, adds a human review, and delivers a PDF with the evidence and fixes in priority order.