63 guides

Can AI search read your site?

Practical guides to the technical side of AI search: which crawlers visit, what they are allowed to read, what they actually receive, and how to fix what stops them. No promises of rankings, just what is documented and how to check it.

Start here

AI crawlers
AI crawler user agents list: the major bots and what each does
A reference list of AI crawler user agents from OpenAI, Anthropic, Perplexity, Google, Apple and others, with each bot's purpose, robots.txt token and docs.
8 min read
CDN and bot protection
Cloudflare and AI crawlers: the settings that block them
Which Cloudflare settings block AI bots: AI crawler policies, AI Crawl Control, Bot Fight Mode, managed robots.txt and WAF rules, and how to allow the ones you want.
10 min read
Rendering and content
Do AI crawlers run JavaScript? What is known, and how to test
Do AI crawlers render JavaScript? Googlebot does. Most AI crawlers did not in Vercel's 2024 study. What that means for your pages and how to test them.
9 min read
Structure and schema
llms.txt: what it is, who reads it, and how to write one
llms.txt is a Markdown summary of your site for language models. What the proposal says, who actually reads it today, and how to write and test one.
9 min read
Access and robots.txt
robots.txt for AI crawlers: a complete guide
How robots.txt works for AI crawlers: the RFC 9309 matching rules, how GPTBot, ClaudeBot and others read the file, and three templates you can copy.
9 min read
Structure and schema
Structured data for AI search: what helps and what is hype
What structured data does for AI search, what Google and AI operators actually document, which JSON-LD types matter, and how to test that your markup is right.
8 min read

Strategy and audits

All 10 in strategy and audits

What AI search readiness is, how to check it, and how to explain it to clients.

The AI search readiness checklist: 20 checks you can run today
An ordered AI search readiness checklist: 20 checks across access, readable content, structure and discovery, each with a quick test and a clear pass.
9 min read
What is AI search readiness? The four layers explained
AI search readiness is whether ChatGPT, Perplexity, Claude, Google AI Overviews and Copilot can reach, read and interpret your pages. Here is what it covers.
9 min read
Why ChatGPT can't see your website: a troubleshooting guide
ChatGPT can't see your website? Work through the six usual causes, from robots.txt and Cloudflare rules to JavaScript and noindex, with a check and fix for each.
10 min read
AI search audit for agencies: how to scope, deliver and price it
A practical AI search audit guide for agencies: what to promise clients, how to scope the work, how it fits a technical SEO retainer, and what to deliver.
10 min read
AI search optimization myths: eight that waste your time
Eight AI search optimization myths, from llms.txt to GPTBot blocks, checked against OpenAI, Google and RFC 9309 documentation, with what to do instead.
8 min read
AI search readiness audit: what it covers and what to expect
What a good AI search readiness audit measures, what the report should contain, and the red flags that mean an audit is selling guesswork instead of evidence.
9 min read
Generative engine optimization vs SEO: what is actually different
Generative engine optimization vs SEO: what GEO, AEO and LLMO share with technical SEO, what is new for AI crawlers, and what is still unproven.
9 min read
Google AI Overviews requirements: what your site actually needs
Google says AI Overviews and AI Mode need nothing beyond normal Search eligibility. Here is what that means, which controls apply and how to check each.
9 min read
How the Crawlito AI readiness score works
The Crawlito AI readiness score comes from fixed rules: four weighted categories, pass or fail checks and a coverage figure. Here is how it is calculated.
8 min read
How to measure AI referral traffic and AI crawler visits
Measure AI referral traffic and AI crawler visits as two separate numbers: verified bot hits from your logs, and human clicks from referrers and utm_source tags.
8 min read

AI crawlers

All 11 in ai crawlers

Who GPTBot, ClaudeBot, PerplexityBot and the other AI crawlers are, and what each one is for.

AI crawler user agents list: the major bots and what each does
A reference list of AI crawler user agents from OpenAI, Anthropic, Perplexity, Google, Apple and others, with each bot's purpose, robots.txt token and docs.
8 min read
AI crawlers in server logs: how to find and count their visits
Find AI crawlers in server logs with grep and awk: count hits per bot, spot 403 and 429 responses, verify IPs, and read Cloudflare's AI crawler data.
9 min read
AI training crawlers vs search crawlers: why the difference matters
AI training crawlers vs search crawlers vs user-triggered fetchers: what each does, what blocking costs, and how to opt out of training but stay in AI search.
9 min read
Applebot-Extended and Applebot: Apple's crawlers and AI
Applebot-Extended does not crawl: it is a robots.txt token that opts content out of Apple's model training. How it differs from Applebot, with examples.
8 min read
Bingbot and AI search: why Bing's crawler still matters
Microsoft Copilot grounds web answers in Bing search. How to check that Bingbot can reach your site, verify real Bingbot traffic and use IndexNow.
9 min read
CCBot and Common Crawl: the open dataset behind many models
What CCBot is, why Common Crawl's open web archive feeds so many AI datasets, how CCBot follows robots.txt, and what blocking it does and does not change.
9 min read
ClaudeBot, Claude-SearchBot and Claude-User explained
ClaudeBot collects training data, Claude-SearchBot indexes for search and Claude-User fetches on request. How to opt out of training and stay visible in Claude.
8 min read
Google-Extended: what it controls and what it doesn't
Google-Extended is a robots.txt token, not a crawler. It covers Gemini training and grounding, not Google Search rankings or AI Overviews. Here is how to use it.
8 min read
GPTBot vs OAI-SearchBot vs ChatGPT-User: OpenAI's crawlers explained
GPTBot vs OAI-SearchBot vs ChatGPT-User: what each OpenAI agent does, how robots.txt applies to each, and how to verify them by IP.
7 min read
How to verify GPTBot IPs and spot fake AI crawlers
User agents are easy to fake. Verify GPTBot IPs and other AI crawlers with published IP lists, reverse DNS or Cloudflare verified bots, then deal with fakes.
9 min read
PerplexityBot and Perplexity-User: how Perplexity reaches your site
PerplexityBot indexes pages for Perplexity search; Perplexity-User fetches pages on request. How to allow, block and verify each, and what blocking changes.
8 min read

Access and robots.txt

All 10 in access and robots.txt

robots.txt rules, robots meta tags, status codes and the settings that decide who may read a page.

robots.txt for AI crawlers: a complete guide
How robots.txt works for AI crawlers: the RFC 9309 matching rules, how GPTBot, ClaudeBot and others read the file, and three templates you can copy.
9 min read
Crawl-delay for AI bots: who honors it and what to use instead
Crawl-delay is not in the robots.txt standard and AI crawler support varies. See which bots honor it and how to slow the rest with 429 and Retry-After.
9 min read
How to allow AI search but block AI training in robots.txt
A copy-paste robots.txt that blocks AI training (GPTBot, ClaudeBot, Google-Extended, CCBot) and allows AI search crawlers, with every line explained.
10 min read
How to test robots.txt against each AI crawler
Test robots.txt the way AI crawlers read it: fetch it raw with curl, check every host, run Google's parser and trace the group each AI token follows.
9 min read
noindex, nosnippet and X-Robots-Tag: what they mean for AI search
How nosnippet, max-snippet, data-nosnippet and noindex control AI Overviews and AI Mode, what other AI tools document, and how to set X-Robots-Tag for PDFs.
8 min read
robots.txt user-agent groups: why a named group overrides *
How robots.txt user-agent groups work under RFC 9309: a crawler obeys one group, rules under * are not inherited, and the longest matching rule wins.
8 min read
robots.txt vs meta robots: blocking crawling vs blocking use
robots.txt stops crawling, noindex stops listing. Why a Disallow can hide your noindex, when to use each, and how the same logic applies to AI crawlers.
9 min read
Seven robots.txt mistakes that block AI crawlers by accident
Seven robots.txt mistakes that quietly block AI crawlers, from a leftover staging Disallow to a 5xx error, each with a before-and-after fix you can test.
9 min read
Site noindex by mistake: how staging settings hide live sites
Five staging leftovers that keep a live site out of Google and AI search, how to spot each one in two minutes with curl, and a launch checklist to prevent them.
9 min read
Status codes and redirects: what AI crawlers do with them
Redirect chains, soft 404s and wrong status codes can stop AI crawlers reaching a page. How 3xx, 4xx and 5xx are read, robots.txt errors, and curl checks.
10 min read

CDN and bot protection

All 6 in cdn and bot protection

How Cloudflare, firewalls, rate limits and challenge pages affect AI crawlers.

Cloudflare and AI crawlers: the settings that block them
Which Cloudflare settings block AI bots: AI crawler policies, AI Crawl Control, Bot Fight Mode, managed robots.txt and WAF rules, and how to allow the ones you want.
10 min read
Cookie walls, age gates and pop-ups: what crawlers actually see
Cookie walls, age gates and pop-ups can replace your content in the HTML crawlers read. When interstitials are harmless, when they hide pages, and how to check.
9 min read
Geo blocking crawlers: how country and IP rules shut out AI
Geo blocking crawlers is easy to do by accident: most AI crawlers connect from US cloud data centers. How to find country and ASN blocks and exempt verified bots.
9 min read
Paywalls and AI crawlers: what gated content they can read
How hard, metered and lead-in paywalls and login walls decide what AI crawlers read, what isAccessibleForFree markup does, and how to choose what to expose.
9 min read
Rate limit AI crawlers without locking out AI search
How to rate limit AI crawlers safely: throttle with 429 and Retry-After, give each bot its own budget, cache to cut origin load, and watch for blocked search bots.
9 min read
WAF blocking crawlers: why a challenge page shows AI bots nothing
A WAF challenge page stops AI crawlers that don't run JavaScript. See what they receive, how to detect it with curl and how to allow verified bots safely.
10 min read

Rendering and content

All 8 in rendering and content

Why content that appears only after JavaScript runs is invisible to many AI crawlers, and how to fix it.

Do AI crawlers run JavaScript? What is known, and how to test
Do AI crawlers render JavaScript? Googlebot does. Most AI crawlers did not in Vercel's 2024 study. What that means for your pages and how to test them.
9 min read
Client-side rendering, SEO and AI: the empty HTML problem
Client-side rendering can hand AI crawlers an empty HTML shell. See what they receive, the symptoms, the fixes ranked by effort and how to verify each one.
9 min read
Lazy loaded content and crawlers: tabs, accordions and scroll
Is lazy loaded content readable by crawlers? Tabs and accordions in the HTML are fine; content fetched on click or scroll is not. How to test each pattern.
8 min read
Prerendering for crawlers: when dynamic rendering still helps
Prerendering for crawlers serves bots an HTML snapshot of a JavaScript app. Google calls it a workaround. When it helps, the risks, and how to test AI crawlers.
8 min read
Raw HTML vs rendered HTML: seeing what a crawler sees
Raw HTML vs rendered HTML: what each one is, how to compare them with view-source, curl, DevTools and Search Console, and when the gap hides content from crawlers.
9 min read
See your page as an AI crawler: curl with the GPTBot user agent
Use curl with the GPTBot user agent and other AI crawler strings to check status, redirects, challenge pages and visible text, then compare with a browser.
8 min read
SSR vs CSR for SEO and AI search: SSG, ISR and islands compared
SSR vs CSR for SEO and AI crawlers: what HTML each rendering strategy sends, how SSG, ISR, islands and streaming compare, and which one fits your type of site.
9 min read
Text in images SEO: what crawlers miss in canvas and video
Text in images, canvas and video is not in the HTML crawlers read. How to find words locked in pixels on your pages, and the alt text and transcript fixes.
8 min read

Structure and schema

All 11 in structure and schema

Headings, canonicals, sitemaps, structured data and llms.txt: helping machines interpret a page.

llms.txt: what it is, who reads it, and how to write one
llms.txt is a Markdown summary of your site for language models. What the proposal says, who actually reads it today, and how to write and test one.
9 min read
Structured data for AI search: what helps and what is hype
What structured data does for AI search, what Google and AI operators actually document, which JSON-LD types matter, and how to test that your markup is right.
8 min read
Article schema done right: Article, BlogPosting and NewsArticle
How to write Article schema in JSON-LD: headline, author as a Person with a url, honest datePublished and dateModified, image, publisher and the common mistakes.
9 min read
Broken internal links: helping crawlers find every page
Broken internal links, orphan pages and JavaScript-only menus hide pages from crawlers. How link discovery works, how to find broken links and how to fix them.
9 min read
Canonical tags and duplicate URLs: how they confuse crawlers
What a canonical tag tells crawlers, the conflicts with redirects, sitemaps and hreflang that get it ignored, and how to audit canonicals with curl in minutes.
10 min read
Common JSON-LD errors and how to fix them
The JSON-LD errors that make structured data unreadable or untrustworthy, from trailing commas to conflicting plugin blocks, with broken and fixed snippets.
9 min read
FAQ schema in 2026: what FAQPage markup still does
Google no longer shows FAQ rich results. What FAQ schema still does, when FAQPage markup fits, the rules to follow and a JSON-LD example you can copy.
9 min read
Heading structure SEO: headings, landmarks and link text
Heading structure SEO for people and machines: match the title and H1, keep heading levels logical, add main and nav landmarks, and write descriptive link text.
9 min read
Organization schema: telling machines who you are
A complete Organization schema example in JSON-LD, which LocalBusiness subtype to use, where to put the markup, and the errors that make it contradict itself.
9 min read
Product schema for shopping answers and rich results
How to write Product schema that Google accepts: Offer, price, availability, GTIN and brand, honest reviews, variants, and a price that matches the page.
8 min read
XML sitemaps for AI discovery: what to list and how to submit
An XML sitemap tells crawlers which URLs exist and when they changed. What to list, accurate lastmod, size limits, robots.txt, Search Console, Bing and IndexNow.
9 min read

Platform guides

All 7 in platform guides

AI search readiness on WordPress, Shopify, Wix, Squarespace, Webflow, Next.js and single-page apps.

Next.js SEO for AI crawlers: rendering, metadata and robots
Next.js SEO for AI crawlers: what the App Router sends by default, where useEffect, Suspense and streamed metadata go wrong, and how to set robots.ts and JSON-LD.
9 min read
React SPA SEO: fixing the empty shell for AI crawlers
React SPA SEO for AI search: why Vite and Create React App sites send crawlers an empty root div, four fixes ranked by effort, and how to verify each one.
8 min read
Shopify and AI crawlers: robots.txt, schema and theme checks
How AI crawlers see a Shopify store: the default robots.txt, robots.txt.liquid edits, theme JSON-LD, app-injected content, duplicate URLs and bot protection.
9 min read
Squarespace AI crawlers: the settings to check
Squarespace's AI crawler checkbox asks 26 bots not to crawl, but not ChatGPT search or Perplexity. See what it changes in robots.txt and which settings hide pages.
9 min read
Webflow robots.txt and AI crawlers: the settings that matter
Where Webflow robots.txt rules live, what the Traffic control toggles block, how to keep webflow.io out of search, and how to add JSON-LD to CMS pages.
9 min read
Wix AI crawlers: what AI search sees on a Wix site
What AI crawlers see on a Wix site: server-rendered pages, the robots.txt editor, structured data settings, and the content that still goes missing.
10 min read
WordPress AI crawlers: the settings that decide what they see
The WordPress settings that decide whether AI crawlers can read your site: search visibility, robots.txt, SEO plugins, security plugins, builders and caching.
9 min read