Successful HTTP responses
4 of 5 URLs
URLs returning HTTP 200–299 among those with a recorded server response.
Examples outside this count
- https://harborbakery.example/old-menu
Sample AI search readiness report
Scanned 11 November 2026. Illustrative website measured from saved test pages.
We checked how AI search crawlers can reach and read harborbakery.example. The site scored 75 out of 100 on the published checks, with 100% coverage. The checks recorded 8 findings (1 high, 4 medium, 2 low, 1 informational). Start by reviewing the robots.txt rule that blocks Claude-SearchBot on /gift-cards. Each finding below lists why it matters, the exact change to make and how to confirm it.
Every URL found during the crawl was checked.
Readiness score
75/ 100
Check coverage 100%
Share of applicable rule weight evaluated.
The score measures published checks, not the probability of being cited.
Start with the most severe findings. When severity is equal, the finding with the widest observed reach comes first.
Open /robots.txt and find the group (User-agent line) that matches the listed bot, or the "*" group if it has none (Applebot uses the Googlebot group before the "*" group).
Check whether the page is meant to have content; if it is a placeholder, add the real copy or remove it from navigation and the sitemap.
For each broken URL in the evidence, decide whether the page should exist.
7 scored checks failed on the checked pages. The list gives every one in a suggested order, starting from the current score of 75. Score after this step is the overall score from the published scorer once that step and every step above it pass, with all other evidence unchanged.
Server configuration · Quick change · Site level
+9 pointsScore after this step 84
Page content · Larger project · 1 page affected
+6 pointsScore after this step 90
CMS and page templates · Template or config change · 1 page affected
+3 pointsScore after this step 93
CMS and page templates · Template or config change · 1 page affected
+2 pointsScore after this step 94
CMS and page templates · Template or config change · 1 page affected
+2 pointsScore after this step 96
Page content · Larger project · Site level
+3 pointsScore after this step 98
Server configuration · Larger project · 2 pages affected
+2 pointsScore after this step 100
The order is points per unit of effort. The effort label is a rough size, not a time estimate. Points are the score change if that check alone passed; each score is rounded once, so the points of single steps need not add up to the last figure. Site level marks a check that is not counted per page: a site-wide file or setting, or a check whose affected URLs are link targets. The plan describes the checked pages only and adds nothing to the score.
Each measure uses the pages, responses or images we could examine. Open the examples to see which pages need a closer look.
4 of 5 URLs
URLs returning HTTP 200–299 among those with a recorded server response.
3 of 3 pages
Pages whose server text is at least 50% of the browser text length. Includes paired views with at least 250 rendered characters. This compares length only.
1 of 3 pages
Distinct HTML destinations beyond the start page linked by another checked HTML page. Other pages on the site may also link to them.
1 of 4 pages
Successful HTML pages with a nonempty meta description. Search engines may also build snippets from the page text.
3 of 4 pages
Successful HTML pages with at least one parseable JSON-LD block containing a named type. Use the structured data checks for field errors.
1 of 2 images
Image elements with an alt attribute in the examined HTML. An empty attribute counts because it can correctly identify a decorative image. Review meaning in context.
Up to five observed URLs with multiple failed checks, ordered by the number of distinct checks.
We checked the published policy against 4 distinct paths on the start origin. Server logs can confirm visits. User-requested fetch behavior varies by provider. Each agent is assessed with the robots.txt group that names it, otherwise with the * group. Applebot is the documented exception: without a group of its own it follows the Googlebot group, then *. The matched line is the first Disallow rule in robots.txt that blocked one of the assessed paths for that agent.
https://harborbakery.example/robots.txt · HTTP 200
| Agent or control | Start page | Paths allowed | Matched robots.txt line |
|---|---|---|---|
| OAI-SearchBot | Allowed | 4 of 4 | — |
| Claude-SearchBot | Allowed | 3 of 4 | Disallow: /gift-cardsBlocked examples: /gift-cards |
| PerplexityBot | Allowed | 4 of 4 | — |
| Googlebot | Allowed | 4 of 4 | — |
| Bingbot | Allowed | 4 of 4 | — |
| Applebot | Allowed | 4 of 4 | — |
| Agent or control | Start page | Paths allowed | Matched robots.txt line |
|---|---|---|---|
| ChatGPT-User | Allowed | 4 of 4 | — |
| Claude-User | Allowed | 4 of 4 | — |
| Perplexity-User | Allowed | 4 of 4 | — |
| Agent or control | Start page | Paths allowed | Matched robots.txt line |
|---|---|---|---|
| GPTBot | Blocked | 0 of 4 | Disallow: /Blocked examples: /, /cakes, /cakes-copy |
| ClaudeBot | Allowed | 4 of 4 | — |
| Google-Extended | Allowed | 4 of 4 | — |
| Applebot-Extended | Allowed | 4 of 4 | — |
| CCBot | Allowed | 4 of 4 | — |
| Meta-ExternalAgent | Allowed | 4 of 4 | — |
| Amazonbot | Allowed | 4 of 4 | — |
| Bytespider | Allowed | 4 of 4 | — |
We compared robots.txt, its Content-Signal lines, the sitemap, llms.txt and the canonical links and robots directives found on the 5 URLs checked in this audit with each other. A URL is reported as disallowed only when robots.txt disallows it for every search crawler we evaluate.
No disagreement between these instructions was observed in the checked pages.
robots.txt declares no Content-Signal line.
robots.txt declares no Crawl-delay line.
These are comparisons of what the site publishes, stated as observations. They do not show how a crawler acts on it, and each list is a sample of the URLs examined. This section is left out when no page could be read. This measurement does not change the score.
We compared the main content area of the browser view with the server HTML on 3 pages whose browser view has at least 50 words of main content. Consent banners, dialogs and chat widgets are left out of both views.
100%
Median share of the distinct main-content words in the browser view that are already present in the server HTML.
On every compared page, all main-content words of the browser view and the compared page elements were already in the server HTML.
The share compares vocabulary, not layout or meaning, and describes the compared pages only. Where a word begins and ends depends on the script: text in scripts written without spaces, such as Chinese, Japanese or Thai, is split by dictionary, so its counts are approximate. It says nothing about crawlers that run JavaScript. This measurement does not change the score.
We measured the server HTML of 4 pages: its size, the share of it that is visible text, and how far into the source the main landmark (a main element or role="main") begins.
1.5 kB
Largest page: 1.7 kB.
53%
Share of the HTML bytes that is visible text.
about 206
Estimate for pages written mainly in Latin script: visible text characters divided by 4.
0 of 4
Google documents that Googlebot reads the first 2 MB of an HTML file. Other crawlers publish no limit.
Pages with the lowest text share first.
| URL | HTML size | Text share | Main content starts |
|---|---|---|---|
| https://harborbakery.example/gift-cards | 444 bytes | 24% | 67% into the source |
| https://harborbakery.example/ | 1.7 kB | 49% | 42% into the source |
| https://harborbakery.example/cakes | 1.5 kB | 56% | 39% into the source |
| https://harborbakery.example/cakes-copy | 1.5 kB | 56% | 39% into the source |
Sizes are of the decoded server HTML alone, without scripts, styles or images loaded separately. Only pages whose HTML was read are counted; responses too large to read are reported with the server answers. The token figure is an estimate, not a tokenizer count. This measurement does not change the score.
We read the machine-readable dates of 4 checked pages: 0 declare a modification or publication date, and 0 answered with an ETag or Last-Modified header. 4 sitemap entries were examined; 0 of them declare a lastmod date.
No checked page declared modification dates more than 30 days apart.
The compared dates are the sitemap lastmod, the JSON-LD dateModified of the page’s main entity and the meta modified time, and only on pages that declare at least two of them. The HTTP Last-Modified header is listed but not compared, because on many hosts it is the time of the last deploy and not of the last content change.
Dates are compared with other dates of the same page, never with the scan date, so this does not say whether the content is current. This measurement does not change the score.
We compared JSON-LD values with the visible text, title, description or image alt text of the same page on 3 pages with a parseable JSON-LD block. 3 values were checked.
Values not found as written in the visible text, title, description or image alt text of their page. Each distinct value is listed once, under the first page it was seen on; up to 8 are listed.
| URL | Type | Field | Value in JSON-LD |
|---|---|---|---|
| https://harborbakery.example/cakes | Article | headline | Our story |
The Organization markup on the checked pages declares one Organization name. Names: Harbor Bakery. URLs declared: https://harborbakery.example/. Only Organization and Corporation nodes at the top level of a block, in its @graph or given as a publisher are read; the URLs are listed and not compared.
A value can be absent for a sound reason, such as wording that differs or text shown only inside an image. Numbers are compared by their digits, so a different currency format or a rounded rating still counts as found. This compares text and does not judge the markup. This measurement does not change the score.
robots.txt policy
OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch content for their search services. Googlebot crawls for Google Search, including AI Overviews and AI Mode; Bingbot crawls for Bing and Copilot answers; Applebot crawls for Siri and Spotlight suggestions. A matching Disallow rule restricts access to the affected paths.
Test each affected path against the published robots.txt using the named search crawler. Confirm every path intended for search is allowed, then rerun the audit.
robots.txt policy
A training or content-use control (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot or Bytespider) is restricted on the sampled paths. This records a publishing preference. Search access is checked separately for the search crawlers.
Test the affected paths for each named training control and each search crawler. Confirm both results match your intended policy.
1 affected page
A page whose only text is the menu, header and footer gives a crawler nothing to describe or quote, and can dilute how the rest of the site is understood.
Open the page with JavaScript disabled and confirm it shows its own heading and at least a paragraph of copy besides the menu and footer.
1 affected page
Search engines use language alternates to select a localized version of a page. A missing or conflicting destination makes the intended version ambiguous.
Inspect each flagged language in the page source and open its absolute destination to confirm the intended localized page loads.
1 affected page
A link label tells readers and crawlers what they will find at its destination. An empty link omits that context.
Inspect the listed links in the page source and confirm each has a label that describes where it goes.
1 affected page
Crawlers follow internal links to discover pages. Links that end in 4xx or 5xx errors waste the crawl and leave the linked content unreachable.
Open each listed URL and confirm it returns 200 or redirects to a page that does, and that the source pages no longer link to error pages.
1 affected page
The sitemap directs crawlers to pages the site wants discovered. An error response at a listed URL prevents them from reading that page.
Fetch the sitemap again and confirm every flagged address either returns its intended page or has been removed.
2 affected pages
When the same text is reachable at several addresses and none is marked as the main one, crawlers split their attention between copies and may pick the wrong address to show.
Open each duplicate URL and confirm it either redirects to the chosen URL or declares it as canonical.
The fixes above, grouped by where they are usually made, so each change can go to the person who owns it.
Each rule appears once across the sample. A points figure shows how the overall score would change if that whole check passed with all other evidence held fixed. Unknown checks lack evidence. A check marked not applicable has no eligible subject in the sample.
| Check | Result | Rule weight |
|---|---|---|
| Start page HTTP response | Passed | 10 |
| Start page redirect destination | Passed | 2 |
| Search crawler policy | Failed+9 pts | 10 |
| User-requested fetch policy | Passed | 6 |
| Training crawler policy | Advisory | Informational |
| Indexing and snippet directives | Passed | 8 |
| Firewall challenges | Passed | 8 |
| Optional llms.txt file | Passed | Informational |
| Check | Result | Rule weight |
|---|---|---|
| Main text in server HTML | Passed | 10 |
| Raw and rendered text lengths | Passed | 8 |
| Document language | Passed | 3 |
| Content beyond navigation and footer | Failed+6 pts | 5 |
| Check | Result | Rule weight |
|---|---|---|
| Document title | Passed | 4 |
| Main page heading | Passed | 3 |
| Heading order | Passed | 2 |
| Main content landmark | Passed | 2 |
| Canonical URL consistency | Passed | 3 |
| Labels on crawlable links | Failed+2 pts | 2 |
| Language alternate destinations | Failed+2 pts | 2 |
| JSON-LD syntax | Passed | 3 |
| Fields in supported structured data | Passed | 3 |
| Check | Result | Rule weight |
|---|---|---|
| Sitemap discovery | Passed | 4 |
| HTTP responses from sitemap entries | Failed+3 pts | 3 |
| Responses from linked internal URLs | Failed+3 pts | 3 |
| Duplicate content across URLs | Failed+2 pts | 2 |
Each category earns the weight of its passed rules divided by the weight of its evaluated rules. Access carries 40%, readable content 30%, structure 20% and discovery 10%. Categories without evaluated weight are left out and the remaining weights are scaled proportionally.
A score requires a successful start page response and at least 80% coverage of applicable rule weight. Each scored rule counts once. The examples and affected URL counts show how widespread a finding is.
Additional measurements. The fix plan orders failed checks by score points divided by an effort weight (quick change 1, template or config change 2, larger project 4); the weights only decide the order, and a step shows Site level when its check is not counted per page. The JavaScript comparison uses pages whose browser view has at least 50 words of main content (the main landmark, else a single article element, else the body without navigation, header, footer and sidebars), leaves out consent banners, dialogs and chat widgets, and reports the share of distinct main-content words of the browser view that are present in the server HTML; word boundaries depend on the script. Text share is visible text as a share of the HTML bytes. Estimated tokens are visible text characters divided by 4, a rule of thumb, not a tokenizer, given only for pages written mainly in Latin script. Page weight counts only pages whose HTML was read. Response times come from one request per page sent from our server. The crawler instructions section compares robots.txt, Content-Signal lines, the sitemap, llms.txt, canonical links and robots directives with each other, reports a URL as disallowed only when robots.txt disallows it for every search crawler evaluated, and is left out when no page could be read. Modification dates of one page are reported as disagreeing when they lie more than 30 days apart. The compared dates are the sitemap lastmod, the JSON-LD dateModified of the page’s main entity and the meta modified time, and only on pages that declare at least two of them. The HTTP Last-Modified header is listed but not compared, because on many hosts it is the time of the last deploy and not of the last content change. Sitemap figures describe the sitemap entries examined, which can be fewer than the sitemap lists. Structured data values are compared with the visible text, title, description or image alt text of the same page, and Organization consistency is about the Organization name only. The 2 MB figure is the size Google documents for Googlebot; every other threshold here is our own heuristic. None of these measurements affects the score.
Rule version 2026.11.3. Training preferences, the optional llms.txt file and the descriptive page measurements carry no additional points.
Use Search Console for impressions, clicks and average position by query and page. Use verified server logs for crawler visits, referral analytics for visits and conversions, and dated prompt observations for citations. These sources supply outcomes beyond this crawl.
Google AI search eligibilityIndexing and snippet controlsImage alternatives