How the Crawlito AI readiness score works

The Crawlito AI readiness score is a number from 0 to 100 that says how many of our published technical checks your site passes, weighted by how much each one matters for AI crawlers reaching and reading your pages. Fixed rules compute it, not an AI model, so the same evidence always produces the same score. This post explains the categories, the weights, how unknown results are handled and what the number does not tell you.

If you want the wider picture first, What is AI search readiness? covers why these four areas matter at all.

What the four scored categories measure

Every check belongs to one of four categories. Each category gets its own score from 0 to 100, and the overall score combines them by fixed weights.

Access carries the most weight because nothing else matters if a crawler is turned away. Perfect headings and schema do not help if robots.txt blocks OAI-SearchBot or a firewall serves a challenge page instead of the content.

Readable content comes next because many AI crawlers do not run JavaScript. Vercel and MERJ's December 2024 analysis found that none of the major AI crawlers they measured rendered JavaScript at the time, so text that appears only after scripts run may never be read.

Structure and Discovery matter, but they tend to refine what a crawler understands rather than decide whether it gets in. That is why they carry less weight.

How each check is scored: pass, fail, unknown or not applicable

Each check is a rule that looks at evidence we collected from your site and returns one of four results:

• Pass: the evidence shows the site meets the check.
• Fail: the evidence shows a problem. The finding gives the number of affected pages, example URLs and the evidence behind it, such as the robots.txt line, response header or HTML element.
• Unknown: we could not collect the evidence to decide. For example, robots.txt timed out, so we cannot say whether it blocks a crawler.
• Not applicable: the check cannot apply to your site. For example, if your pages carry no JSON-LD, the checks for valid JSON-LD have nothing to test.

Inside a category, checks carry different weights. Whether the homepage can be fetched counts for far more than whether an llms.txt file exists. The category score is:

category score = 100 x (weight of passed checks) / (weight of passed + failed checks)

Unknown and not applicable checks are left out of the denominator. We do not count an unknown as a fail, because that would punish you for our missing evidence. We do not count it as a pass either, because that would hide a possible problem.

A check passes or fails for the site as a whole. If one page out of 400 carries a stray noindex, the noindex check fails, and the finding lists that page. The score does not drop further because more pages are affected; the affected-page count and severity tell you how big the problem is and what to fix first.

How the overall score is combined

The overall score is a weighted average of the four category scores. Here is a worked example with made-up numbers:

The calculation is (75 x 40 + 100 x 30 + 60 x 20 + 50 x 10) / 100 = 77.

If a whole category has nothing evaluated, for example every Discovery check came back unknown, that category drops out and the remaining weights are rescaled so they still add up to 100%. The other three categories then decide the number on their own, the category is shown as not evaluated, and the coverage figure falls to tell you so.

What coverage means, and when we show no score

Every report shows a coverage figure next to the score. Coverage is the weight of checks we could evaluate (pass or fail) divided by the weight of all checks that apply to your site. Not applicable checks are excluded from both sides.

Say the checks that apply to a site carry 100 points of weight in total, and robots.txt timed out during the crawl. The robots.txt checks for search, user-triggered and training crawlers become unknown. If those carry 16 points, coverage is 84%: the score still appears, and the lower coverage figure tells you part of the evidence is missing.

Two situations replace the number with a label:

1. Coverage below 80%. The report shows Incomplete. A score built on less than four fifths of the evidence would look precise without being reliable, so we show the coverage figure and the category scores instead.
2. The homepage does not return a successful (2xx) response. The report shows Access blocked, and the homepage finding records the status code we received. If the homepage returns a 403 from a firewall, that is usually the most important finding of the whole audit, and a number would bury it.

You can still read the category scores and findings in an Incomplete or Access blocked report. Only the overall number is withheld. If your robots.txt disallows CrawlitoBot by name for the URL you submit, we stop before fetching any page; About CrawlitoBot explains how.

Why a training opt-out never lowers the score

Many sites block training crawlers on purpose. That is a business decision, not a readiness problem, and the score treats it that way.

The major operators document separate user agents for training and for search. OpenAI says its OAI-SearchBot and GPTBot settings are independent, so you can allow OAI-SearchBot for ChatGPT search while disallowing GPTBot for training. Anthropic's crawler article describes ClaudeBot as collecting content that may contribute to training, while Claude-SearchBot and Claude-User serve search and user requests. Google says Google-Extended does not affect inclusion in Google Search.

So our robots.txt checks group crawlers by purpose:

The user-triggered row has a caveat. Anthropic says its bots honor robots.txt, but OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the fetch. We still flag a Disallow for these agents, because it states that you do not want those fetches and it usually comes from a rule meant for something else.

The training check still appears in the report so you can confirm the opt-out is doing what you meant. What we watch for is the common side effect: a User-agent: * group with Disallow: /, written to stop training bots, that also catches the search crawlers. A file like this passes both scored robots.txt checks, and the informational training check simply reports that GPTBot and ClaudeBot are opted out:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

How to allow AI search but block AI training walks through this setup, and training, search and user-triggered crawlers explains why the split matters.

What Claude does, and does not do, in the report

In a full audit, Claude, Anthropic's model, has one job: turning findings the rules have already verified into plain-language explanations, fix steps and a short summary. It cannot add a finding, remove one or change its severity, and it receives only the structured findings with short evidence excerpts, never the full HTML of your pages. The score is calculated before any explanation is written, so the wording of a fix cannot move the number. The free check does not use Claude at all; its explanations are fixed template text.

This matters for two reasons. The result is reproducible: the same evidence always gives the same findings and the same score. And every finding traces back to evidence you can check yourself, such as the robots.txt line that matched, the response header or the HTML element.

What the score does not measure

A score of 100 means your site passes every published check that applies to it. It does not mean ChatGPT, Perplexity, Claude or Google will cite you. Whether an assistant uses a page depends on the question, the competing sources, how each system ranks results and many other things outside your site.

Google says plainly that its AI features need no special technical work: a page must be indexed and eligible to show with a snippet, and no new machine-readable files or special schema are required. That is a good reminder that readiness is about removing barriers, not earning a place. It is also why llms.txt carries a very low weight in our score: it is an optional proposal, and no major AI search operator documents reading it on third-party sites. Our llms.txt guide covers when it is still worth writing one.

If you want to know whether AI tools actually use your content, measure that separately with server logs and referral data, as described in how to measure AI crawler visits and AI referral traffic. The score tells you whether the technical door is open.

Frequently asked questions

Is the Crawlito score generated by AI?

No. Fixed rules produce every finding and the score. In a full audit, Claude only writes the explanations and fix steps from findings the rules have already verified, and it cannot change a finding or its severity.

Why does my report say Incomplete instead of showing a score?

We could evaluate less than 80% of the applicable checks, by weight, usually because pages or robots.txt timed out or returned errors. Fix the cause, such as a firewall rule or a slow server, and run the check again.

Will blocking GPTBot or ClaudeBot lower my score?

No. Training crawlers carry zero weight. Blocking them only affects the score if the same rule also blocks search or user-triggered crawlers, for example through a shared User-agent: * group.

Does a score of 100 mean AI search tools will cite my site?

No. It means every published check that applies to your site passes. Whether an assistant cites you depends on many factors outside your site, and no one can promise that.

Do more affected pages lower the score further?

No. A check passes or fails for the site, whether one page or a hundred are affected. The report gives the affected-page count and severity so you can prioritize.

Check your own score

The fastest way to see how this works is to run it on one of your pages. The free single-page check needs no account and runs the same rules on that one page, showing its score, coverage and findings. For every page up to 500, with human review and prioritized fixes, see the full audit; what an audit covers explains what you receive.