AI search audit for agencies: how to scope, deliver and price it

An AI search audit for agencies is a technical readiness review you can sell next to SEO work: it checks whether AI search crawlers can reach a client's pages, read the main text in the server's HTML, and parse the page structure. You sell it honestly by promising to find and remove barriers, never to deliver citations or traffic. Scope it by page count and by who controls the hosting and CDN, deliver evidence for every finding, and quote the fixes separately.

What an AI search audit actually checks, in client terms

Clients do not need the vocabulary. They need three questions answered:

1. Can the AI tools get in? HTTP status, redirects, robots.txt rules for each named crawler, robots meta tags and X-Robots-Tag headers, and whether a CDN or WAF serves a challenge page instead of the content.
2. Can they read what they get? Whether the main text is in the raw HTML or only appears after JavaScript runs, and whether key facts are locked in images, canvas or video.
3. Can they make sense of it? Titles, headings, canonicals, structured data that parses and matches the visible page, sitemaps and internal links.

If a client wants the longer explanation, send them what AI search readiness is. For your own scoping, what an AI search readiness audit covers lists the checks in detail.

The reason this is a separate service, and not a line in a technical SEO crawl, is that the crawlers differ in ways an SEO tool does not model. OpenAI documents four agents with different jobs: OAI-SearchBot for ChatGPT search, GPTBot for model training, ChatGPT-User for user-initiated fetches, and OAI-AdsBot for ad landing pages. Anthropic separates ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user-initiated fetches), and Perplexity separates PerplexityBot (search) from Perplexity-User (user-initiated fetches). A site can be in perfect shape for Googlebot and still be blocked for OAI-SearchBot because of one robots.txt group or one firewall rule.

What not to promise clients

This is where agencies get into trouble. The honest position is narrow: an audit removes technical barriers. Whether an assistant then cites a page depends on the question asked, the competition, the content itself and choices made by each AI company, none of which you control.

Two facts help here, because they come from the operators themselves:

• Google says that to be eligible as a supporting link in AI Overviews and AI Mode, a page must be indexed and eligible to show with a snippet, and that there are no additional technical requirements: no new machine-readable files, AI text files or special schema. It also says inclusion is not guaranteed. Google-Extended does not control AI Overviews either; Googlebot rules and snippet controls such as nosnippet and noindex do.
• Google's AI optimization guide says Google Search ignores llms.txt. Selling llms.txt as an AI Overviews fix is selling something Google has said it does not use.

Eight AI search myths is a useful piece to send prospects who arrive with inflated expectations.

How to scope an AI search audit

Ask these questions before you quote. Each one changes the effort.

The training question deserves a minute on the call. OpenAI says its OAI-SearchBot and GPTBot settings are independent, so a site can allow ChatGPT search and disallow training. A client who blocked "AI" with one broad rule may have blocked both. Our guide to allowing AI search while blocking AI training gives the robots.txt for each operator.

A quick pre-sale check you can run in five minutes

Before the call, look at the homepage and one money page so you can say something specific. First, read robots.txt and look for groups that name AI crawlers:

curl -s https://client.example/robots.txt

Next, request a key page with OAI-SearchBot's user agent (OpenAI's example string; the version number may change) and print the status line and any Cloudflare challenge header. This sends a normal GET and discards the body, because some firewalls answer HEAD requests differently:

curl -s -o /dev/null -D - \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  https://client.example/pricing | grep -iE "^HTTP|^cf-mitigated"

Finally, check whether a sentence you can see in the browser is in the raw HTML, before any JavaScript runs:

curl -s https://client.example/pricing | grep -c "a phrase from the page"

A cf-mitigated: challenge header means Cloudflare served a challenge page, not the content. That header marks challenges only: an outright block shows up as a 403 or similar status without it. A count of 0 for a sentence you can see in the browser suggests it is added by JavaScript, though it can also mean the text is split by markup or HTML entities, so try a second phrase before you mention it.

Treat this as a conversation starter, not a finding. A borrowed user agent sent from your own IP does not reproduce rules that match the crawler's real IP ranges. Bot protection that verifies crawlers by published IP lists or reverse DNS will treat your request differently from the real OAI-SearchBot, in either direction. That gap is exactly why the full audit matters.

How it fits next to a technical SEO retainer

AI search readiness overlaps heavily with technical SEO. Do not sell it as a replacement, and do not charge twice for the same checks.

Two practical models work:

• Add-on to an existing retainer. Run the audit once, fold the new checks into your monthly technical review, and add crawler visit reporting. How to measure AI crawler visits and AI referral traffic covers the log queries and analytics filters.
• Standalone project. For clients who do not retain you, sell a fixed-fee audit with one recheck, then quote fixes. This often turns into a retainer once the client sees the findings.

Be clear about reporting limits. Google counts traffic from AI Overviews and AI Mode inside the Search Console Performance report under the Web search type, with no separate AI report. Bing's AI Performance report, in public preview since February 2026, counts citations across Microsoft Copilot, AI-generated summaries in Bing and select partner integrations, not clicks. Neither shows ChatGPT or Perplexity.

What to deliver

A good deliverable lets the client, or their developer, act without calling you. Include:

1. A crawler access table. One row per crawler, grouped by purpose (search, training, user-triggered), with the robots.txt result, the HTTP result and any firewall finding. Mark deliberate training opt-outs as intended, not as failures.
2. Evidence for every finding. The exact robots.txt lines, response headers, or a raw-HTML excerpt next to the rendered text. Evidence is what lets a developer fix it without arguing.
3. A prioritized fix list. Order by impact on search crawlers first: a blanket block or a challenge page outranks a missing Organization schema. Give each fix an owner: content, developer, hosting or CDN admin.
4. Platform-specific steps. "Edit robots.txt" is not useful on Shopify or Wix. Name the setting or template.
5. A recheck date. Robots.txt changes are not instant: OpenAI says its search systems can take about 24 hours to adjust, and Perplexity says changes may take up to 24 hours.

The firewall section is often the one that surprises clients most. For example, Cloudflare's Free-plan Bot Fight Mode cannot be bypassed with WAF custom rules, so "just add an allow rule for GPTBot" does not work there. Cloudflare and AI crawlers walks through the settings that block crawlers and where each one lives.

Here is the kind of row that belongs in the access table:

Crawler        Purpose   robots.txt          HTTP (homepage)     Finding
OAI-SearchBot  Search    Allowed             403, challenge      Firewall blocks ChatGPT search
GPTBot         Training  Disallowed          not fetched         Intended opt-out, no action
PerplexityBot  Search    Allowed             200                 OK
Claude-User    User      Allowed             200                 OK

Pricing considerations

There is no settled market rate for this work yet, so price it the way you price technical SEO: on effort and risk, not on hype.

• Fixed fee for the audit. Clients understand a fixed price for a defined page count. Tier it, for example up to 500 pages and above, and quote very large sites separately.
• Fixes quoted from findings. A robots.txt change takes minutes; moving content out of client-side rendering can take weeks. Quoting fixes before the audit means guessing.
• Recheck included. One recheck after fixes is what turns a report into a result the client can see.
• Monthly reporting as the retainer. Crawler visits from server or CDN logs, a diff against the last audit, and the referral traffic numbers you can actually measure.
• Your time on the call. Explaining findings to a client's developer or IT team is often the biggest cost. Price it in.

If you would rather not build the crawl and evidence tooling yourself, Crawlito offers a white-label agency audit: the same full-site audit (up to 500 pages, deterministic checks, reviewed by a person, delivered as a report page and PDF with evidence and prioritized fixes), at a wholesale price from $150 per audit. The report carries your name, logo and colours instead of Crawlito's, and its Request a fix button emails you or is hidden, so your client never contacts us. The included recheck works the same way. What you charge the client is your call; set it from your own effort on scoping, calls and fixes.

Frequently asked questions

Is an AI search audit just a technical SEO audit with a new name?

No, but they overlap. It adds per-crawler robots.txt and firewall checks, a raw HTML versus rendered comparison for crawlers that do not run JavaScript, and reporting on AI crawler visits, which most technical SEO audits do not cover.

Can I tell clients the audit will get them into ChatGPT answers?

No. You can tell them the audit finds and removes technical reasons ChatGPT search, Perplexity or Claude cannot reach or read their pages. Whether an assistant then cites them depends on factors outside the site.

Should clients block AI training crawlers?

That is a business decision for the client, not a technical requirement. Blocking training tokens such as GPTBot, ClaudeBot or Google-Extended does not remove a site from the matching search products, because the search crawlers are controlled separately.

Do clients need llms.txt?

It is optional. Google Search says it ignores llms.txt, and at the time of writing none of the major AI search operators document that their crawlers read it on third-party sites. Treat it as a low-priority extra, never as the headline fix.

How often should an audit be repeated?

Recheck after fixes ship, then whenever the site changes platform, CDN or bot settings. A light monthly recheck catches regressions such as a firewall or plugin update that starts blocking crawlers again.

Start with one client page

Pick the client page that matters most, usually pricing or the main service page, and run it through the free single-page check. It needs no account, and the result gives you something concrete to bring to the scoping call. When the client signs off on a full audit, order the white-label version through the agency audit form.