Crawl budget and AI search: diagnose demand before tuning
Diagnose crawl budget concerns with URL inventories, verified logs and server evidence before changing crawler access or removing useful pages.
By Crawlito · 6 min read
In short
- Crawl budget is not one shared allowance distributed among all search engines and AI services.
- Low discovery, poor indexing and server overload are different problems that need different evidence.
- Group requests by useful content, duplicate states and errors before deciding what to restrict.
On this page
Crawl budget describes limits and choices within a crawler's discovery process, not a universal quota that your website spends across all AI companies. Before trying to optimize it, establish whether you have a crawling problem, an indexing problem or an overloaded server. Those symptoms can look similar in a dashboard but require different fixes.
A small site with a few missing pages should usually begin with access, links and content checks. A large, frequently changing catalog may need deeper analysis of URL expansion and request patterns. The right level of work follows the evidence, not the popularity of the term.
Separate three questions that are often combined
First, can the page be found and fetched? Second, does a search system choose to retain or index it? Third, does an answer or search result use it for a particular query? A successful crawl proves only part of the first question.
Google's crawl-budget documentation frames its guidance primarily around large or rapidly changing sites. It describes capacity and demand as separate factors, and notes that crawling does not ensure indexing. Treat that as Google's explanation of its own system, not a specification for every AI operator.
For your investigation, write a symptom statement with a date and affected URL group. “New help articles have not appeared in verified request logs during the observation window” is more useful than “AI has stopped crawling us.” The former can be tested; the latter combines several uncertain assumptions.
Also distinguish an absent log record from a failed visit. Logs may cover only one edge location, omit cached requests or retain too short a period. Confirm the data's scope before diagnosing the behavior of an external service.
Build a request picture around page purpose
Export a representative time window and group URLs by template or purpose. Useful groups include product pages, editorial pages, category pages, filter combinations, internal search, assets and errors. Avoid grouping only by user-agent name, because that hides what the requests actually reached.
Use the verification process in identifying real AI crawlers before treating self-reported user agents as verified visitors. Keep unverified traffic in a separate bucket. A bot can claim a recognizable name without being operated by that company.
An investigation table might contain:
| Group | Requests | Unique URLs | Response pattern | Next question |
|---|---|---|---|---|
| Current articles | Measured count | Measured count | Mostly successful | Are new articles represented? |
| Filter states | Measured count | Measured count | Repeated similar pages | Is the URL space bounded? |
| Retired routes | Measured count | Measured count | Errors or redirects | Are old links still generated? |
| Product pages | Measured count | Measured count | Mixed availability | Is the origin healthy? |
Populate it with your own measurements. The relationship between repeated requests and distinct useful pages is often more informative than the total bot count alone.
Check server constraints independently
If visitors and crawlers experience timeouts, inspect the delivery stack before revising editorial strategy. Look at origin latency, application errors, cache behavior and resource saturation during the same window. A slow database query on filtered pages can affect the rest of the catalog.
Correlate timestamps carefully. A request spike at the CDN and a database incident an hour later do not establish cause. Use a common timezone and keep the original timestamps so the comparison can be reproduced.
The status-code guide explains the access signals involved. For this investigation, record the actual distribution by route type. A healthy average can conceal one important template returning errors to every request.
Avoid solving an overload problem by globally allowing more bots. Conversely, do not remove all search access simply because some automated traffic is abusive. Separate verified services, unwanted traffic and ordinary visitors, then apply the narrowest operational fix that addresses the measured problem.
Reduce unnecessary URL work at its source
Look for links that create endlessly changing or duplicative addresses. Common examples include session values in internal URLs, unlimited price ranges, repeated filter parameters and calendars with no useful boundary. The key question is whether your own interface advertises those states.
Fixing the generator is often more durable than maintaining an expanding list of exclusions. Decide which URL states deserve public content, which are temporary interface views and which are invalid. Then align navigation, canonical choices and access rules with that policy.
Do not assume every duplicate needs a redirect or every parameter needs a block. A pagination parameter may be essential for discovery. Review representative positive cases before applying a broad pattern, and keep a rollback path for changes that affect a large portion of the site.
A good remediation record identifies the generating component, the unwanted pattern, an example useful URL that must remain available, and the expected result after release. That makes the work reviewable by developers and content owners without relying on the vague instruction to save crawl budget.
Measure a fix without inventing a ranking story
Choose an operational success criterion before deployment. Examples include stopping the generation of invalid filter links, reducing server errors on current products or restoring discovery of newly published guides. Each criterion should have a corresponding observation you can collect.
Compare equivalent windows where possible and note changes in inventory, marketing activity and releases. A holiday sale or a new catalog import can change demand independently of your technical fix. Avoid assigning all movement in request volume to the change you just made.
Keep crawl measurements separate from referrals and conversions. The traffic measurement guide covers that distinction. A bot request is not a visitor, and a visitor is not automatically a qualified lead.
Report the narrow result honestly: “The invalid route is no longer linked and returns the intended response” is a useful verified outcome. “This increased our chance of AI citations by a precise percentage” would require evidence that ordinary access logs cannot provide.
Decide when deeper analysis is worth the effort
For a small maintained site, a clear sitemap, working links and readable pages may resolve the practical issue. Spend the next hour inspecting a missing page rather than building a sophisticated request dashboard if the problem is a leftover noindex directive.
Deeper analysis becomes more useful when the URL space is large, publishing is frequent, server pressure is measurable or new content repeatedly goes undiscovered despite sound basic access. Even then, start with a focused sample and expand only when the evidence justifies it.
Set a review cadence based on change. A stable brochure site does not need the same monitoring as a marketplace with constant inventory churn. Large template releases, filter redesigns and infrastructure changes are sensible triggers for another inspection.
Document unknowns alongside findings. You may know that a page was fetched but not why a search service did not use it. Keeping that boundary clear prevents unnecessary technical changes made in pursuit of an outcome the current evidence cannot explain.
Frequently asked questions
Do AI crawlers share Google's crawl budget?
No shared cross-company budget is established by Google's documentation. Treat each service's requests and published policies separately rather than assuming one bot consumes another's allocation.
Does fewer crawler traffic mean the site is healthier?
Not necessarily. Fewer unnecessary requests can be useful, but a drop caused by blocked product or article pages is a different outcome. Measure the affected URL groups.
Should a small site prioritize crawl-budget work?
Begin with access, discovery and content checks unless there is concrete evidence of URL expansion or overload. Complex tuning is rarely the best first response to one missing page.
What should an audit deliver?
A clear symptom, the evidence window, affected URL patterns, a proposed fix and a repeatable verification step. It should distinguish measured facts from theories about search selection.
Start with the affected page in the free check, then expand the investigation only when the findings point to a wider site issue.
Filed under Strategy and audits. Topics: crawl-budget, server-logs, ai-search