How Claude web search finds and cites sources
How Claude web search works: when it searches, the providers Anthropic lists, how citations work, what the API exposes and how to check your site is reachable.
By Crawlito · 9 min read
In short
- Claude searches when a question needs current or changing information, runs one or more searches, and attaches citations to the sources it used.
- Anthropic's subprocessor list names Brave Search, TurboPuffer and Parallel Web Systems under Web Search, and its help centre says image results are powered by Bing.
- Anthropic does not document how these providers and its own Claude-SearchBot crawl combine, or how Claude chooses which sources to cite.
- Brave says its crawler uses no distinct user agent and will not crawl a page Googlebot cannot crawl, so your rules for Googlebot also affect one of Claude's listed search providers.
- In the API, each web search citation carries the source url, title and up to 150 characters of cited_text, and developers can restrict results with allowed_domains or blocked_domains.
On this page
- When Claude decides to search
- Where Claude's search results come from
- Why Brave's crawler rules matter for Claude
- How citations are attached to Claude's answers
- What developers can control with the web search tool
- What Anthropic does not document
- How to check that Claude search can reach your pages
- Frequently asked questions
- Check what Claude's crawlers receive
Claude web search works by letting the model decide when a question needs current information, running one or more searches through Anthropic's search tool, and writing an answer with citations to the pages it used. Anthropic's subprocessor list names three web search providers (Brave Search, TurboPuffer and Parallel Web Systems), and Anthropic also runs its own search crawler, Claude-SearchBot. What Anthropic does not publish is how those pieces are combined or how Claude picks the sources it cites.
This post covers the search side: when Claude searches, where results come from, what a citation contains and what the developer API exposes. For the three Anthropic bots and the robots.txt rules for each, read ClaudeBot, Claude-SearchBot and Claude-User explained. That post is not repeated here.
When Claude decides to search
Anthropic's help article on enabling and using web search says Claude invokes a search tool when you ask about topics that benefit from current information; its example is a question about the latest developments in quantum computing. A user can also ask for it directly ("Search the web") or tell Claude not to search. The same article notes that in the new Claude experience there is no web search toggle and Claude searches when it helps. On Team and Enterprise plans, an Owner or Primary Owner first has to enable web search for the workspace under Organization settings > Capabilities.
The web search tool documentation for developers is more specific. Claude searches for:
- recent events, news or announcements
- current prices, rates, scores or statistics
- organizations, people or products whose details might have changed
- explicit requests to look something up
It answers without searching for established facts, maths, coding concepts, creative writing, analysis of content already in the conversation and greetings.
For a site owner, that list says something useful. Pages that hold changing facts (pricing, opening hours, product specs, policies, release notes) are the kind of content a search is likely to look for. Pages that restate stable general knowledge are more likely to be answered from what the model already knows. Grounding versus training data explains the difference between those two paths.
One question can mean several searches
The API docs say the search process "can repeat multiple times throughout a single request" before Claude writes its final answer. They add that single-fact queries typically use one to three searches, while comparisons or research across several entities can use ten or more. In an API response, each server_tool_use block shows the query Claude sent, which is the closest you can get to seeing the phrasings that led to a page being found.
Where Claude's search results come from
Three documented sources feed into what Claude can find:
| Source | What Anthropic documents | Where |
|---|---|---|
| Brave Search | Listed as a subprocessor for "Web Search", all products | Anthropic subprocessor list |
| TurboPuffer | Listed for "Web Search", all products except Claude for Government | Same list |
| Parallel Web Systems | Listed for "Web Search", all products except Claude for Government | Same list |
| Bing | Powers image results shown alongside answers | Web search help article |
| Claude-SearchBot | Anthropic's crawler that "navigates the web to improve search result quality" | Anthropic crawler article |
The subprocessor list was read on 2026-11-15 and changes from time to time. A subprocessor entry tells you a company is involved in web search. It does not tell you what share of results each provides, whether one is used as a primary index, or how results are ranked. The help article itself names only Bing, and only for images.
Anthropic's crawler article adds one more piece: blocking Claude-SearchBot "prevents our system from indexing your content for search optimization", which "may reduce your site's visibility and accuracy in user search results". That wording implies Anthropic indexes some content itself alongside the providers it lists, but it does not say how the two are merged.
Compare this with OpenAI, whose help article mentions third-party search providers without naming them (how ChatGPT search finds sources), and Perplexity, whose PerplexityBot is documented as surfacing and linking websites in Perplexity's search results (how Perplexity finds and cites sources).
Why Brave's crawler rules matter for Claude
If Brave Search is one of the indexes behind Claude's results, Brave's own crawling rules affect whether your pages can be in it. Brave's crawler help page documents three things that surprise many site owners:
- No distinct user agent. Brave says its crawler "does not advertise a differentiated user agent", to avoid being shut out by sites that only allow Google. You cannot find it in logs by name, and the page documents no robots.txt token of its own.
- It follows Googlebot's access. Brave says that if a domain or page is not crawlable by Googlebot, its bot will not crawl it either. Brave does not explain how it decides this, but a robots.txt rule that blocks Googlebot is the plain case.
- robots.txt does not remove pages. Brave says robots.txt is not used to prevent indexing. To delist a page you use a
noindexrobots directive, and Brave has to re-fetch the page to see it. You can request a re-fetch at search.brave.com/submit-url.
In practice, the robots.txt group that Googlebot obeys is the one to read when you want to know Brave's access. By Brave's stated rule, a file like this one keeps Brave out of /guides/ as well, even though Brave is never named:
User-agent: Googlebot
Disallow: /guides/
User-agent: *
Allow: /The Brave Search crawler and its user agent explains why it cannot be identified in logs and how to opt out with noindex. None of this guarantees that Claude surfaces a page Brave indexes, since Anthropic does not say how it uses Brave's results.
How citations are attached to Claude's answers
The help article says every web search response includes citations, so readers can check the sources, and that responses may include links for further reading and quotes. Image results link back to their original pages.
The API shows what a citation actually contains. Citations are always on for web search, and each one is a web_search_result_location block:
{
"type": "web_search_result_location",
"url": "https://www.example.com/pricing",
"title": "Pricing | Example",
"cited_text": "The Team plan costs $12 per user per month, billed annually...",
"encrypted_index": "Eo8BCioIAhgBIiQy..."
}Three fields matter to a site owner:
urlis the page that is credited. Anthropic does not say whether canonical tags are taken into account, so redirect chains and duplicate URLs can leave a version you did not intend with the link.titleis documented only as the title of the cited source. Anthropic does not say how it is derived, but a clear, unique<title>is the safest input you control.cited_textholds up to 150 characters of the passage the answer relies on. A sentence that states the fact plainly, close to the words a user would ask, is easier to quote cleanly than one spread across a table cell and a footnote.
Each search result also carries a page_age field, which Anthropic describes as when the site was last updated. The docs do not say where that date comes from, so do not assume it is read from your sitemap or from dateModified markup. Keeping visible dates accurate is still sensible.
The docs also say that when developers show API output directly to end users, citations to the original source must be included. That applies to apps built on the API, not only to Claude's own interface.
What developers can control with the web search tool
If you build on the Claude API, the tool definition gives you controls that do not exist in the consumer app. At the time of writing there are three tool versions: web_search_20250305 (basic search), web_search_20260209 (adds dynamic filtering, where Claude runs code to filter results before they reach its context) and web_search_20260318 (adds a response_inclusion setting). The basic version looks like this:
{
"type": "web_search_20250305",
"name": "web_search",
"max_uses": 5,
"allowed_domains": ["example.com", "docs.example.com"],
"user_location": {
"type": "approximate",
"country": "US",
"timezone": "America/New_York"
}
}The documented rules, at the time of writing:
| Setting | Behaviour |
|---|---|
allowed_domains / blocked_domains | Use one or the other, not both. Entries have no scheme. example.com includes its subdomains; docs.example.com matches only that subdomain. Paths such as example.com/blog are supported for search. |
| Wildcards | Only in the path (example.com/*/articles), never in the domain (*.example.com is rejected) |
max_uses | Caps searches per turn; failed searches count |
user_location | Localizes results by city, region, country or timezone |
| Organization settings | Admins can disable web search or restrict domains in the Claude Console; a request's allowed_domains must be a subset of the organization's allowed list |
| Price | $10 per 1,000 searches plus token costs; failed searches are not billed |
Two practical uses for site owners. You can point a test at your own domain with allowed_domains to see which of your pages the search returns for a question, and how Claude quotes them. And if your product is built on Claude, allowed_domains keeps its answers grounded in your own docs.
In the API, reading a specific URL is a separate tool, web fetch. Anthropic's docs say it does not currently support websites rendered with JavaScript, and they list robots.txt among the reasons for a url_not_allowed error. Why Claude cannot access a URL goes through those errors.
What Anthropic does not document
| Question | Documented? |
|---|---|
| How results from Brave, TurboPuffer, Parallel and Claude-SearchBot are combined | No |
| Ranking or citation criteria | No |
| A form to submit a site to Claude search | No |
| Whether Claude-SearchBot reads XML sitemaps | No |
| Whether structured data or llms.txt affects inclusion | No |
| A referral parameter on links from Claude | Not in the articles above |
| How often Claude-SearchBot recrawls | No; the crawler article only says Anthropic is "thoughtful about how quickly we crawl the same domains" |
Treat any claim about these as unverified until Anthropic publishes it.
How to check that Claude search can reach your pages
- Check the Googlebot group in robots.txt, because Brave follows Googlebot's access. Then check your groups for Claude-SearchBot and Claude-User.
curl -s https://www.example.com/robots.txt | grep -iA3 -E "googlebot|claude"- Look for Anthropic's bots in your logs and the status codes they received:
grep -E "Claude-SearchBot|Claude-User" /var/log/nginx/access.log \
| awk '{print $9}' | sort | uniq -c- Make sure the facts are in the raw HTML. Fetch the page with
curland search for a sentence a citation would quote. Anthropic's web fetch tool does not support JavaScript-rendered sites, and Anthropic does not document JavaScript rendering for its crawlers either. - Check for
noindexin the HTML and theX-Robots-Tagheader, since Brave uses it to delist pages. - Ask Claude a question your page answers with web search on, and read the cited URLs and quoted text. One test is not proof either way, because results vary.
Frequently asked questions
Does Claude use Brave Search?
Anthropic's subprocessor list names Brave Search as a web search subprocessor for all products, alongside TurboPuffer and Parallel Web Systems. Anthropic does not say how much of Claude's results each one supplies.
Does blocking Googlebot affect Claude?
It can. Brave says its crawler will not crawl a page Googlebot cannot crawl, and Brave is one of Claude's listed search providers.
Can I submit my site to Claude search?
Anthropic does not document a submission form. Allow Claude-SearchBot and Claude-User, keep your Googlebot rules open for public pages, and make sure your content is in the server HTML.
Why does Claude cite a different URL from my canonical page?
The citation's url is the cited source's URL, and Anthropic does not say whether canonical tags are applied. Fix redirects and duplicate URLs so the version you want is the one indexes find.
Will allowing Claude-SearchBot get my site cited?
No setting guarantees a citation. Allowing access removes one barrier, and Anthropic warns that blocking Claude-SearchBot may reduce visibility, but it does not publish how it chooses sources.
Check what Claude's crawlers receive
The parts of Claude search you control are access and readability. Run the free check on a page you want Claude to be able to quote, to see whether Anthropic's crawlers and Googlebot are allowed and whether the main content is in the HTML they get.
Filed under Strategy and audits. Topics: anthropic, claudebot, ai-search, ai-crawlers