Skip to content

Brave Search crawler: why it has no user agent to block

The Brave Search crawler sends no distinct user agent and skips what Googlebot cannot crawl. How that changes robots.txt, log analysis and opting out with noindex.

By Crawlito · 7 min read

In short

  • Brave says its search crawler does not advertise a differentiated user agent, so there is no Brave token to name in robots.txt and no Brave name to grep for in logs.
  • Brave says that if a domain or page is not crawlable by Googlebot, its own bot will not crawl it either, so anything you block for Googlebot is also off limits to Brave.
  • Brave states that robots.txt is not used to prevent indexing; to delist a page you add a robots noindex directive and Brave has to re-fetch the page to apply it.
  • Brave's help page says only pages returning 404 or 410 are removed automatically, and the submit-url form or not-found@brave.com speeds up removal and re-fetching.
  • Brave publishes no crawler IP ranges or hostnames at the time of writing, so a request claiming to be Brave's crawler cannot be verified the way Googlebot can.
On this page

The Brave Search crawler has no user agent of its own. Brave's crawler help page says it "does not advertise a differentiated user agent" so that sites allowing only Google do not shut it out, and that it will not crawl any domain or page Googlebot cannot crawl. So you cannot block or allow Brave by name: your Googlebot rules decide its access, and removal from Brave Search works through noindex plus a re-fetch, not through robots.txt.

That matters beyond Brave's own results page. Brave says its search results are served from its own index, its AI Answers feature gives a summary answer with references to sources, and the Brave Search API is marketed for grounding AI agents and applications. For how Brave's rules can affect Claude's web search, see how Claude web search finds sources. This post covers what that missing user agent means in practice: robots.txt, opting out, log analysis and Google-only allowlists. For the named AI bots and their tokens, see the AI crawler user agents list.

What Brave documents about its crawler

Brave's help pages are short, so it is worth being exact about what they say and what they leave out. At the time of writing:

TopicWhat Brave saysSource
User agentNot differentiated, to avoid discrimination by sites that allow only GoogleCrawler help page
Access ruleIf Googlebot cannot crawl a domain or page, Brave's bot will not crawl it eitherSame page
robots.txtNot used to prevent a page from being indexedSame page
DelistingUse the robots noindex directive; Brave must re-fetch the page to apply itSame page
Re-fetchSubmit the URL through Brave's submit URL formSame page
Deleted pagesOnly pages returning 404 Not Found or 410 Gone are removed automaticallyNot Found help page
URL discoveryPartly informed by the opt-in Web Discovery Project in the Brave browserCrawler help page
IndexIndependent of other search engines; results served solely from itIndex and Right to be Forgotten page

What Brave does not document: the user agent string it does send, its IP ranges or reverse DNS hostnames, its crawl rate, or which robots.txt group it reads. Anything you read elsewhere about a "BraveBot" token is not backed by Brave's own pages.

Why there is no Brave group in robots.txt

RFC 9309, the robots.txt standard, says crawlers "set their own name, which is called a product token, to find relevant groups", and that the token should be a substring of the User-Agent header they send. A crawler that does not announce a name has no token for you to target. Writing a group like this does nothing you can rely on, because Brave has never said it answers to that name:

# Not documented by Brave: do not rely on this
User-agent: BraveBot
Disallow: /

Instead, Brave ties its access to Googlebot's. The practical reading is that whatever your Googlebot group blocks, Brave also stays out of:

User-agent: Googlebot
Disallow: /checkout/
Disallow: /drafts/

User-agent: *
Disallow: /checkout/

With this file, Brave's stated rule means it will not crawl /checkout/ or /drafts/, because Googlebot cannot. Brave does not say whether it also reads the * group, so if you want something kept away from Brave, put it in the group Googlebot obeys. How Googlebot itself picks its group, including why a Googlebot group ignores your * rules, is explained in robots.txt groups explained.

The trade-off is plain: you cannot allow Googlebot to a path and keep Brave out of it with robots.txt. If that is your goal, noindex is the documented tool, and it applies to Brave's index rather than to its fetching.

Brave's removal process differs from blocking. Follow it in this order:

  1. Add noindex to the page, either in the HTML or as an HTTP header:
html
<meta name="robots" content="noindex">
http
HTTP/1.1 200 OK
X-Robots-Tag: noindex
  1. Keep the page crawlable for Googlebot. Brave has to re-fetch the page to see the directive, and it will not fetch what Googlebot cannot. A page that is both disallowed in robots.txt and tagged noindex gives Brave no way to read the tag. This is the same trap Google describes for its own crawler, covered in noindex and nosnippet for AI search.
  2. Request a re-fetch through the submit URL form so you do not wait for the next routine crawl.
  3. For deleted pages, return 404 or 410. Brave says only those two status codes trigger automatic removal, so a deleted page that redirects to the homepage or returns 200 with an error message may stay listed. You can submit the URL through the same form or email not-found@brave.com.
  4. For pages with personal data about you, Brave's crawler page points to its Right to Be Forgotten process, which takes requests by email to privacy@brave.com and says delisting can take up to 30 days.

Two points worth knowing. Brave's page names only the generic robots noindex directive; it does not mention a Brave-specific meta name, so do not invent one. And Brave does not document separately how a delisted page is treated by its AI features or by Search API results, so do not assume more than removal from the index. For the wider picture across ChatGPT, Perplexity and Google, see removing a page from AI search.

How to find Brave's crawler in server logs

Short answer: you cannot identify it with certainty. With no documented user agent, IP list or hostname, there is nothing to match against and nothing to verify. Practical consequences:

  • A request whose user agent says "Brave" is not evidence. Any client can send any string, and Brave has not published one for its crawler.
  • You cannot confirm a suspected Brave visit the way you confirm Googlebot, with a reverse DNS lookup that must end in googlebot.com, google.com or googleusercontent.com and a matching forward lookup, as Google's verification guide describes.
  • Your "unknown bot" bucket may include Brave. Crawl-like traffic that matches no documented crawler could come from Brave, from scrapers or from tools. Treat it as unattributed, not as Brave.

The useful log work is indirect. Check that Googlebot gets 200 responses on the pages you care about, because that is the access Brave mirrors. Then check that your important pages appear in Brave Search with a site: query. Our guide to identifying an unknown bot user agent covers the general triage for unlabelled traffic.

What Google-only allowlists mean for Brave

Brave's reason for hiding its user agent is aimed at one pattern: sites that let Google in and turn everyone else away. How much that matters depends on how your allowlist decides who is Google.

Your ruleEffect on Brave, from what Brave documents
robots.txt allows Googlebot, disallows *Brave's rule only says it skips what Googlebot cannot crawl; it does not say whether it also obeys the * group, so its access here is not documented
robots.txt disallows Googlebot from a pathBrave will not crawl that path
Firewall allows only verified Google IPs or reverse DNSBrave does not say it crawls from Google addresses, so expect its fetches to be refused at the edge regardless of robots.txt
Firewall blocks requests that claim Googlebot but fail verificationOnly affects Brave if its requests claim to be Googlebot, which Brave does not document
noindex on a pageBrave delists it after re-fetching

The third row is the one that catches people. Brave's Googlebot parity is a promise to respect Googlebot's limits, not a way past network rules. If your WAF or bot protection admits only verified Google crawlers, Brave and every other independent index are locked out at the network level. Whether that is acceptable depends on whether you want to be in Brave Search and in the tools that use its index.

How to check your setup for Brave

  1. Read the Googlebot group in your live robots.txt, since that is the access Brave mirrors:
bash
curl -s https://example.com/robots.txt | grep -i -A10 "user-agent: googlebot"
  1. Look for noindex in the HTML and headers of pages you want in Brave Search, and confirm it is present on pages you want out:
bash
curl -sI https://example.com/page | grep -i x-robots-tag
curl -s https://example.com/page | grep -io '<meta[^>]*robots[^>]*>'
  1. Check deleted URLs return 404 or 410, not a redirect or a 200.
  2. Review firewall and bot protection rules for any rule that admits only verified search engine IPs.
  3. Search Brave with site:example.com and compare against your important URLs. Submit missing or stale ones through the submit URL form. Being re-fetched does not guarantee a page is listed or shown in AI answers.

Frequently asked questions

What is the Brave Search crawler user agent?

Brave does not publish one. Its help page says the crawler does not advertise a differentiated user agent, to avoid being excluded by sites that allow only Google.

Can I block Brave Search in robots.txt?

Not by name, because there is no documented Brave token. Brave says it will not crawl what Googlebot cannot, so blocking Googlebot is the only documented way to keep it out, and robots.txt does not remove pages from its index.

Add a robots noindex directive, keep the pages fetchable, and request a re-fetch through Brave's submit URL form. Deleted pages should return 404 or 410, which Brave removes automatically once re-fetched.

Does Brave use Google's results?

Brave says its results are served from its own independent index. A separate, optional browser feature called Google fallback mixing lets the Brave browser check Google for some queries and mix results in the browser.

Can I verify that a request came from Brave's crawler?

No. At the time of writing Brave publishes no IP ranges, hostnames or user agent string for its crawler, so there is nothing to verify against.

Check what Googlebot, and so Brave, can reach

Because Brave mirrors Googlebot's access, one robots.txt or firewall mistake can remove you from both. Run the free check on an important page to see whether major crawlers can fetch it and whether a noindex is present. For a whole site, an audit reviews robots.txt, headers and edge rules together with evidence for each finding.

Filed under AI crawlers. Topics: ai-crawlers, robots-txt, noindex, server-logs