Skip to content

ExaSearchBot: Exa's crawler, robots.txt token and signatures

ExaSearchBot is the crawler behind Exa's search API. See its user agent, ExaUser and image agents, robots.txt fallback order, signature check and noindex.

By Crawlito · 7 min read

In short

  • ExaSearchBot is the crawler that discovers and indexes public pages for Exa, a search engine developers query through a web search API, and its robots.txt token is ExaSearchBot.
  • Exa documents two other agents: ExaUser for pages fetched on a user's request and ExaSearchBot-Images for images.
  • Without an ExaSearchBot group, Exa says the crawler follows the rules you set for major search engines, and only then the * group.
  • Exa says every ExaSearchBot request is signed with RFC 9421 HTTP Message Signatures under Web Bot Auth, with Ed25519 keys in a public key directory; its crawler page lists no IP ranges.
  • robots.txt stops crawling, not indexing: Exa removes a page after it re-fetches it and finds a noindex meta tag or X-Robots-Tag header.
On this page

ExaSearchBot is the crawler that builds the index behind Exa, a search engine that developers reach through a web search API. It identifies itself as Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/), obeys robots.txt groups for the ExaSearchBot token, and signs every request so you can verify it without an IP list. If you want apps built on Exa to be able to find your pages, allow it; if not, a two-line robots.txt group blocks it, and noindex removes pages it already holds.

Everything below comes from Exa's crawler page, which showed a last-updated date of 13 November 2026 when we read it, and from Exa's API documentation. For the general mechanics of signed requests, see Web Bot Auth explained; for the other crawlers you will meet in logs, see the AI crawler user agents list.

What Exa is and why its crawler matters

Exa's documentation describes the product as a web search API that takes a natural language query and returns ranked results with clean page content. Developers call its /search endpoint to find pages and its /contents endpoint to get text, highlights or summaries for a list of URLs. That makes Exa a search backend for whatever application calls it, and Exa's crawler page says ExaSearchBot is the robot that discovers and indexes the public pages in that index.

Exa's crawler page puts the purpose plainly: search and retrieval, so people and applications can find, retrieve and cite your content and be sent back to your site. In the terms of our training, search and user crawlers guide, ExaSearchBot belongs with the search crawlers. Exa's page does not say whether crawled content is used to train models, so do not assume either way.

Allowing ExaSearchBot only removes a barrier. Which applications use Exa, and whether any of them quote your page, depends on things outside your site.

The three Exa user agents

Exa documents three names. Only the first has a full user agent string on the crawler page at the time of writing.

AgentWhat Exa says it doesDocumented details
ExaSearchBotCrawls the public web to discover and index pagesUser agent string, robots.txt token, signed requests
ExaUserFetches pages on a user's request, rather than by crawlingName only
ExaSearchBot-ImagesCrawls imagesName only

The ExaSearchBot string is:

Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)

Three things are not documented on the crawler page, so treat them as unknown rather than guessing:

  • the full user agent strings for ExaUser and ExaSearchBot-Images;
  • whether ExaUser and ExaSearchBot-Images read robots.txt, and under which token;
  • whether those two agents sign their requests the way ExaSearchBot does.

Exa's API does have a feature that fits the ExaUser description. Its livecrawling page explains a maxAgeHours setting: a positive number uses cached content newer than that many hours and fetches the page fresh otherwise, 0 asks for content fetched within the last 60 seconds, and -1 uses only the cache. The page does not say which user agent those fresh fetches use, so we cannot confirm they arrive as ExaUser.

How ExaSearchBot reads your robots.txt

Exa says ExaSearchBot respects the Robots Exclusion Protocol and chooses its group in this order:

  1. A group addressed to ExaSearchBot.
  2. If there is none, the rules you have set for major search engines.
  3. Otherwise, the * group.

Step 2 is unusual and worth understanding. Under RFC 9309, a crawler with no matching group falls back to *. Exa instead borrows the rules you wrote for search engines first. Its page does not list which search engines count, so think of groups for Googlebot and Bingbot as likely candidates without relying on it.

Here is why that matters. Take this file:

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /

The site owner probably meant "only Google". Exa's documented order means ExaSearchBot may follow the Googlebot group and crawl the whole site. The reverse happens too: if your Googlebot group disallows a section, ExaSearchBot may skip it even though your * group allows it.

If you want a clear answer, give Exa its own group. Exa says this takes precedence. To set your own rules:

User-agent: ExaSearchBot
Disallow: /private/

To block it from the whole site:

User-agent: ExaSearchBot
Disallow: /

A crawler follows only the group that matches it, not that group plus *, so an ExaSearchBot group must repeat any rules from other groups that you also want it to follow. Our post on robots.txt groups covers that merging behaviour. Exa does not document a separate token for images, so blocking ExaSearchBot-Images by name is not something you can rely on at the time of writing.

How to verify that a request really comes from Exa

Exa does not publish an IP list. It verifies by signature instead. According to its crawler page, every ExaSearchBot request is signed with HTTP Message Signatures (RFC 9421) following the Web Bot Auth scheme, and carries Signature, Signature-Input and a Signature-Agent header identifying Exa. Exa says that on Cloudflare, Akamai and other providers that support Web Bot Auth, the crawler is validated at the edge with nothing for you to configure.

The public keys live at Exa's key directory:

https://crawler.exa.ai/.well-known/http-message-signatures-directory

When we fetched it on 15 November 2026, it returned a JSON Web Key Set with one Ed25519 key, the media type for these directories and a five-minute cache lifetime:

http
HTTP/2 200
cache-control: max-age=300
content-type: application/http-message-signatures-directory+json
signature-input: binding0=("@authority";req);created=1794752184;keyid="sDCp...SIA";alg="ed25519";expires=1794752484;tag="http-message-signatures-directory"
json
{"keys":[{"kid":"sDCpsJpgItYcmF_KhZozOUGIGL6a49yxE3f0yQR5SIA","kty":"OKP","crv":"Ed25519","x":"f1UrAERwBTuaxqbngzcny0_rm7JycUzgbb0bUUvh0yw"}]}

The directory response is itself signed, so a verifier can check the key list came from Exa. Keys rotate, so fetch the directory and respect its cache header rather than hard-coding the key above. You can check it yourself:

bash
curl -si https://crawler.exa.ai/.well-known/http-message-signatures-directory

To confirm a request in your logs is genuine:

  1. Make sure the signature headers reach your origin. A CDN may verify and strip them, in which case its bot or security log is the evidence. Web Bot Auth explained shows an nginx log format for these headers.
  2. Check that Signature-Agent names Exa, then match the keyid in Signature-Input to a kid in Exa's directory.
  3. Verify the signature per RFC 9421 and check that created and expires make it currently valid.

Exa's point is that this check holds whatever IP address the request came from. A request that claims to be ExaSearchBot but carries no valid signature does not match what Exa documents for its crawler. For checking crawlers that do publish IP lists, see how to verify real AI crawlers.

How to remove pages from Exa's index

Blocking in robots.txt stops future crawling. It does not, by itself, remove a page Exa already indexed, and Exa says so directly: robots.txt controls crawling, not indexing. To remove a page, serve a noindex signal:

html
<meta name="robots" content="noindex">

or the equivalent HTTP header, which is the usual choice for PDFs and other non-HTML files:

http
X-Robots-Tag: noindex

Exa says ExaSearchBot drops the page after it next re-fetches it. That has a practical consequence: the crawler must still be allowed to fetch the page to see the noindex. If you disallow the URL in robots.txt at the same time, the crawler cannot read the tag. Add noindex first, let it be re-fetched, and only then block the path if you also want crawling to stop. Exa does not say how soon a re-fetch happens.

Exa's page also lists how the crawler behaves: it checks robots.txt before fetching, rate-limits per site, and fetches public content only, without trying to get past logins, paywalls or CAPTCHAs and without submitting forms. For anything else, it gives support@exa.ai as the contact and asks you, if possible, to write from an address on the domain in question, which helps them verify you.

Should you allow ExaSearchBot?

It depends on whether you want to be findable through tools built on Exa's search API.

GoalWhat to do
Be findable in apps that search through ExaAllow ExaSearchBot, and check your WAF does not challenge it
Keep private sections outAdd an ExaSearchBot group with Disallow lines for those paths
Not be crawled at allUser-agent: ExaSearchBot with Disallow: /
Remove pages already indexedServe noindex, keep the page fetchable until it is re-crawled
Avoid inheriting your search engine rulesWrite an explicit ExaSearchBot group

If you use a CDN or WAF that verifies Web Bot Auth, allowing verified traffic is the cleaner rule than allowing the user agent string, which anyone can copy. The Brave Search crawler guide covers another independent search index that AI tools draw on, if you are reviewing these together.

Frequently asked questions

Is ExaSearchBot an AI training crawler?

Exa describes its purpose as search and retrieval, building an index that people and applications query. Its crawler page does not say whether crawled content is used for model training.

What happens if I have no ExaSearchBot group in robots.txt?

Exa says the crawler then follows the rules you set for major search engines, and if there are none, the * group. Add an explicit ExaSearchBot group if you want different rules.

Does Exa publish IP ranges for ExaSearchBot?

Its crawler page does not list IP ranges. It verifies by signature instead, with Ed25519 public keys at https://crawler.exa.ai/.well-known/http-message-signatures-directory.

Will blocking ExaSearchBot in robots.txt remove my pages from Exa?

Not on its own. Exa says robots.txt controls crawling, not indexing, so serve noindex and let the crawler re-fetch the page to have it dropped.

Can I block ExaUser separately?

Exa names ExaUser as the agent for pages fetched on a user's request, but at the time of writing it does not document a user agent string or robots.txt behaviour for it, so a robots.txt rule for that name may not take effect.

Check what ExaSearchBot can reach

After adding or changing an ExaSearchBot group, make sure the rest of your stack agrees with it: robots.txt, WAF rules and noindex tags can each contradict the others. The free check tests one page against the main AI crawlers with no account needed. If you want every crawler group, edge rule and page-level signal reviewed across the site, an audit covers up to 500 pages with evidence and prioritized fixes.

Filed under AI crawlers. Topics: ai-crawlers, robots-txt, web-bot-auth, server-logs