Meta-ExternalAgent and Meta-WebIndexer: Meta's AI crawlers
Meta-ExternalAgent crawls for AI training and product indexing, Meta-WebIndexer serves Meta AI search. See what each does and which Meta fetchers skip robots.txt.
By Crawlito · 7 min read
In short
- Meta documents Meta-ExternalAgent as crawling for uses such as training foundation AI models or improving products by indexing content, and Meta-WebIndexer as crawling to improve Meta AI search results.
- Meta says allowing Meta-WebIndexer in robots.txt helps it cite and link to your content in Meta AI's responses; that is a documented benefit, not a promise of citations.
- Meta does not document whether Meta-ExternalAgent data feeds Meta AI search, so you cannot assume that blocking it changes how you appear there.
- Meta says Meta-ExternalFetcher may bypass robots.txt, and facebookexternalhit might bypass it for security or integrity checks such as malware scans.
- At the time of writing, Meta's crawler page lists no IP ranges, so a user agent string in your logs is a claim, not proof.
On this page
- What are Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalFetcher?
- Does blocking Meta-ExternalAgent remove you from Meta AI search?
- How to allow Meta-WebIndexer and block training-style crawling in robots.txt
- Which Meta fetchers can bypass robots.txt?
- Why you should not block facebookexternalhit
- How to find Meta crawlers in your server logs
- How to check what Meta's bots get from your pages
- Frequently asked questions
- Check your own page
Meta-ExternalAgent is the crawler Meta describes as collecting content for training foundation AI models and for improving products by indexing content directly. Meta-WebIndexer is a separate crawler that Meta says helps improve Meta AI search results. They are different robots.txt tokens, so you can treat them differently: block the first, allow the second. Meta's own page does not say that blocking Meta-ExternalAgent removes you from Meta AI search, and it does not say it keeps you in.
Everything below comes from Meta's web crawlers page as it stood at the time of writing. Check it before you ship a rule, because Meta can change it. Nothing here promises that Meta AI will cite your pages. Allowing a crawler only removes a technical barrier.
What are Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalFetcher?
Meta lists several crawlers on one page. These are the ones that matter for AI visibility, plus two neighbours you will see in the same logs.
| Token | What Meta says it does | robots.txt |
|---|---|---|
Meta-ExternalAgent | Crawls the web for uses such as training foundation AI models or improving products by indexing content directly | No bypass stated; Meta's example uses a Disallow rule |
Meta-WebIndexer | Navigates the web to improve Meta AI search result quality | No bypass stated |
Meta-ExternalFetcher | Fetches individual links at a user's request, for product functions such as evaluating and improving agentic AI | Meta says it "may bypass robots.txt rules" |
Meta-ExternalAds | Crawls the web for uses such as improving advertising and other business-related products | No bypass stated; Meta's example uses a Disallow rule |
facebookexternalhit | Fetches a page when someone shares its link on Facebook, Instagram or Messenger, to read the title, description and thumbnail | Meta says it might bypass robots.txt for security or integrity checks |
Meta documents user agent strings for the three AI tokens in this form. Each also appears with a documentation link in parentheses after the version:
meta-externalagent/1.1
meta-webindexer/1.1
meta-externalfetcher/1.1Match on the token and ignore the version number, which may change. If you want the same split for other vendors, training vs search vs user crawlers maps them side by side, and the AI crawler user agents list has the full table.
Does blocking Meta-ExternalAgent remove you from Meta AI search?
Meta does not say so. Its page ties Meta AI search to Meta-WebIndexer, and says that allowing Meta-WebIndexer in your robots.txt "helps us cite and link to your content in Meta AI's responses." The same page describes Meta-ExternalAgent as a training and product-indexing crawler. It does not say whether pages fetched by Meta-ExternalAgent are used in Meta AI search answers.
So the documented position is:
- Allow Meta-WebIndexer if you want Meta AI to be able to cite and link to you.
- Block Meta-ExternalAgent if you want to opt out of the uses Meta lists for it. That wording is broader than training alone, because it includes indexing content to improve products.
- Do not assume the two are linked. Meta does not document that they are, in either direction.
If you want evidence for your own site, watch your logs after a change (see below) and treat any effect on Meta AI as unconfirmed. This is the same logic as allowing AI search while blocking AI training for other vendors, with one difference: Meta's training token also covers product indexing, so blocking it is a slightly wider decision.
How to allow Meta-WebIndexer and block training-style crawling in robots.txt
Add a group for each token. A crawler follows the group that names it, so a named group for Meta-WebIndexer applies instead of any User-agent: * rules, as RFC 9309 describes. Token matching is case-insensitive under that RFC, so Meta-WebIndexer and meta-webindexer are the same.
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Meta-WebIndexer
Allow: /
User-agent: *
Allow: /If you only want to keep some paths private, narrow the rule instead. Meta's own example for Meta-ExternalAgent uses Allow: / plus Disallow: /private/:
User-agent: Meta-ExternalAgent
Allow: /
Disallow: /private/Two cautions:
- A blanket
Disallow: /under*blocks Meta-WebIndexer too, unless you add a named group for it, so check any generic "block everything" template before you copy it. The group-matching rules are the ones in RFC 9309. - Meta says changes can take up to 24 hours, because crawlers may cache robots.txt. RFC 9309 says a crawler should not use a cached copy for more than 24 hours unless the file is unreachable.
For a full template that includes other vendors, see robots.txt for AI crawlers.
Which Meta fetchers can bypass robots.txt?
Two, according to Meta's wording:
- Meta-ExternalFetcher "may bypass robots.txt rules" because it performs fetches that a user requested. A Disallow for it is not a dependable switch. Meta does not document another way to stop it, so if you must keep it out, you would need a server-side rule such as a firewall or WAF rule on the user agent.
- facebookexternalhit "might bypass robots.txt when performing security or integrity checks, such as checking for malware or malicious content."
For Meta-ExternalAgent, Meta-WebIndexer and Meta-ExternalAds, the page states no bypass. That is not the same as a guarantee that they obey every rule in every case, and the page does not describe how Meta checks compliance. Robots.txt is a request to crawlers, not access control: RFC 9309 says its rules "are not a form of access authorization."
Why you should not block facebookexternalhit
facebookexternalhit is not an AI training or AI search crawler. Meta says it fetches pages shared on its apps so that it can read and show the title, description and thumbnail. If you block it at the firewall or return errors to it, a shared link may show up without a usable preview. That hurts people sharing your content on Facebook, Instagram or Messenger, and it has nothing to do with Meta AI.
Meta lists these requirements for it:
- Your server must support gzip and deflate encodings.
- Open Graph properties must appear within the first 1 MB of the page, or they are cut off.
- The page must be crawlable within a few seconds.
- The server must handle the
Rangeheader correctly, or ignore it.
Meta also suggests allowlisting either the user agent strings or the IP addresses used by the crawler, and calls IP addresses "more secure." At the time of writing the page does not list those addresses. If a bot-protection rule is returning 403 or a challenge page to this crawler, your link previews can break.
How to find Meta crawlers in your server logs
Search for the tokens, case-insensitively, and ignore version numbers.
grep -i -E "meta-externalagent|meta-webindexer|meta-externalfetcher|meta-externalads|facebookexternalhit" access.log \
| awk '{print $1, $7, $9}' | tail -20To count requests per Meta token:
grep -i -o -E "meta-externalagent|meta-webindexer|meta-externalfetcher|meta-externalads|facebookexternalhit" access.log \
| tr 'A-Z' 'a-z' | sort | uniq -c | sort -rnThen read the results with these points in mind:
- A user agent is only a claim. Anyone can send it. Meta's page lists no IP ranges for the AI tokens, so verify against your CDN's verified-bot data if it has any, and treat unverified traffic with caution. AI crawlers in server logs shows how to separate real crawlers from spoofed ones.
- Requests after a Disallow are not always a bug. For Meta-ExternalFetcher they are expected, per Meta. For the others, wait out the 24-hour caching window before concluding anything.
- Check status codes. A run of 403, 429 or 5xx responses to a Meta token suggests a firewall or rate-limit rule is in the way.
How to check what Meta's bots get from your pages
You can approximate a request from your own machine. Use the user agent strings Meta documents:
curl -s -o /dev/null -D - -A "meta-webindexer/1.1" https://example.com/
curl -s -A "meta-webindexer/1.1" https://example.com/pricing | grep -c "Plans start at"The first command shows the status and headers. The second tells you whether a sentence you can see in the browser is present in the raw HTML. A count of 0 means it only appears after JavaScript runs. Meta's page does not say whether Meta-WebIndexer or Meta-ExternalAgent run JavaScript, so the safe assumption is that main content belongs in the initial HTML.
The limit of the test: it comes from your IP address, not Meta's, so a firewall rule based on IP or reputation can treat the real crawler differently. Your CDN's security log is the better evidence. For robots.txt rules, testing robots.txt for AI crawlers shows how to confirm which group applies to each token.
Frequently asked questions
Is Meta-ExternalAgent the same as Meta-WebIndexer?
No. Meta describes Meta-ExternalAgent as crawling for uses such as training foundation AI models or improving products by indexing content, and Meta-WebIndexer as crawling to improve Meta AI search results. They have separate robots.txt tokens.
Should I block Meta-ExternalAgent?
That depends on whether you want to opt out of the uses Meta lists: AI model training and product indexing. Blocking it is a documented, supported choice, and Meta's own example shows a Disallow rule. Keep in mind it does not stop Meta-ExternalFetcher or facebookexternalhit.
Does Meta-ExternalFetcher respect robots.txt?
Meta says it may bypass robots.txt rules because it fetches pages that a user asked for. Do not rely on a robots.txt rule to stop it.
Will allowing Meta-WebIndexer get me cited in Meta AI?
No promise is possible. Meta says allowing it helps Meta cite and link to your content in Meta AI's responses, but whether a page is cited depends on factors Meta does not publish.
How long do robots.txt changes take to apply to Meta's crawlers?
Meta says to allow up to 24 hours, because crawlers may cache robots.txt. Test again after a day.
Check your own page
The free check reads one page the way AI crawlers do, with no account needed, and shows the robots.txt rules that apply to the named AI crawlers, plus the robots meta tag and whether the main text is in the raw HTML. If you want it across a whole site, an audit covers up to 500 pages with evidence and prioritized fixes.
Filed under AI crawlers. Topics: meta, ai-crawlers, robots-txt, server-logs