Skip to content

Link preview bots: Slackbot, Twitterbot, LinkedInBot and blocking

Link preview bots like facebookexternalhit, Slackbot and Twitterbot are not AI crawlers. Here is what breaks when a firewall or robots.txt blocks them.

By Crawlito · 7 min read

In short

  • Link preview bots such as facebookexternalhit, Slackbot-LinkExpanding, Twitterbot, WhatsApp and Discordbot fetch one shared URL to build a card; their operators describe them as preview fetchers, not AI training or AI search crawlers.
  • Slack says it does not honor robots.txt for link expansion, while X says Twitterbot follows robots.txt and shows no card if the page is blocked and no image if the image URL is blocked.
  • Meta says Open Graph properties must appear in the first 1 MB of a page, and WhatsApp says the head must appear within the first 300 KB of the HTML.
  • A bot challenge or a 403 from a firewall breaks previews, because these fetchers do not run JavaScript; Discord shows no preview for any non-2xx response.
On this page

Link preview bots are the fetchers that build the title, description and image card when someone shares your URL in Facebook, Slack, X, LinkedIn, WhatsApp or Discord. They are not AI crawlers: each fetches one URL because a person shared it, and the operators document them as preview tools, separate from their AI agents. If a firewall rule or a robots.txt line blocks them, the usual result is a bare link, a card without an image, or a stale title.

This matters for AI readiness audits because the same bot rules that are written to keep AI scrapers out often catch these fetchers too. Below is what each one is documented to do, what breaks, and how to check and fix it without opening your site to everything.

A preview bot runs when a link is posted. Slack describes its link robot as acting on behalf of a person and says it is not a crawler because it does not follow links. Meta's web crawler documentation says facebookexternalhit fetches shared links to gather the title, description and thumbnail, and lists it separately from meta-externalagent (training and product indexing) and meta-webindexer (Meta AI search).

That separation is the key point. Blocking facebookexternalhit does not opt you out of Meta's AI training, and blocking meta-externalagent does not break Facebook previews. The same is true at other companies: the preview fetcher and the AI crawler use different user agents and are controlled separately. Our guide to Meta's AI crawlers covers the Meta side in detail.

These are the tokens the operators document, at the time of writing. Treat any detail not in the source column as unconfirmed.

PlatformUser agent tokenrobots.txtDocumented limits and cachingSource
Facebook, Instagram, Messengerfacebookexternalhit/1.1Generally followed; Meta says it may bypass robots.txt for security or integrity checksOpen Graph tags must be in the first 1 MB; sends a Range header; content must be crawlable within a few secondsMeta
SlackSlackbot-LinkExpanding 1.0, images via Slack-ImgProxyNot honoredFetches as little as it can with Range; caches for about 30 minutesSlack
XTwitterbotFollowedCaches card content for 7 days after a link is postedX Cards troubleshooting
WhatsAppWhatsApp/2.x.x.x A, I or NNot statedHead must be in the first 300 KB; og:image under 600 KB and at least 300 px wideWhatsApp
DiscordDiscordbotNot stated10 second total fetch, 5 redirects, previews cached for about 30 minutes, no JavaScriptDiscord
LinkedInLinkedInBot is commonly seen in logsNot statedImage at least 1200 x 627 px, under 5 MBLinkedIn

LinkedIn's help page does not name its user agent or describe robots.txt handling, so the LinkedInBot token above comes from server logs rather than documentation. X's Cards guides are older pages from the Twitter era, so check them against what you see in your own logs.

What breaks when a firewall or bot rule blocks them?

Most broken previews on otherwise healthy sites come from bot protection, not from missing tags.

Challenge pages

A JavaScript challenge or CAPTCHA asks the client to run code. Preview fetchers do not. Discord's documentation states that tags added by client-side scripts are never seen because no code is executed, and that no preview is shown if the response is not a 2xx. A challenge usually comes back as a 403 or 503 with a placeholder page, so the fetcher gets no tags and the card is empty. Our post on WAF challenge pages shows how to recognise one in a raw response.

Blocked images on another host

The page may load while the image does not. If og:image points to a CDN or storage bucket with hotlink protection, or to a host behind a different firewall policy, the fetcher gets a 403 for the image and shows a card without a thumbnail. LinkedIn's help page says that if an image meets its requirements but does not appear, your website may be blocking LinkedIn from pulling it.

Rate limits and slow responses

Meta says content must be crawlable within a few seconds, and Discord allows 10 seconds for the whole fetch, including measuring images. A rate limit that returns 429, or an origin that is slow on uncached pages, produces the same empty card. When a link is posted in a busy channel or a large group, several fetchers may arrive at once.

Heavy pages

Meta reads Open Graph properties only from the first 1 MB, and WhatsApp needs the <head> within the first 300 KB. A head stuffed with inline scripts, styles or base64 images can push the tags past those limits.

What breaks when robots.txt blocks them?

robots.txt only matters for fetchers that read it, and the operators disagree. Slack says it does not honor robots.txt for link expansion, so a Disallow will not stop Slack previews. X says Twitterbot follows robots.txt: if the page is disallowed, no card is shown, and if the image URL is disallowed, no thumbnail is shown.

The common accident is a catch-all group. A file like this blocks Twitterbot from every image even though the page itself is allowed:

User-agent: *
Disallow: /wp-content/uploads/

If you need that rule for other bots, give the preview fetchers their own group. A crawler that matches a named group ignores the * group, so repeat any private paths you still want excluded:

User-agent: Twitterbot
User-agent: facebookexternalhit
Disallow: /account/

User-agent: *
Disallow: /wp-content/uploads/
Disallow: /account/

robots.txt groups explained covers how a crawler picks its group and why rules are not merged.

How to check whether preview bots can read your page

Start with the raw response. Fetch the page with a preview fetcher's user agent and look at the status code and the tags:

bash
curl -s -o page.html -w '%{http_code} %{size_download}\n' \
  -A 'facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)' \
  https://www.example.com/blog/launch-post
grep -io '<meta[^>]*og:[^>]*>' page.html

Repeat with Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots) and Twitterbot/1.0. Then fetch the image URL from og:image the same way and confirm a 200 with an image/ content type.

A spoofed user agent tests your application logic, not your firewall's view of the real bot. Your request comes from your own IP address, so a rule that checks network identity can treat the real fetcher differently. Two places give you the real picture:

  1. Your logs. Search access logs or your CDN's firewall events for the tokens in the table and note the status codes. A run of 403s for facebookexternalhit or Twitterbot is the answer. Verifying real AI crawlers shows how to confirm a request's origin with reverse DNS.
  2. The platform's own tools. Facebook's Sharing Debugger and LinkedIn's Post Inspector fetch from the platform's own network and show what they received.

Work from the narrowest change outward.

  1. Exempt the preview fetchers from challenges. In your CDN or WAF, add a skip or allow rule for the specific user agents, ideally combined with the provider's verified-bot signal where it covers them. Meta says allowlisting by IP address is more secure than allowlisting by user agent string, because anyone can send a user agent.
  2. Allow the image host too. Apply the same exception to the domain that serves og:image, and check hotlink protection.
  3. Put Open Graph tags early in the head. Move them above large inline scripts and styles so they fall inside Meta's 1 MB and WhatsApp's 300 KB windows. Render them on the server; a fetcher that runs no JavaScript cannot see tags injected later. See raw versus rendered HTML.
  4. Handle Range requests. Meta and Slack send a Range header. Meta says your server should return the bytes requested or ignore the header; an error on ranged requests breaks previews.
  5. Refresh the cache after the fix. Use the Sharing Debugger for Meta and the Post Inspector for LinkedIn. Slack and Discord cache for about 30 minutes, and X for 7 days after posting. Discord suggests sharing the URL with a new query string to see changes sooner.

For the tags themselves, our Open Graph tags guide lists the minimum set and the image sizes each platform documents.

Frequently asked questions

The operators do not describe them that way. Meta lists facebookexternalhit as the preview fetcher and uses separate agents for AI training and Meta AI search, and Slack describes its link robot as acting for a person who posted the link.

Will blocking facebookexternalhit stop Meta from training on my site?

No. Meta's training and product indexing crawler is meta-externalagent, which has its own robots.txt token. Blocking facebookexternalhit only breaks previews of your links on Facebook, Instagram and Messenger.

No. Slack says it does not currently honor robots.txt files for link expansion. To stop previews you would need a server or firewall rule, which also stops colleagues from seeing a useful card.

Why does my preview still show the old image after I fixed it?

The platform cached the earlier result. Re-scrape with the platform's own tool where one exists, purge your CDN first, and allow for the documented cache periods, such as about 30 minutes for Slack and Discord.

Check your own pages

The free check fetches a page the way an automated client does and reports challenge pages, blocked status codes and robots.txt rules, which are the same problems that break link previews. If you manage several templates or sites, an audit tests them together and lists the fixes in priority order.

Filed under AI crawlers. Topics: ai-crawlers, bot-protection, robots-txt, waf, metadata