Diffbot and Diffbot-User: what they crawl and how to block
What the Diffbot crawler and Diffbot-User do, how Diffbot reads robots.txt (no Allow support), and how to allow or block them with robots.txt or your server.
By Crawlito · 7 min read
In short
Diffbot documents two user agent tokens: Diffbot, a proactive crawler for its Knowledge Graph and web search, and Diffbot-User, for requests made on behalf of a person using Diffbot software.
Diffbot says its crawls follow robots.txt Disallow and Crawl-delay by default, but its Crawl docs state that Crawl does not adhere to the Allow directive.
Because Allow is ignored, a Diffbot group with Disallow: / and an Allow line for one folder blocks the whole site for Diffbot, not just the rest of it.
Diffbot's docs say robots.txt can be overridden for sites that have a partnership or agreement, and Crawl API jobs expose an obeyRobots parameter that turns compliance off.
Customers running their own crawls can set a custom User-Agent, so Diffbot-powered traffic does not always contain the word Diffbot.
The Diffbot crawler collects public web pages for Diffbot's Knowledge Graph and web search, and Diffbot-User fetches a page when a person asks Diffbot software to. Diffbot says its crawls follow robots.txt by default, including Disallow and Crawl-delay, but its Crawl documentation says it does not adhere to the Allow directive. To block it, a User-agent: Diffbot group with Disallow: / is the documented route; for a hard stop, match the token at your server or firewall as well.
This post covers what Diffbot's own documentation says, the parts that differ from how most crawlers read robots.txt, and copy-ready rules for both choices. For the general question of whether AI crawlers obey robots.txt, see do AI crawlers respect robots.txt; for the full list of tokens from other operators, see the AI crawler user agents list.
What the Diffbot crawler and Diffbot-User are
Diffbot's robots.txt FAQ names two user agents and describes their purpose. At the time of writing:
Token
What Diffbot says it does
Diffbot's own comparison
Category
Diffbot
General, proactive crawling to build a general search engine; sites can be discovered and cited in the Diffbot Knowledge Graph and web search results
Googlebot or Bingbot
Search and indexing
Diffbot-User
Requests made on behalf of a human user browsing a URL with Diffbot software
Mozilla or AppleWebKit, that is, a browser
User-triggered
The same page states that Diffbot does not crawl pages to train generative AI foundation models, and that it reduces server load with caching, compression, conditional GETs and predictive scheduling. Those are Diffbot's statements about its own behaviour; there is no independent way to verify the training claim from your side.
The split maps onto the categories in training, search and user-triggered crawlers. Diffbot behaves like a search crawler that runs on its own schedule. Diffbot-User is closer to ChatGPT-User or Perplexity-User: it fetches a URL because someone asked for it.
There is also a third source of Diffbot traffic that carries neither token reliably: crawls that Diffbot's customers run with its hosted tools. That is covered below.
How Diffbot reads robots.txt
Diffbot's Crawl overview says that by default Crawl follows a site's robots.txt, including the disallow and crawl-delay directives. Three details make it behave differently from a crawler that follows RFC 9309 to the letter.
Allow lines are ignored
The Crawl overview says plainly that Crawl does not adhere to the Allow directive. RFC 9309 defines allow as a standard rule and says the most specific match wins, so most crawlers read this group as "only the docs folder":
User-agent: Diffbot
Disallow: /
Allow: /docs/
Going by its documentation, Diffbot ignores the Allow line and applies only Disallow: /, so it is blocked from the whole site, /docs/ included. If you want Diffbot in one folder and out of the rest, list the folders to exclude instead:
The reverse case matters too. A broad Disallow in a shared group with a narrower Allow meant to reopen a path will not reopen it for Diffbot.
Crawl-delay is honoured
Crawl-delay is not part of RFC 9309, and support varies: Google, for example, says Googlebot does not support it. Diffbot names it as one of the directives Crawl follows by default, so a line such as Crawl-delay: 10 in a Diffbot group is a reasonable way to slow it down without blocking it. The page does not say how the value is applied. For how other crawlers treat the directive, see Crawl-delay and AI crawlers.
robots.txt can be overridden by agreement
Both the FAQ and the Crawl overview say robots.txt instructions can be ignored or overridden for sites that have a partnership or agreement with the party running the crawl, and describe this as often quicker than waiting for the site to change its file. The Crawl API exposes a related job parameter: the create a crawl reference lists obeyRobots with a default of 1, and says to pass obeyRobots=0 to ignore a site's robots.txt.
So the default is compliance, but compliance is a setting on a crawl job, not a fixed property of every request that comes from Diffbot's infrastructure. If you have not agreed to anything with a Diffbot customer, nothing in the documentation tells you how that setting is policed for jobs aimed at your site. That gap is a reason to add server-level enforcement if a block really matters to you.
Why Diffbot traffic may not say Diffbot
Diffbot's customers use its Extract and Crawlbot products to fetch URLs and crawl sites of their own choosing. The robots.txt FAQ recommends that those customers set a User-Agent that represents their organisation, and keep Crawlbot's robots.txt option switched on, which it is by default.
Diffbot's custom user agents guide, written for Extract custom APIs, shows how this looks in practice: its test request arrives with a third party's bot string, not a Diffbot one, and the guide then shows how to add a custom word to it so a site owner can whitelist that customer's requests. The Crawl API reference adds a customHeaders parameter for headers sent while crawling, and a useProxies option that forces the use of proxy IPs.
Two consequences follow:
A rule that matches the string Diffbot will catch Diffbot's own crawler, but not necessarily a customer crawl that uses a custom User-Agent.
Which robots.txt group a customer crawl with a custom User-Agent reads is not documented on the pages cited here. Do not assume a User-agent: Diffbot group covers it.
The documented route is a group that names the token. RFC 9309 matches the product token case-insensitively, and Diffbot and Diffbot-User are different tokens, so name both if you want to cover both:
Diffbot's FAQ refers to naming the appropriate Diffbot user agents in robots.txt. It does not say in so many words whether Diffbot-User, as a user-triggered fetcher, checks robots.txt on every request. Other operators document that their user-triggered fetchers may ignore the file, so treat the Diffbot-User line as a stated preference and enforce at the server if you need certainty.
To slow Diffbot down instead of blocking it, keep it allowed and add a delay:
Diffbot's Crawl overview gives the allow form as a group with an empty Disallow:
User-agent: Diffbot
Disallow:
That matters when your file has a strict User-agent: * group. Under RFC 9309, a crawler that finds a group naming its token uses that group and ignores the * group, so this group replaces the wildcard rules for Diffbot. Do not write Allow: / here and rely on it: Diffbot says it ignores Allow, and an empty Disallow means the same thing to every crawler.
How to block Diffbot at the server
robots.txt asks; the server decides. RFC 9309 itself says the rules are not a form of access authorization. To refuse requests, match the user agent at your web server or edge. A case-insensitive match on diffbot covers both documented tokens.
For nginx, inside the server block:
nginx
if ($http_user_agent ~* "diffbot") {
return 403;
}
For Apache in .htaccess with mod_rewrite enabled:
apache
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} diffbot [NC]
RewriteRule .* - [F,L]
Leave /robots.txt reachable if you want honest crawlers to keep reading your rules. Our guide to blocking AI crawlers in nginx, Apache and .htaccess covers exceptions, testing and the order of rules in more detail.
Two limits apply. A user agent match stops only traffic that announces itself, which excludes customer crawls with custom User-Agents. And the pages cited here do not publish IP ranges for Diffbot's crawlers, so an IP-based rule or a reverse DNS check would rest on undocumented assumptions.
How to check what Diffbot is doing on your site
Search your access logs for the token and see which paths it requests and how often:
Look for disallowed paths requested after your change, and note the status code your server returned.
Test your rules against Diffbot's parsing. If your Diffbot group relies on an Allow line, rewrite it with Disallow lines only, then check the result in a robots.txt tester, remembering that the tester will apply Allow even though Diffbot does not.
Frequently asked questions
Is Diffbot an AI training crawler?
Diffbot says it does not crawl pages to train generative AI foundation models. It describes the Diffbot crawler as building a general search engine and its Knowledge Graph, comparable to Googlebot or Bingbot.
Does Diffbot follow Allow rules in robots.txt?
No. Diffbot's Crawl overview says Crawl does not adhere to the Allow directive, so write Diffbot groups with Disallow lines only.
Does Diffbot respect Crawl-delay?
Yes, according to Diffbot. Its documentation lists crawl-delay alongside disallow as directives its crawls follow by default.
Can Diffbot ignore my robots.txt?
Its documentation says robots.txt can be ignored or overridden for sites with a partnership or agreement, and Crawlbot jobs have an obeyRobots setting that turns compliance off. If a block must hold, enforce it at the server too.
Does blocking Diffbot affect ChatGPT, Claude or Perplexity?
A User-agent: Diffbot group applies only to crawlers that match that token. OpenAI, Anthropic and Perplexity document their own crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, which follow the groups that name them or your * group.
Check your robots.txt rules
Before you rely on a Diffbot group, confirm that the rest of your file still says what you intend for the crawlers you do want. The free check reads one page the way AI crawlers do and shows which robots.txt rules apply to each named crawler, with no account needed.
Filed under AI crawlers. Topics: · ai-crawlers, robots-txt, server-logs, ai-training
Do AI crawlers respect robots.txt? What OpenAI, Anthropic and Perplexity document, which user-triggered fetchers are exempt, and what the research found.
A reference list of AI crawler user agents from OpenAI, Anthropic, Perplexity, Google, Apple and others, with each bot's purpose, robots.txt token and docs.
Identify an unknown bot user agent: read its product token and info URL, check operator docs, verify IPs or signatures, then look it up in RDAP and Cloudflare Radar.