Skip to content

DuckAssistBot: how DuckDuckGo's AI answers fetch your pages

DuckAssistBot is DuckDuckGo's crawler for cited AI answers. See what it does, whether it trains models, how to block it in robots.txt and verify its IPs.

By Crawlito · 7 min read

In short

  • DuckDuckGo describes DuckAssistBot as its crawler that fetches pages in real time to support AI-assisted answers that cite their sources.
  • DuckDuckGo says the data DuckAssistBot collects is not used to train AI models.
  • You opt out by disallowing the DuckAssistBot user agent in robots.txt; DuckDuckGo says the change takes effect after 72 hours.
  • DuckDuckGo says opting out does not affect organic search rankings or whether your site appears in DuckDuckGo results.
  • Verify a request by checking its IP against the list DuckDuckGo publishes at duckduckgo.com/duckassistbot.json, because a user agent string alone proves nothing.
On this page

DuckAssistBot is the crawler DuckDuckGo uses to fetch web pages in real time for its AI-assisted answers, which cite their sources. DuckDuckGo says the data is not used to train AI models. You can opt out by disallowing DuckAssistBot in robots.txt, and DuckDuckGo says the change takes effect after 72 hours. This guide covers what the bot is, how to block or allow it, how to confirm a request is genuine and what to look for in your logs.

Everything here comes from DuckDuckGo's own DuckAssistBot help page and its published IP list, read at the time of writing (11 November 2026). Where DuckDuckGo says nothing, this post says so rather than guessing.

What is DuckAssistBot?

DuckDuckGo's help page describes DuckAssistBot as a web crawler for DuckDuckGo Search that crawls pages in real time to support AI-assisted answers, and says those answers cite their sources. That is the whole official description. It is one of several agents covered in the AI crawler user agents list.

The user agent is listed on the help page as:

DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)

The version number is the one shown at the time of writing and can change. In robots.txt and in your log filters, use the token DuckAssistBot and ignore the version.

It is separate from DuckDuckBot, which DuckDuckGo's DuckDuckBot page describes as the crawler that keeps improving search results and the search experience. That page lists its own user agent (DuckDuckBot/1.1), its own IP list at duckduckgo.com/duckduckbot.json, and says it follows the robots.txt standard. The two agents have different tokens, so a rule for one does not apply to the other.

What DuckDuckGo does not say on the help page: whether a fetch happens because a specific person asked a question, how often the bot visits, how it treats crawl-delay, or how it treats Allow rules. Do not build a policy on any of those until DuckDuckGo documents them.

Does DuckAssistBot train AI models?

No, according to DuckDuckGo. The help page says the data DuckAssistBot collects is not used to train AI models. That statement is the operator's own and cannot be checked from outside your server, the same caveat that applies to every crawler's claim about training. Our guide to training, search and user-triggered crawlers explains why the purpose of a crawler matters when you decide what to block.

In practice this puts DuckAssistBot on the "fetch to answer" side of the line, not the "collect to train" side. If your reason for blocking AI bots is training, DuckDuckGo says this one is not doing it.

How do you allow or block DuckAssistBot in robots.txt?

DuckDuckGo's documented opt-out is to disallow the DuckAssistBot user agent in your domain's robots.txt. To opt out of your whole site:

User-agent: DuckAssistBot
Disallow: /

To keep it out of one section only:

User-agent: DuckAssistBot
Disallow: /members/

To allow it, which is the default when no rule mentions it and no * group blocks it, you can state that explicitly:

User-agent: DuckAssistBot
Allow: /

Group matching follows RFC 9309. Product tokens are matched without regard to case, and a crawler that finds a group naming it uses that group; the * group applies only when no group matches. So once you add a DuckAssistBot group, your User-agent: * rules stop applying to it. If you block * from /private/ and add a DuckAssistBot group that only allows /, DuckAssistBot is no longer told to stay out of /private/. Repeat the paths you want protected inside its group. robots.txt for AI crawlers covers group rules and longest-match in full.

Two cautions from the RFC apply here too. A robots.txt that returns a 5xx error means "unreachable", and crawlers must assume everything is disallowed. A 4xx such as 404 means "unavailable", and the crawler may access anything. And the RFC states plainly that these rules are not a form of access authorization, so robots.txt is a request to a crawler that chooses to honor it, not a lock. For content that must stay private, use authentication.

To confirm what your live file says to each AI token, follow how to test robots.txt against each AI crawler.

How long does an opt-out take to apply?

DuckDuckGo says an opt-out takes effect 72 hours after you update robots.txt, and that the crawler then stops crawling your site. That is longer than the delay OpenAI and Perplexity document for their search crawlers: OpenAI says about 24 hours, Perplexity says up to 24 hours.

Practical consequences:

  • Plan the change at least three days before anything depends on it.
  • Expect to keep seeing DuckAssistBot requests in your logs for up to three days after you deploy the rule. That is the documented delay, not evidence that the rule is broken.
  • If requests continue well past 72 hours, first confirm the live robots.txt really contains the rule (right host, right protocol, correct token), then check that the requests are real DuckAssistBot traffic using the IP list below.
  • If the problem persists, DuckDuckGo's help page says to email crawling@duckduckgo.com.

How do you verify DuckAssistBot with the published IP list?

Anyone can send DuckAssistBot in a user agent header, so the string proves nothing. DuckDuckGo publishes the IP addresses the bot uses at duckduckgo.com/duckassistbot.json. At the time of writing the file is a JSON object with a creationTime timestamp and a prefixes array whose entries each hold one ipv4Prefix value, written as a single address with a /32 mask.

This check needs only curl and Python 3. It also copes with ranges, in case DuckDuckGo adds any later:

bash
check_ddg() {
  curl -s https://duckduckgo.com/duckassistbot.json | python3 -c '
import json, sys, ipaddress
ip = ipaddress.ip_address(sys.argv[1])
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"), strict=False)
        for p in json.load(sys.stdin)["prefixes"]]
print("match" if any(ip in n for n in nets) else "no match")
' "$1"
}

check_ddg 203.0.113.7

Replace 203.0.113.7 with the client IP from your log line. match means the address is on DuckDuckGo's published list for DuckAssistBot. no match on a request that claims the name means it did not come from those addresses.

Two things to keep in mind. The list is a snapshot, so fetch it at check time instead of copying addresses into a firewall rule by hand. And if your server sits behind a CDN or load balancer, the first field in the log may be the proxy's address, so log the forwarded client IP first. The general method is in how to verify AI crawlers and spot fakes. DuckDuckGo's help page does not mention reverse DNS for DuckAssistBot, so use the IP list.

How to spot DuckAssistBot in server logs

Search for the token, not the full string, since the version can change:

bash
grep -i "DuckAssistBot" access.log | awk '{print $1, $4, $7, $9}' | tail -20

The fields printed are the client IP, timestamp, requested path and status code in the common log format; adjust the numbers if your format differs. Then run the IP check on each distinct address:

bash
grep -i "DuckAssistBot" access.log | awk '{print $1}' | sort -u | while read -r ip; do
  echo "$ip $(check_ddg "$ip")"
done

What to look for:

  • Status codes. A 403 or 429 for a verified address means a firewall or rate-limit rule is turning the bot away. If you want to be a source for DuckDuckGo's AI answers, that is a configuration problem to fix. If you intended the block, it is working.
  • Requests after an opt-out. After the 72-hour window, verified DuckAssistBot requests for pages you disallowed should stop. Unverified requests using the name are not DuckDuckGo.
  • No visits at all. Not seeing the bot does not mean it is blocked. DuckDuckGo's help page gives no crawl schedule and does not say what triggers a fetch, so a quiet site can go without visits.

AI crawlers in server logs shows how to count hits per bot and read status codes across all the major AI crawlers at once.

Will blocking DuckAssistBot change your DuckDuckGo rankings?

DuckDuckGo says no. Its help page states that opting out does not affect organic search rankings or whether your site appears in DuckDuckGo search results. What the opt-out does, per the same page, is make DuckAssistBot stop crawling your site.

This matches the structure of the other AI crawlers. Blocking an answer-time fetcher is a separate decision from blocking the crawler behind ordinary search. If you want to stay out of AI-assisted answers but keep normal DuckDuckGo results, Disallow for DuckAssistBot is the documented route, and you leave DuckDuckBot alone.

It also works in the other direction. Allowing DuckAssistBot removes a technical barrier, nothing more. DuckDuckGo's help page does not say how it chooses which pages to cite, so no setting on your side can make an answer quote you.

Frequently asked questions

Does DuckAssistBot obey robots.txt?

DuckDuckGo documents robots.txt as the way to opt out: disallow the DuckAssistBot user agent and it stops crawling your site after 72 hours. Its help page does not describe how the bot handles other directives such as Allow or crawl-delay. Because a user agent can be faked, requests that ignore your rules may not be DuckAssistBot at all, so verify the IP before blaming the bot.

Does DuckAssistBot use my content to train AI models?

DuckDuckGo says no: the data is not used in any way to train AI models. The claim is DuckDuckGo's own and is not something you can audit from your logs.

Is DuckAssistBot the same as DuckDuckBot?

No. They are documented as separate crawlers, each with its own user agent and its own IP list. DuckDuckBot is described as improving DuckDuckGo's search results; DuckAssistBot as fetching pages for AI-assisted answers. A robots.txt group for one does not apply to the other.

Can I block it at the firewall instead of robots.txt?

You can, and it takes effect immediately, but it is a blunter tool. A firewall block does not tell DuckDuckGo you have opted out, and a rule keyed only on the user agent can be sidestepped by anyone who changes the string. If you block by IP, take the addresses from the published list and refresh them automatically. Prefer robots.txt for a stated policy.

Check what your site says to DuckAssistBot

The free check reads one page the way AI crawlers do, with no account needed, and reports the robots.txt rules that apply to the named AI crawlers it checks. Compare that with the DuckAssistBot group in your own robots.txt to see whether it says what you meant. If you want it checked across a whole site, an audit covers up to 500 pages.

Filed under AI crawlers. Topics: duckduckgo, ai-crawlers, robots-txt, server-logs